Easy Performance Benchmarks in Elixir Using Benchee

Written by Percy Grunwald

— Last Updated February 25, 2019

If you have code you need to optimize, never assume that any change will actually improve performance. Always benchmark your code to confirm your changes have had the desired effect. Using benchee is the best way I’ve found to benchmark Elixir code, so let’s see how you can use it in your projects.

Using benchee to benchmark your code

To get started using benchee, simply add it to your mix.exs and run deps.get:

...
  defp deps do
    [
      ...
      {:benchee, "~> 0.13", only: :dev}
    ]
  end
...

You can use benchee in IEx, but I recommend making an Elixir script (.exs) to make it easy to re-run later. I generally write a script called benchmark.exs that calls Benchee.run/2.

As an example, let’s compare the performance of Enum.map/2 and a for loop for multiplying a range of numbers by 2:

# benchmark.exs

range = 1..1000

Benchee.run(%{
  "Enum.map" => fn ->
    Enum.map(range, fn num -> num * 2 end)
  end,
  "for loop" => fn ->
    for num <- range, do: num * 2
  end
})

You can then execute this script with mix run benchmark.exs to benchmark your code:

$ mix run benchmark.exs 
Operating System: macOS
CPU Information: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
Number of Available Cores: 8
Available memory: 16 GB
Elixir 1.8.1
Erlang 21.2.5
...
Name               ips        average  deviation         median         99th %
for loop       23.86 K       41.92 μs    ±31.90%          38 μs          90 μs
Enum.map       18.20 K       54.95 μs    ±24.75%          51 μs         112 μs

Comparison: 
for loop       23.86 K
Enum.map       18.20 K - 1.31x slower

Passing values in from the command line

You might want to vary the difficulty of your benchmark from the command line:

$ mix run benchmark.exs $MAX_VALUE

In that case, you can use System.argv/0 to get the arguments from the script invocation:

max_value =
  System.argv()
  |> List.first()
  |> String.to_integer()

range = 1..max_value

Benchee.run(%{
  "Enum.map" => fn ->
    Enum.map(range, fn num -> num * 2 end)
  end,
  "for loop" => fn ->
    for num <- range, do: num * 2
  end
})

This allows you to execute the test with the range 1..1_000_000 like this:

$ mix run benchmark.exs 1000000
...
Name               ips        average  deviation         median         99th %
for loop         15.05       66.46 ms    ±15.67%       67.69 ms      100.92 ms
Enum.map         12.86       77.77 ms    ±11.09%       78.87 ms       99.09 ms

Comparison: 
for loop         15.05
Enum.map         12.86 - 1.17x slower

Don’t sacrifice readability for minor performance improvements

The results above might suggest that you should always use for instead of Enum.map/2, but I would answer a few questions before accepting this conclusion:

1. How big is the difference actually?
2. Will the maintainability/readability of my code suffer by making the change?

Let’s answer these questions for this example case.

How big is the difference actually?

With a range of 1..1000, even though Enum.map/2 was 1.31x slower, the average difference between the functions was only 13.03μs – or 13 millionths of a second. That’s so small I would definitely not consider changing existing Enum.map/2 in my code to use for.

At 1..1_000_000 the average difference was 11ms, which might actually make some impact on your application, but I also think that most applications never need to iterate over 1 million items. Let’s put this in the “maybe” pile.

Will the maintainability/readability of my code suffer by making the change?

There’s a big reason you might want to use Enum.map/2 instead of for: pipe-ability.

Consider code like this:

inputs
|> get_results()
|> Enum.map(&process_result)
|> Enum.sort_by(&Map.get(&1, :username))

Perfectly clean Elixir code. Now imagine we want to replace Enum.map/2 with for. Since you can’t pipe into for, you would need to do something like this:

results = get_results(inputs)
processed_result = for result <- results, do: process_result(result)

Enum.sort_by(processed_results, &Map.get(&1, :username))

Definitely not as readable as the first, and not a change I would consider making for 13 microseconds faster execution. If it could get me 11ms faster execution, I might consider it, but I would really need to be sure that the inputs to my application would be large enough to get that improvement.

Conclusion

benchee is a really useful tool to compare the speed of Elixir functions and can be really useful if you need to optimize your code. However, remember that writing the best code is not always a matter of just choosing the fastest function for each task, as you could decrease the readability and maintainability of your code by doing so.

When in doubt, I would suggest writing the cleanest, most expressive code you can and evaluate any potential changes for not only performance impact, but also impact on the readability and maintainability.

Comment & Share