Introduction to Testing Elixir Applications with ExUnit

Introduction

Every language needs a solid test framework, and that framework needs to provide mechanisms that allow a developer to exercise the features of the language. To that end, Elixir comes bundled with ExUnit to allow developers to make use of all the features Elixir provides without having to compromise on unit tests.

In this tutorial, we will discuss the basic idea behind units and test-driven development. Then, we’ll learn how to test a simple parallel map function in Elixir using a typical test-driven development workflow and show some of the conveniences offered by ExUnit. In doing so, we will exercise a number of Elixir’s functional, concurrent, and message-passing features, while testing that we are using those features as intended.

Goals

By the end of this tutorial, you will:

• Understand the basic structure and function of ExUnit unit tests,
• Grasp the differences between testing pattern matches vs. equivalence,
• Add tests for log output and message passing to drive development of new capabilities in our function, and
• Reduce duplication by using an ExUnit “context”.

Prerequisites

For this tutorial, you will need a working installation of Elixir 1.3.2, 1.3.3, or 1.3.4.

Introduction to ExUnit

To get started, we need to create a new Elixir project: mix new hello_exunit

In the directory created by Mix, we find a directory called test, which contains two files:

• test_helper.exs
• hello_exunit_test.exs

The first thing to note is that all of our tests must be contained in Elixir scripts with the .exs extension, not the usual compiled .ex extension.

The test_helper.exs script just contains the ExUnit.start() term, which is required before we use ExUnit.

The hello_exunit_test.exs script contains a basic test that demonstrates how to assert a basic truth:

  test "the truth" do
    assert 1 + 1 == 2
  end

You can run all tests from the root directory of the project by running: mix test

Like most test frameworks, ExUnit doesn’t give us many details about tests that pass since we only need to take action on failing tests.

While simple, this first test is informative, as it introduces us to a couple of basic but important concepts in unit testing Elixir code. The first bit to notice is the assert macro, which receives an Elixir term and evaluates its “truthiness”. In this case, we’re effectively asking it to evaluate whether this statement is true: “the expression 1 + 1 is equivalent to the expression 2.

To see this action in reverse, modify the test to read:

  test "the truth" do
    assert 1 + 1 == 3
  end

When we run this test, we see a failure:

1) test the truth (HelloExunitTest)
   test/hello_exunit_test.exs:5
   Assertion with == failed
   code: 1 + 1 == 3
   lhs:  2
   rhs:  3
   stacktrace:
     test/hello_exunit_test.exs:6: (test)

Finished in 0.05 seconds
1 test, 1 failure

Here we can see the ways ExUnit can be very helpful in troubleshooting failing tests. ExUnit’s output for a failed test looks very similar to pattern match errors in our normal Elixir code, even when we are asserting with ==. This makes interpreting our test output more familiar, and generally easier.

There’s another informative though subtle piece of this test, too. One of Elixir’s most powerful features is pattern matching via the = operator. It’s important to note that this test does not test a pattern match, as it uses the ==, or the equivalence operator. In ExUnit, a pattern match that succeeds (i.e. Elixir is able to make the left-hand side of the expression match the right-hand side) is always a success.

We can see this in practice with the following test:

  test "good match" do
    assert a = 3
  end

When run, ExUnit will report that this test passed since match is legitimate. It will, however, still warn us that we have an unused variable:

warning: variable a is unused
  test/hello_exunit_test.exs:10

..

Finished in 0.05 seconds
2 tests, 0 failures

Similar to our earlier failed test, a failed pattern match is exposed clearly by ExUnit’s output, as shown by this test:

  test "bad match" do
    assert 2 = 3
  end

  1) test bad match (HelloExunitTest)
     test/hello_exunit_test.exs:13
     match (=) failed
     code: 2 = 3
     rhs:  3
     stacktrace:
       test/hello_exunit_test.exs:14: (test)

.

Finished in 0.05 seconds
3 tests, 1 failure

Test-driven Development

The core ideas behind test-driven development (TDD), are that code should be developed with very short cycle times, and the code should only address the specific requirements that have been laid out. To achieve these goals, TDD encourages writing a failing test that attempts to test whether a specific requirement has been met, and then updating the application code as minimally as possible, to make the test pass.

You may have noticed that we’ve not yet written any application code, and we’re going to continue on the same path as we start building our parallel map function.

Test-driving Our Function

To start, let’s delete all of the tests from our hello_exunit_test.exs script and start fresh. The first requirement we have for our parallel map function is that it simply manages to map values in either a list or a tuple by applying whatever function we provide it. For now, we’re effectively testing a bare-bones wrapper around Elixir’s Enum.map/2 function, but we’ll extend it soon. First, let’s write our test and be sure to include the import line for convenience:

  import HelloExunit.PMap

  test "pmap maps a list" do
    assert [2,4,6] = pmap([1,2,3], fn x -> x * 2 end)
  end

Running this test will fail, since we’ve not yet build the PMap module, nor the pmap/2 function within that module:

  1) test pmap maps a list (HelloExunitTest)
     test/hello_exunit_test.exs:5
     ** (UndefinedFunctionError) function HelloExunit.PMap.pmap/2 is undefined
(module HelloExunit.PMap is not available)
     stacktrace:
       HelloExunit.PMap.pmap([1, 2, 3], #Function<0.6591382/1 in
HelloExunitTest.test pmap maps a list/1>)
       test/hello_exunit_test.exs:6: (test)

Finished in 0.04 seconds
1 test, 1 failure

The first thing we need to do is define our module and our function, so let’s do so now in lib/pmap.ex:

  defmodule HelloExunit.PMap do
    def pmap(coll,fun) do
      Enum.map(coll, fun)
    end
  end

Now, if we run our test again, it should pass just fine.

Testing Message Receipt

Time for our next requirement — pmap/2 should run an asynchronous task to calculate the new value for each element in the list. Testing this is a bit more involved, as by default there are no mocks or stubs in ExUnit. Using such things in Elixir is generally discouraged, so we should try to find a way to test this requirement without using those mechanisms. This is a case where Elixir’s message passing can help us out.

One way to test that we are indeed spawning asynchronous tasks to handle our computation is to have each task send a message back to our pmap/2 function, which we can wait for. Let’s write the test, and then update our code:

  test "pmap spawns async tasks" do
    pmap([1,2,3], fn x -> x * 2 end)

    assert_receive({pid1, 2})
    assert_receive({pid2, 2})
    assert_receive({pid3, 2})
    refute pid1 == pid2
    refute pid1 == pid3
    refute pid2 == pid3
  end

This test fails, as expected:

  1) test pmap spawns async tasks (HelloExunitTest)
     test/hello_exunit_test.exs:10
     No message matching {pid1, 2} after 100ms.
     The process mailbox is empty.
     stacktrace:
       test/hello_exunit_test.exs:12: (test)

Our second test introduces a macro — assert_receive/3. When we need to make sure that a particular message is received by the calling process, in this case our test, we use assert_receive/3 to wait for some amount of time , by default 100ms, for a message that matches the pattern we specify to be received. When it doesn’t show up, we get the failure message as shown above.

Now, let’s update our code, to make the test pass:

  def pmap(collection, function) do
    caller = self
    collection
    |> Enum.map(fn x -> spawn_link(fn ->
                  send caller, { self, function.(x) }
                  end)
                end)
  end

Let’s run the tests again:

  1) test pmap maps a list (HelloExunitTest)
     test/hello_exunit_test.exs:6
     match (=) failed
     code: [2, 4, 6] = pmap([1, 2, 3], fn x -> x * 2 end)
     rhs:  [#PID<0.116.0>, #PID<0.117.0>, #PID<0.118.0>]
     stacktrace:
       test/hello_exunit_test.exs:7: (test)

Regression Tests

We’ve introduced a regression. We’re no longer returning any value from our function, so the first test has started to fail. Our new asynchronous test, however, works as expected. This is the reason we keep around tests that might test a subset of functionality that another test implicitly exercises. It’s possible that a new test with more and/or slightly different logic could pass, but existing functionality is broken in making the new test pass. This is the general idea behind regression tests — keep tests around that will highlight broken functionality that might result from future test-driven code.

Let’s fix our code so that both tests pass as following:

  def pmap(collection, function) do
    caller = self
    collection
    |> Enum.map(fn x -> spawn_link(fn ->
                  send caller, { self, function.(x) }
                  end)
                end)
    |> Enum.map(fn task_pid -> (receive do { ^task_pid, result } -> result end) end)
  end

Now, we receive the results and return them. We have to make sure they are in order – hence pattern matching on the pinned task_pid variable. Now, we have a new problem — the message box for the calling process is empty. This causes our async test to fail again. For the sake of this tutorial, we’ll add an extra message send event that will indicate completion of the calculation without being consumed for the return value.

First, we’ll update the test:

  test "pmap spawns async tasks" do
    pmap([1,2,3], fn x -> x * 2 end)

    assert_received({pid1, :ok})
    assert_received({pid2, :ok})
    assert_received({pid3, :ok})
  end

Next, we update the code as follows:

  def pmap(collection, function) do
    caller = self
    collection
    |> Enum.map(fn x -> spawn_link(fn ->
                  send caller, { self, function.(x) }
                  send caller, { self, :ok }
                  end)
                end)
    |> Enum.map(fn task_pid -> (receive do { ^task_pid, result } -> result end) end)
  end

Checking for Negative Conditions

Both tests should pass now. There’s a small detail we’ve missed, though — our asynchronous test doesn’t actually validate that three separate tasks did the calculations. It’s possible that our function’s implementation could just send messages to itself to “trick” us into thinking multiple tasks ran concurrently, so let’s fix that by introducing a new macro:

  test "pmap spawns async tasks" do
    pmap([1,2,3], fn x -> x * 2 end)

    assert_received({pid1, :ok})
    assert_received({pid2, :ok})
    assert_received({pid3, :ok})
    refute pid1 == pid2
    refute pid1 == pid3
    refute pid2 == pid3
  end

The refute macro is the opposite of assert – it passes when the expression given does not evaluate to true.

DRYing the Tests

It’s generally wise to follow the DRY philosophy when writing tests: Don’t Repeat Yourself. Now that our tests are working, let’s consider ways to reduce duplication in the test code itself before adding more tests. We can use ExUnit’s setup callback for this. This callback is run before each test, and it returns a map ,here named context, that contains whatever information you might want to access during the test. Here, we will just add an input list and an output list that can be used throughout our tests. Update your test script to resemble the following:

  setup _context do
    {:ok, [in_list: [1,2,3],
           out_list: [2,4,6]]}
  end

  test "pmap maps a list", context do
    assert context[:out_list] == pmap(context[:in_list], fn x -> x * 2 end)
  end

  test "pmap spawns async tasks", context do
    pmap(context[:in_list], fn x -> x * 2 end)

    assert_received({pid1, :ok})
    ...

As described in the ExUnit documentation, returning a tuple containing {:ok, keywords} will merge the keywords key-value pairs into the context map and make them available to every test.

Testing Log Output

For our final test, let’s add a requirement that the PMap function must log a debug message to indicate that the function has started calculating values. Our final test will look as follows:

  test "pmap logs a completion debug log message", context do
    assert capture_log(fn ->
        pmap(context[:in_list], fn x -> x * 2 end)
      end) =~ "[debug] pmap started with input: " <> inspect(context[:in_list])
  end

For this test to run, we’ll need to include this line toward the top of the file:

  import ExUnit.CaptureLog

This test introduces the capture_log/2 macro, which accepts a function and returns a binary containing any Logger messages that may have been emitted by that function. Here, we assert that the binary fuzzy matches the log entry we intend to emit from pmap/2. Because capture_log/2 can potentially capture log output from any function that is running during our tests, we should also change our use line to the following to avoid this behavior:

use ExUnit.Case, async: false

This will cause all of the tests defined in this test module to run serially instead of asynchronously, which is certainly slower, but safer if we were to expand our test suite to capture more log output.

Now, let’s update our function to emit the log message:

  require Logger

  def pmap(collection, function) do
    caller = self
    Logger.debug("pmap started with input: #{inspect(collection)}")
    collection
    |> Enum.map(fn x -> spawn_link(fn ->
    ...

Now, all the tests should be passing, and we’re on our way to write better tests for our application.

Test Enforcement via Automated Continuous Integration

As with any code project, a great way to ensure consistent code quality and enforce regression testing is to employ some manner of automatic continuous integration (CI) system to run your tests for you. There are a number of such services available that can run ExUnit tests for you automatically under different circumstances, triggered by certain conditions such as a merge to the master branch of your code repository. One such service is Semaphore CI, which will import your Elixir project and automatically configure a set of build and test jobs that can run against your code to ensure consistency and sanity.

Here are the quick steps needed to get our Elixir project built in Semaphore:

1. Once you’re logged into Semaphore, navigate to your list of projects and click the “Add New Project” button:
2. Select your cloud organization, and if you haven’t already done so, select a repository host:
3. Select the repository that holds the code you’d like to build:
4. Select an appropriate version of Elixir and confirm the job steps are appropriate, then click “Build Project” at the bottom of the page:
5. Watch your job build, and check out the results.

Conclusion

In the course of this tutorial, we have:

• Gained a basic familiarity with the structure of ExUnit unit tests,
• Learned how to use ExUnit to test features that that are core to Elixir’s strengths, and
• Used a typical Test Driven Development process to implement a fully-tested Elixir application

With this knowledge, we can build stronger and better Elixir projects that can be safely extended and improved thanks to ExUnit. If you have any questions and comments, feel free to leave them in the section below. Happy building!