Split Thread: Fixtures vs Factories in Elixir

Please try; otherwise your comment is nothing but negative energy.

Factories hide a lot of complexity and make it really easy for you to test toward positive cases (it also typically uses the same flows that you should be testing). With fixtures, you can pretty much ensure that you have data that looks a lot more like what you’re going to see in real data. Factories are far more likely to result in subtle logic bugs than fixtures, because they’re not typically just applied data.

I haven’t found a good way to do fixtures in Elixir and have been tempted more than once to try to port Rails fixtures over for Ecto (but don’t have the time), but I enforced the use of fixtures over factories on my last two Rails projects and we had more tests that ran faster than the last time I worked with a Rails project that used factories. The factories themselves introduced a 30%+ slowdown in test running.

I’m using simplified factories in my current Elixir projects, but I hate them. They have been the source of a half-dozen test bugs across the projects, and result in more churn than I would like.

In my previous Rails projects, when I needed to test a particular set of “empty” table behaviours, I would just truncate the table in question and build from there. But that only applies to the first few times your code runs in the real world, most of the time.

Interesting. Could you maybe give a code example of what the difference is exactly between a fixture and a factory and why fixtures are better?

I’ve been doing some quick googling but they all point into the direction that factories are better because fixtures hide the data you are using in the tests

am I misunderstanding fixtures? Usually I have something like this:

test/support/my_data_fixture.ex

defmodule MyAppTest.MyDataFixture do
  @default_fields [foo: "bar", baz: nil]
  def new(supplied_fields) do
    fields = Keyword.merge(supplied_fields, @default_fields)

    MyData
    |> struct(fields)
    |> apply_to_db_or_other_state()
  end
end

Interesting. Could you maybe give a code example of what the difference is exactly between a fixture and a factory and why fixtures are better?

It’s hard, because part of the point is that fixtures aren’t code. Fixtures are a set of data that represents a baseline set of data that covers a large portion of your tests for the unit under test. If a particular unit could benefit from similar but not quite the same data, you can manipulate the fixture data (after it’s loaded) in the test (or test set, e.g., a describe block).

With Rails fixtures, you define the data as YAML files, which the FixtureSet code turns into SQL inserts into the database not using your model’s new/save functions. (It uses your model to help determine relations, but it sidesteps all of the validation and other business logic.) Because it’s a YAML file, you might have something like:

# users.yml
luke:
  first_name: Luke
  last_name: Skywalker
  title: Jedi Knight

You can refer to this fixture as users(:luke), which is pretty much the same as create(:user, :luke) would be in one of the factory providers…without any hidden behaviour that might be present in the code behind either the user or user_luke factories.

There’s more to it than that (fixture YAML files are parsed through ERB prior to processing, which means you can generate large amounts of data as fixtures if you need to do so through loops).

Essentially, though, the point is that it’s named (mostly) static data that is loaded through direct interfaces rather than through your application code in the first place.

I’ve been doing some quick googling but they all point into the direction that factories are better because fixtures hide the data you are using in the tests

Yeah. That’s nonsense promulgated by people who don’t understand fixture data and the value of having a small but reasonable set of data loaded quickly into your database. Let’s also be clear, betterspecs is also lying about what a fixture is in the example provided. Look at the linked issue, and you’ll see a ton of discussion about it in the Rails context, and proper use of fixtures looks nothing like the example provided on the betterspecs page.

I’m talking about database fixtures here, but fixtures are often used by the very people who deride them in contexts other than the database:

• If you’re doing CSV parser tests, the test files themselves are fixtures. The only time you’d write a CSV to disk and then read it would be to test the roundtrip capability of your CSV generator and parser, for example.
• If you’re doing payload tests, if you have VCR files or JSON response files that represent a payload…that’s fixtures.

With fixtures, you understand your data much better because you have to think about it in terms of the data, not in terms of your objects (don’t get me started on a rant about OO modelling vs data modelling and why you can’t do the former if you don’t understand the latter).

That’s basically a factory. A simple factory (the one that I am using in my current Elixir codebase is more complex, but not that much more complex), but it’s not a fixture. In database terms, you’d create some SQL statements to seed your test data (usually before your test transactions start, but in Ecto you’d need to run that in the same process, so…) and then manipulate the records that are present to get them into the shape you need for a particular test—but the default set of data loaded would cover > 80% of your tests (no, really, and it would only take 2–3 records per table to do that most of the time).

Edit: deleted since i didn’t see @halostatue’s response above the response to mine.

I did some reading on fixtures and I think it wouldn’t be so hard to write something.

test/support/fixture.ex

defmodule MyAppTest.MyFixture do
  require EEx

  file = Path.join(__DIR__, "fixtures/my_fixture.yaml")
  EEx.function_from_file(:defp, :to_rows, file, [:assigns])

  @spec load(pos_integer, (pos_integer -> %{assigns: map})) :: :ok
  def load(count, generator) do
    0..count-1
    |> Enum.flat_map(fn idx ->
      idx
      |> generator.()
      |> to_rows
      |> Yaml.from_string  # this is not the correct function name, but I don't usually use yaml.
    end)
    |> Enum.each(&add_to_database/1)
  end

  defp add_to_database(map) do
    ...
  end
end

Another argument in favour of fixtures over factories is that they are much faster as you don’t need to insert a bunch of data into the database at the start of each test, it is inserted once at the start of the suite.

edit: Oh! I now see @halostatue already covered that!

I’m not sure if trading the need to handle cleanup / setup each time is worth the time saved by not preparing data for each test individually.