Author here. I didn't mention it because I wasn't writing an evaluation of fixtures. Just writing about how to make better use of fixtures. I actually use both fixtures and factories depending on the project specifics and also whether it is even my decision to make. :)
Author here. Yes, what you describe sound where much like what I call Factories (and that's what they're usually called in Ruby land, and some other languages).
The problem arises when they're used to generate Database records, which is a common approach in Rails applications. Because you're generating a lot of them you end up putting a lot more load on the test database which slows down the whole test suite considerably.
If you use them to generate purely in memory objects, this problem goes away and then I also prefer to use factories (or generators, as you describe them).
Ah, ok, now I understand. Ok, I wasn't talking about that. From what I understand about property based testing it's sort of half way between regular example based testing and formal proofs: It tries to prove a statement but instead of a symbolic proof it does it stohastically via a set of examples?
Unfortunately, I'm not aware of a good property based testing library in Ruby, although it would be useful to have one.
Even so I'm guessing that property based testing in practice would be too resource intensive to test the entire application with it? You'd probably only test critical domain logic components and use regular example tests for the rest.
Oh, that's a very different set of requirements than I was thinking, and I missed that context even though you did mention database testing at one point. You're right, property-based testing is less helpful in that situation, because your database may contain legacy data that your current application code must be able to read but also shouldn't be able to write.
I'm familiar with snapshot testing for UI and I agree with you, they can work really well for this because they're usually quick to verify. And especially if you can build in some smart tolerance to the comparison logic, it can be really easy to maintain.
But how would you do snapshot testing for behaviour? I'm approaching the problem primarily from the backend side and there most tests are about behaviour.
I'm also primarily on the back end. Like most backenders, I spend my workdays on http endpoints that return json. When I test these the "snapshot" is a json file with a pretty-printed version of the endpoint's response body. Tests fail when the file generated isn't the same as the existing file.
Ah, Ok, yes, for API endpoints it makes a lot of sense. Especially if it's a public API, you need to inspect the output anyway, to ensure that the public contract is not broken.
But, I spend very little or no time on API endpoints since I don't work on projects where the frontend is an SPA. :)
When you say inheritance do you mean DRY as in "Don't repeat yourself"?
I'm not sure what you mean by inheritance in tests but DRY is criminally overused in tests. That could be a whole separate article but the tradeoffs are very different between test and app code and repetition in the test code is much less problematic and sometimes even desirable.
Both actually. But having to open up three files to figure out how this thing is setup and then override setup to change it slightly in my one case. You get the idea. A really good DSL can help in the areas where creating the SUT is very complex.
In theory you're 100% right, a true unit test is completely isolated from the rest of the system and than a lot of the problems disappear.
In reality, that is also not free. It imposes some restrictions on the code. Sometimes being pragmatic, backing off from the ideal leads to faster development and quicker deliver of value to the users. Rails is big on these pragmatic tradeoffs. The important thing is that we know when and why we're making the tradeoff.
Usually I go with Rails defaults and usually it's not a problem. Sometimes, when the code is especially complex and perhaps on the critical path, I turn up the purity dial and go down the road you describe exactly for the benefits you describe.
But when I decide that sticking to the defaults the is right tradeoff I want to get the most of it and use Fixtures (or Factories) in the optimal way.
Author here. I'm a big fan of factories but the slowness is a real drag on large test suites. If you're considering switching, remember that you can do it gradually, there's no law against using both fixtures and factories in the same project, in some cases (mostly on very complex domain data models) even makes sense: fixtures for the base setup that all tests share, factories for additional test specific records.
Thanks! While I have you, since you seem to know what's up with this stuff, I'm going to ask you a question I have been curious about, in Rails land too.
While I see the pro's (and con's) of fixtures, one thing I do _not_ like is Rails ordinary way of specifying fixtures, in yaml files. Especially gets terrible for associations.
It's occured to me there's no reason I can't use FactoryBot to create what are actually fixtures -- as they will be run once, at test boot, etc. It would not be that hard to set up a little harness code to use FactoryBot to create objects at test boot and store them (or logic for fetching them, rather) in specified I dunno $fixtures[:some_name] or what have you for referal. And seems much preferable to me, as I consider switching to/introducing fixtures.
But I haven't seen anyone do this or mention it or suggest it. Any thoughts?
I use the pattern you describe, but not in Ruby. I use code to build fixtures through sql inserts. The code creates a new db whose name includes a hash of the test data (actually a hash the source files that build the fixtures).
Read-only tests only need to run the bootstrap code if their particular fixture hasn’t been created on that machine before. Same with some tests that write data but can be encapsulated in a transaction that gets rolled back at the end.
Some more complex tests need an isolated db because their changes can’t be contained in a db transaction (usually because the code under test commits a db transaction). These need to run the fixture bootstrap every time. We don’t have many of these so it’s not a big deal that they take a second or two. If we had more we would probably use separate, smaller fixtures for these.
Your thinking is sound. At the end of the day Rails default fixtures is nothing more than some code that reads yaml files and creates records once at the start of test suite run.
So you can definitely use FactoryBot to create them. However, the reason I think that's rarely done is that you're pretty likely to start recreating a lot of the features of Rails fixtures yourself. And perhaps all you need to do is to dynamically generate the yaml files. Rails yaml fixtures are actually ERB files and you can treat is an ERB template and generate its code dynamically: https://guides.rubyonrails.org/testing.html#embedding-code-i...
If that is flexible enough for you, it's a better path since you'll get all the usual fixture helpers and association resolving logic for free.
I feel like i don't _want_ the association resolving logic really, that's what I don't like! And if it's live ruby instead of YAML, it's easy to refer to another fixture object by just looking it up as a fixture like normal? (I guess there's order of operation issues though,hm).
And the rest seems straightforward enough, and better to avoid that "compile to yaml" stage for debugging and such.
We'll see, maybe I'll get around to trying it at some point, and release a perversely named factory_bot_fixtures gem. :)
What I feel is really missing from factories is the ability to do bulk inserts of a whole chain of entries (including of different kinds). That is where 95% of the inefficiency comes from. As an additional bonus it would make it easy to just list everything single record that was created for a spec
Hm, if you missed it, perhaps I should have emphasised this part more, maybe add a blank line before it ...
Regarding the fact that the test does not check that the scope returns "all" active projects, that's a bit more complex to address but let me let tell you how I'm thinking about it:
The point of tests is to validate expected behaviours and prevent regressions (i.e. breaking old behaviour when introducing new features). It is impossible for tests to do this 100%. E.g. even if you test that the scope returns all active projects present in the fixtures that doesn't guarantee that the scope always returns all active projects for any possible list of active projects. If you want 100% validation your only choice is to turn to formal proof methods but that's whole different topic.
You could always add more active project examples. When you write a test that is checking that "Active projects A,B and C" are returned that is the same test as if your fixtures contained ONLY active projects A,B and C and then you tested that all of them are returned. In either case it is up to you to make sure that the projects are representative.
So by rewriting the test to check:
1. These example projects are included.
2. These other example projects are excluded.
You can write a test that is equally powerful as if you restricted your fixtures just to those example projects and then made an absolute comparison. You're not loosing any testing power. Expect you're making the test easier to maintain.
Does that make sense? Let me know which part is still confusing and I'll try to rephrase the explanation.
I want to start by saying that I agree with what you're trying to accomplish here. And I agree with some of the ways you go about it. I'm trying to find the right words to covey what I mean here, but... the best I can come with is... what I'm saying here isn't "you're wrong because", it's "what you're doing seems to miss some situations; here's what I do that helps for those".
> The "doesn't include non-active projects objections is easy", please check the Example 1 test again, there's a line for that:
You're correct; I totally missed that.
> In either case it is up to you to make sure that the projects are representative.
That's fair, but that's also the point you're trying to address / make more robust by how you're trying to write tests (what the article is about). Specifically
- The article is about: How to make sure you're tests are robust against test fixtures changing
- That comment says: It's up to you to make sure your test fixtures don't change in a way that breaks your tests
> You can write a test that is equally powerful as if you restricted your fixtures just to those example projects and then made an absolute comparison. You're not loosing any testing power. Expect you're making the test easier to maintain.
By restricting your fixtures to just the projects (that are relevant to the test), you're making _the tests_ easier to maintain; not just the one test but the test harness as a whole. What I mean is that you're reducing "action at a distance". When you modify the data for your test, you don't need to worry about what other tests, somewhere else, might also be impacted.
Plus you do gain testing power, because you can test more things. For example, you can confirm it returns _every_ active project.
All that being said, what I'm talking about relies on creating the test data local to the tests. And doing that has a cost (time, generally). So there's a tradeoff there.
I think I'm getting what you mean and I almost completely agree with you, let me address one part, the only part where I don't agree:
> Plus you do gain testing power, because you can test more things. For example, you can confirm it returns _every_ active project.
Imagine this:
1. You start with some fixtures. You crafted the fixtures and you're happy that the fixtures are good for the test you're about to write.
2. You write a test where you assert the EXACT collection that is returned. This is, as you say, a test that "confirms the scope returns _every_ active project".
3. You now rewrite the test so that it checks that the collection includes ALL active projects and excludes all inactive projects.
Do you agree that nothing changed when you went from 2 to 3? As long as you don't change the fixtures, those 2 version of the test will behave exactly the same: if one passes so will the other and if one fails so will the other. As long as fixtures don't change they have exactly the same testing power.
If you agree on that, now imagine that you added another project to the fixtures. Has the testing power of the tests changed just because fixtures have been changed?
> If you agree on that, now imagine that you added another project to the fixtures. Has the testing power of the tests changed just because fixtures have been changed?
No, _but_ (and this is a big _but_) you're not testing the contract of the method, which (presumably) is to return all and only active projects.
Testing that it returns _some_ of the active methods is useful, but there are cases where it won't point out an issue. For example, image
- Over time, more tests are added "elsewhere" that use the same fixtures
- More active projects are added to the fixture to support those tests
- The implementation in the method is changed to be faster, and an off-by-one error is introduced; so the last project in the list isn't returned
In that ^ case, testing that _some_ of the active projects are returned will still return true; the bug won't be noticed.
Not directly related to the above, but I'll note that I would also split 2/3 into different tests.
- Make sure all projects returned are active
- Make sure projects returned includes all active projects
I think that's more of a style thing, but I _try_ to stick to each test testing one and only one thing. I don't always do that, but it's a rule of thumb for me.
I'm with you on the one assertion per test. I bundled two assertions into the same test here because my whole point was to have them effectively together describe a single test, just in a more maintainable manner.
Regarding the fact that I'm not fully testing the contract of the method, you're absolutely correct. But also, no example based test suite is fully doing that. As long as the test suite is example based it is always possible to find a counter-case where the contract is violated but the test suite misses it.
These counter-cases will be more contrived and less likely the better the test suite. So all of us at some point decide that we've done enough and that more contrived cases are so unlikely and the cost of mistake is so small that it's not worth it to put in the extra testing effort. Some people don't explicitly think about it but that decision is still made one way or another.
This is a long way of saying that I both agree with you but that also, in most cases, I would still take the tradeoff and go for more maintainable tests.
There's always a scenario where this can break though. What happens if someone introduces a test that confirms that marking `active1` as inactive works. Then it depends on the test order whether your initial test still passes.
It's required for tests to clean up after themselves. With Rails and fixtures this is handled by default: each test runs inside a transaction which is rolled back at the end of the test. That way each test starts with the same initial state.
Have you used it over the last few years? It has it been rapidly improving, mainly because Shopify put a team full time on it. It doesn’t take a lot of people to optimize a VM/interpreter it just has to be the right people.
And the question is always “fast enough for what?” Different languages are more suitable for different types of projects. I wouldn’t code a rendering engine in Ruby but for web apps it’s amazing.
Yes, every web app I’ve worked on the past ~18 years has been with Rails. I’ve seen it all except an efficient app. Sure, Ruby and Rails never bankrupted these companies but they’d all have been better off with something else. Certain cloud bills would’ve been much smaller for sure.
Those optimizations to the VM are just very workload specific and become less relevant today when you’re using containers and fractional CPU/mem. It also doesn’t take much for a dev to write the wrong code and make them irrelevant again. Even if you get everything right you’re leaving so much performance on the table it feels like crumbs.
For small web apps Rails is fine though. I just never worked on one. The issue is perhaps no one threw the code away when it got big.
Rails apps can get very expensive server wise because the “IO is slow anyways” attitude means more servers will be needed to serve the same amount of requests. For a specific bad case I worked at, the cloud bill was the same cost of 15 senior developers. And it was an app without external users (I was actually responsible for the external parts of it, it was isolated and not in Rails).
Excessive abstraction at the ORM can also make it extremely difficult to optimize db queries, so each user request can trigger way more DB queries than necessary, and this will require more db power. I have seen this happening over and over due to abstraction layers such as Trailblazer, but anything that is too layered clean-code style will cause issues and requires constant observation. And refactoring is made difficult due to “magic”. Even LLMs might find it too much.
Another problem with the slowness is that it slows down local development too. The biggest test suite I ever saw took 2 hours to run in a 60-machine cluster, so 120 hours of CI. Impossible to run locally, so major refactoring was borderline impossible without a huge feedback cycle.
The solution for the slow development ends up being hiring more developers, of course, with each one responsible for a smaller part of the app. In other companies these kind of features I saw would be written by people over days, not by team over months.
The terseness of both Ruby and Rails is also IMO countered by the culture of turning 10-line methods into bigger classes and using methods and instance variables instead of local variables. So it also hurts both readability (because now you have 5x more lines than needed) but also hurts optimization and stresses the garbage collection. If you know this, you know. I have seen this in code from North+Latin American, European and Japanese companies, so it’s not isolated cases. If you don’t know I can provide examples.
I have seen this happening with other tech too, of course, but with Rails it happens much much faster IME.
It is also 100% preventable, of course, however a lot of advice on how to prevent these problems will clash with Ruby/Rails traditions and culture.
These are just examples out of personal experience, but definitely not isolated cases IMO.
Just want to reiterate what the sibling commenter said, it's dead on with my experience.
Static typing would be the main thing teams would've been better off with. I was big on dynamic languages, love Clojure/LISPs and still work with Ruby and JS today, but you just can't trust 100 developers with it. Last company I worked for I ran the dev team and did some bug analysis: conservatively 60% of bugs were things a simple static type system would've caught.
Very few business logic bugs. We had loads of tests but these simple bugs still popped up. Someone in team A would change a method's return type, find and replace in the codebase, but miss some obscure case from team D. Rinse and repeat. Nothing complicated, just discipline but you can't trust discipline on a 500k LoC codebase and a language with no guardrails.
Performance would've been the other main advantage of static typing. While most people think their Rails app will be IO-bound forever that's really downplaying their product. In actuality every company that mildly succeeds will start to acquire CPU-bound workloads and it'll come a point where they are the bottleneck. One might argue that it is at this point you ditch Ruby but in reality no one really wants to run a polyglot company: it's hard to hire, hard to fill in gaps, hard to manage and evaluate talent.
People underestimate the impact of performance on the bottom line these days with phrases like "memory is cheap; devs are not". Like the sibling commenter put it the monthly cloud bill on that last company would've paid about 20 dev salaries. Most of that was for the app servers. That for an app that served about 500 req/sec at peak. You can imagine the unnecessary pressure that puts on the company's finances.
Better choices would've been Go, Rust, even something on the JVM.
Static typing has come, there's RBS, which has (finally) coalesced (after adding support for inlining in code) as the blessed type notation, supported by both steep and sorbet. Considering that big companies have adopted it, I'd say that the community agrees and has done something about it. But as you can imagine, many ruby apps have been stuck in legacy for years.
About performance, not sure how you think static typing could solve it, but considering the significant investment recently in JITs, in particular YJIT and ZJIT, again, the big apps seem to agree with you and have done something about it?
Even if you ditch CRuby for the JVM, you can still use JRuby, and still leverage the language while not being pulled down by the runtime.
The thing about "it's IO bound anyway so who cares" is that it forces you to scale the app much earlier.
At a company I worked, a single Golang instance (there was backup) was able to handle every single request by itself during peak hours, and do authentication, partial authorization, request enrichment, fraud detection, rate limiting and routing to the appropriate microservice. I'm not saying it was a good idea to have a custom Ingress/Proxy app but it's what we had.
By contrast, the mesh of Rails applications required a few hundred machines during peak time to serve the same number of requests, and none of it was CPU-heavy. It was DB heavy!
If it had been a Golang or JVM or Rust app it would require a much smaller fleet to serve.
Personally, I even slightly prefer to use Factories and I also previously wrote about a better way to use them: https://radanskoric.com/articles/test-factories-principal-of...