Testing is one of my favorite subjects.
For me, automated testing has two goals:
Enables faster iteration loops
Ensure that when I change code, the old code behaves as it did before.
In the projects I'm involved in, I try to make writing and running a test easier than executing the code in the browser manually.
I don't care if my tests are DRY; I even see it as an antipattern in tests.
I want my test to be:
Reliable: they don't break randomly, don't fail when unrelated code is changed
Easy to understand: when I change logic, often the test needs to change as well
Fast: if tests are slow, people won't run them
A good test suit doesn't replace good system monitoring, error tracking and alerting.
One of the barriers to entry for writing tests is the "ergonomics" of our tools. Requiring too much ceremony or boilerplate for every test will put off even the most enthusiastic testers.
One of the reasons testing is common in Ruby Land is tools do a lot of the setup/tear down for you. Much of this comes from the tight integration of libraries like Rails, RSpec, FactoryBot, and Capybara.
In JavaScript land, things are decoupled, and you need to do too much setup/configuration. You must build your testing tools to integrate the different parts to have a sound testing story. Writing tests should be frictionless so that you can focus only on testing logic.
The second friction point in testing is speed - If it's faster to run the test, I'll do it more often. Having to wait makes you test less frequently.
The best tooling investment I ever made is integrating the vim-test and running the current test from my editor with a simple shortcut. 😇
As we talk about friction, often, people don't know what to test or how to test something. This adds friction and makes people not to write tests.
When I design systems, removing this friction is very important for me.
Here are my tips and "rules" for writing tests, which have helped me spend less time deliberating over whether to write a test and encouraged me to just do it.
Today, I’m going to cover
So, lets get started 👉
Terms
Let's start with some terms I'll be using. In the realm of automated testing (as in everything in software engineering), terms are all messed up.
SUD - a system under test; this is they mode of the system while it executes tests. Often is named “test environment”.
E2E tests, known as end-to-end tests or feature specs in Ruby land. These are tests which verify a full feature from start to finish..
Example: Login, fill in a form, and verify the form is stored in the database and email is sent. These often involve running tests in actual or simulated browsers.
In the Ruby, we often use Capybara, while in the JavaScript world, tools like Cypress are common.
Smoke tests are tests that verify code runs without an error but don't care much if the result of code execution is what is expected from the program. They just make sure nothing has blown up.
They are a good place to start when you don’t have tests or you do major rewrites.
Brittle test - test that fails "randomly". The most frustrating thing with automated tests. Tests that fail time to time. I have whole section in this post about them. 😬
The Shape of a good test
Every test you write should have four phases.
Setup - Put SUD in a constant state, set data needed for the test
Action - Perform the action we want to verify is doing what we expect it to do
Verify - Verify that the result of the action is what we expect it to be
Teardown - Puts SUD back to its initial state. This is often done automatically by testing libraries. Example: Resetting the database of all created records during the test.
Example
// test suit
describe(emojiForCountryCode.name, () => {
// test case
it('returns the country emoji for given country', () => {
// no setup, because test doesn't depend on anything
// action
const value = emojiForCountryCode('BG');
// verify
expect(value).toEqual('🇧🇬');
// no teardown was needed
});
});
Here is a similar example in Ruby:
describe Post do
describe '#state' do
it 'is scheduled when scheduled at a time in the future' do
# setup - create post
post = create :post, scheduled_at: 1.day.from_now
# action and verify
# written on single line, because more readable
expect(post.state).to eq 'scheduled'
# teardown - cleans the database, done automatically
end
end
end
All my tests in every language and testing framework follow this structure.
It is fine to have multiple assertions in the `verify` step. The goal is not to have `action verify action verify`, but not a single assertion.
So, the following is fine:
describe Post do
describe '#schedule' do
it 'marks post as scheduled' do
# setup - create post
post = create :post
# action
post.schedule_to 1.day.from_now
# verify the action by multiple assertions
expect(post.state).to eq 'scheduled'
expect(post.scheduled_at).to eq 1.day.from_now
end
end
end
There is one group of tests where I don't follow the four phases of testing, - the E2E tests. There, I do multiple actions and verify that the action is completed before continuing to the main step.
setup - create an unpublished blog post
action - visit post edit page
verify - the post is shown on the edit page
action - select new "published_at" date from date picker
verify - the correct date selected from the date picker
action - submit the form
verify - no errors and a success message is shown
verify - post's "published_at" is changed in the database
action - go to the index page
verify - the post is shown on the page
This is because E2E often fails "randomly," and we want to know where exactly something failed. In my example - imagine if I had a custom date picker and the JS broke for it. I didn't test the picker to set the correct value. Then, I get an error that the post has not been published. How much time am I going to lose trying to find this error?
Another thing I do with E2E tests, which is slightly controversial, is use a single test scenario for multiple related flows. This is because setup/teardown steps in E2E take too much time.
So, what I do is to test multiple things in E2E.
# I like to name my E2E tests like this
test "comment > create > edit > destroy" do
# test create comment with empty form
# test create a comment with fill-in form
# test edit of comment
# test destroy of comment
end
Dealing with brittle tests
This is one of the most frustrating experiences you can have as a developer. Some test randomly fails in CI. You have worked hard on your PR, and when you merge it into the main branch, the test randomly fails. 😡
In my experience, the main reasons for a test to fail "randomly" are:
1/ Time
Often, this is because there is some dependency on time in our test or code.
For example, we expect 30.days.ago
, the month will be changed. However, if the month is 31 days or 28 days. A good practice is to freeze time with something like Timecop or isolate current time as a dependency.
Issues can arise when we compare time, too.
Example:
now = Time.now
post = create :post
expect(post.created_at).to eq now // might be 1ms difference
2/ Leakage of state between tests
In E2E tests running a browser, those can be - cookies, local stage, etc.
I once had a randomly failing E2E test because the previous tests set something in `localStorage`, and the current test was "assuming" localStorage was empty. 🤦♂️
My tests in CI usually run in random order, so I had this failure when those two tests were run one after the other.
When you debug failing tests in CI with random order running, make sure locally you. Run tests in the same order, bypassing the random test order seed number.
Similar issues can arise when there are leftovers from the previous run
Fixing this is to make sure `setup` / `teardown` are strict.
E2E tests often suffer from those issues because cleaning after a test like this is often very complicated.
3/ Async code
This also affects E2E tests. In general E2E test tend to be the brittle ones 🤷♂️.
Example: click a button and then assert something in database changes. However, you assert that the request is still being processed.
Fix 1: Use wait_for_ajax
utility like this one.
Fix 2: Fix before verifying in the database; check if page state is changed (this breaks with optimistic UI changes)
I usually do both things, just in case.
Another example: Animation is running, so the element is not visible on the page
Fix 1: Disable animation
Fix 2: Have timeouts on your DOM assertions
4/ Database order
You make a query and expect post1, post2, and post3, but you get post2, post1, and post3.
The fix is to have an explicit order set in your query or to sort before verifying if you don't care about the order.
5/ Network
If your tests make external network requests, you will see “random” failures. The solution is to block all ongoing network requests in the SUD system.
Don't forget to turn off this in E2E browser tests as well.
Ruby on Rails testing tips
I prefer RSpec over mini-test. I can write a whole post about it.
Most of my tips can be reused with other libraries as well.
What should have a test in Ruby on Rails project?
Always test
The "Active" part of Rails - ActiveJob, ActiveRecord, ActiveModel
GraphQL mutations and resolvers classes
If the GraphQL type method gets complex, move to the resolver and test resolver
Utilities, Service, Search, Form, and Value objects.
Don't test
The "Action" part of Rails - ActionController, ActionViews, ActonMailer, …
ViewComponents (for those previews are sufficient, imho)
DLS based code with something like ActiveAdmin
if you have complicated logic, move it out and test it
private methods
private objects who are already tested by the interface of the object using them
Write previews
Tips for handling context/describe/it nesting
Use .class_method
for class methods and #instance_method
instance method naming. It helps a lot when when reading the output.
describe User do
# test class method User.find_and_authorize
describe ".find_and_authorize"
# test instance method User.new.show_cookie_banner?
describe "#show_cookie_banner?"
end
Don't have a single `it` in multiple `context`/`describe`
describe Object do
describe '.class_method' do
context 'a' do
context 'b' do
it 'does something'
end
context 'c' do
it 'does something'
end
end
end
end
# convert to
describe Object do
describe '.class_method' do
it 'does something when a and b'
it 'does something when a and c'
end
end
It is fine to have describe
for each method of object
describe Object do
describe '.class_method' do
it 'does something'
end
describe '#instance_method' do
it 'does something as well'
end
end
Tips for using "let"
Avoid overusing “let”. The key “overusing”, it is fine to it time to time. These are my rules:
Use
let
when the object global for test and doesn't need configuration likeuser
oraccount
. Basically objects you don’t care about muchIf any of the
let
objects need to be modified in “setup” phase, replacelet
with factory method that creates whats needed and have params.If you have more than 2 lets (and especially if they are calling each other), consider inlining them and using factory methods
Don’t overwrite
let
betweendescribe
blocksDon't ever use
let!
it creates a lot of mystery guest issues
Here and here are good articles illustrating issues with “let”, so use with cation.
General RSpec tips
Use the
expect(value).to eq true
syntax.Use have_attributes matcher for asserting multiple attributes per object
Never use subject - it is very implicit and makes tests hard to follow.
Define custom matches for common assertions, especially for supporting domain functionality.
Example: expect(user).to have_received_notification_about(post)
Use shared examples if you want to verify object from a certain type behaves as an object of this type; good example are concerns behaviors
describe Post do
it_behaves_like 'votable', factory: post
end
Tips for using Factories
I like using FactoryBot, which creates records for testing.
Every database model should have a factory defined for
Factories should be bare minimal (article)
Use traits to create variants of factories (docs)
If you notice in a lot of tests, you do orchestration of 2-3 records to set factories → extract trait
Include use “config.include FactoryBot::Syntax::Methods “ in your configration
this gives “build” and “create” factory creation methods
One big issue with factories is that they often create a lot of background records for a single record.
An example is a system with products and categories. Creating a simple product will also create a category, maybe users, an account where this user belongs... etc. You get the point.
I have found a couple of strategies to deal with this.
Use “build” instead “create” where possible
Define the minimum amount of association in default factories. Everything that the database allow nullable columns
Re-use records from similar relationships.
FactoryBot.define do
factory :money_transfer do
association :user
association :building
association :source, factory: :withdraw
end
end
This will create 1 user, 1 building, 1 withdraw and 1 user and 1 building for the withdraw: Total: 5
FactoryBot.define do
factory :money_transfer do
association :source, factory: :withdraw
user { source.user }
building { source.building }
end
end
This will create the minimum: 1 user, 1 building, 1 withdraw.
End-to-End (E2E) testing tips
I find a lot of value in those tests.
E2E tests are the best storytellers of what your system actually does.
Often, system breaks and bugs spear in the integration between layers. So having a test that tests a feature from UI to through the backend to the database and email delivery schedule is very useful.
My main goal of the E2E test is to verify
Layers of my system work.
Help me with big refactoring when I change a lot in database models and logic, but UI and flows should keep working
People don't write those tests because they are slow (to write and run) and often brittle.
Most of the problems are about tooling around writing feature tests.
Good tooling is your saver here. In almost every project I worked on in the last decade, I had a lot of tooling built around making E2E easier.
To have success with E2E tests, you need to optimize your tools for
speed of writing
easy debug when they break.
Most of my advice here will be about RSpec and Capybara. But again, a lot of my tips work with other technologies.
Tip 1: Proper configuration
RSpec.configure do |config|
# resize the browser before every test
# often, tests fail because UI pushes elements out of the viewport
# if you need to test desktop / mobile sizes, you can use "it" tags
config.before(:each, type: :feature) do
Capybara
.current_session
.driver
.browser.manage
.window
.resize_to(2_000, 2_000)
end
# when there is a failed test
# 1. make and save screenshot of the page
# 2. print page url and text
# can't tell you how many hours of debugging those have saved me
config.after(:each, type: :feature) do |example|
if example.exception.present?
timestamp = Time.zone.now.strftime('%Y_%m_%d-%H_%M_%S')
filename = "#{timestamp}__#{example.full_description}.png"
screenshot_path = Rails.root.join('tmp', 'screenshots', filename)
puts "Saving screenshot: #{screenshot_path}"
Capybara.page.save_screenshot(screenshot_path)
if Capybara.page.current_url.present?
puts ''
puts 'PAGE URL: '
puts Capybara.page.current_url
puts 'PAGE TEXT: '
puts Capybara.page.text
end
end
end
end
Tip 2: Write the test interactively
When I need to write a complicated feature spec, I do the following:
Create an empty test case
Call the needed factory to set the database
Put a "debug" statement
Run the tests
Then, in the interactive console
I write the code step by step to verify the feature.
When I'm happy with the step code, I paste in the test
Exit when I'm happy with the result
Run the test with pasted code
Clean the test
Move to next task
Tip 3: Use data attributes for matching test elements
// instead of this
element.find('.description button.expand-button').simulate('click');
// write this
element.find('[data-test="test"]').simulate('click');
This clarifies what is used for testing and which is part of UI. In this way, a simple UI redesign doesn't break the test.
Tip 4: Wait for ajax
The big cause for brittle tests is to have a UI action that makes an Ajax call, where you check for Ajax call results before the call has been processed.
Because of this, in all my projects, I have a method named wait_for_ajax
; it is always custom for every application.
It waits for ongoing Ajax calls to be done before continuing the test.
Here is a wait_for_ajax, I used with React application using GraphQL.
Tip 5: Write helpers for custom operations
In E2E, you often have to perform a group of related operations. It is useful to have shared helpers for those.
For example, if you have a fancy calendar picker. Every time you use it, you will have to write the following:
- click data-test="calendar-picker"
- enter the correct year in a text input
- pick a correct month from month picker
- click on the correct day
- click the "ok" button to save
Having a single helper function makes a lot more sense, so when you change "ok" with "done", you can do it in one place.
select_calendar_picker(date)
Here are some other examples for grouping operations:
login_as(user)
enter_comment_with(text)
submit_comment_with(text)
close_modal()
enter_in_from(values)
You can hide your `verify
` checks in those helpers.
This is essential to have a maintainable E2E test suite. Ideally, a good test should read like a story, and its implementation details would be hidden.
One of my favorite helpers is `submit_form
` (I write it customarily on every project because forms and custom form elements differ between projects):
```ruby
# regular capybara with helpers for custom inputs
fill_in 'Name', 'Rado'
fill_in_custom_calendar_picker 'Date label', date
check 'Verify'
select_from_custom_select 'Select', 'value'
# helper that hides all forms interactions
submit_form(
name: 'Rado',
date: date,
verify: true,
select: 'value'
)
```
Here is an example of an E2E test from Angry Building. It tests the CRUD of our issue-type module.
feature "Issue Module" do
scenario 'types > new > list > update > destroy' do
building = create :building
sign_in_operator(building)
visit account_issues_path(building.account)
# test: create issue type
click_on I18n.t(:page_title_issue_types)
click_on I18n.t(:action_new_issue_type)
submit_form
expect_form_errors :blank
# NOTICE this helper
# it will raise error if there is validation error
# name - is string
# severity - is select
# visible_in_app - is checkbox
submit_form!(
name: 'New Issue Type',
severity: 'warning',
visible_in_app: false,
)
expect_flash_message :create
expect(page).to have_content 'New Issue Type'
type = building.account.issue_types.first!
expect(type).to have_attributes(
name: 'New type',
severity: 'warning',
visible_in_app: false,
)
# test: update issue type
click_on_edit
submit_form(
name: 'Updated Issue Type',
severity: 'low',
visible_in_app: true,
)
expect_flash_message :update
expect(page).to have_content 'Updated Issue Type'
type.reload
expect(type).to have_attributes(
name: 'Updated Issue Type',
severity: 'low',
visible_in_app: true,
)
# test: destroy issue type (without related issues)
issue = create :issue, building: building, type: type
click_on_destroy
expect_to_be_destroyed type
expect_not_to_be_destroyed issue
end
end
JavaScript testing tips
Again, many of my tips (especially the E2E ones) apply for other environments.
As mentioned, testing JavaScript code is not as fun as testing Ruby. This is mostly due to tooling. Jest is good, but it can't compare with RSpec.
The code in most npm packages isn't designed with testing in mind. In JavaScript project’s, often you write a lot of glue code to connect many tiny packages.
How I test JavaScript depends on whether I use JavaScript for frontend or backend.
If JavaScript is used for backend
I do a lot more automated testing with rules similar to what I do with Ruby.
I try to isolate and not test external dependencies.
If JavaScript is used for frontend
I mostly test pure utility function
The heavy lift of testing is done vie E2E tests
In React, I sometimes just test hooks
I do "Extract hook refactoring" and then test the hook (video, code)
On tip: when testing with JavaScript, don't hard code names of functions:
// if you change the function name you don't need to change the test
// bonus, you know naming your function works.
describe(emojiForCountryCode.name, () => {
// ...
});
If you use TypeScript, do you still need tests? 🤔
TypeScript is like a smoke test, making sure your functions are called with the functions, the right arguments, and components are integrated. It helps a lot with refactoring.
It is still useful to have tests for more complicated logic and E2E tests
Conclusion
I can write a lot more on the subject of testing. However, this is getting a bit too log. 😅
People don't write automated tests due to a lack of tooling and not knowing how to write tests. I hope that these tips have helped you on both fronts.
If you have any questions or comments, you can ping me on Threads, LinkedIn, Mastodon, Twitter or just leave a comment below 📭
Thanks, Rado! It was very useful as always :)