Tips for Testing

Testing is one of my favorite subjects.

For me, automated testing has two goals:

1. Enables faster iteration loops

2. Ensure that when I change code, the old code behaves as it did before.

In the projects I'm involved in, I try to make writing and running a test easier than executing the code in the browser manually.

I don't care if my tests are DRY; I even see it as an antipattern in tests.

I want my test to be:

1. Reliable: they don't break randomly, don't fail when unrelated code is changed

2. Easy to understand: when I change logic, often the test needs to change as well

3. Fast: if tests are slow, people won't run them

A good test suit doesn't replace good system monitoring, error tracking and alerting.

One of the barriers to entry for writing tests is the "ergonomics" of our tools. Requiring too much ceremony or boilerplate for every test will put off even the most enthusiastic testers.

One of the reasons testing is common in Ruby Land is tools do a lot of the setup/tear down for you. Much of this comes from the tight integration of libraries like Rails, RSpec, FactoryBot, and Capybara.

In JavaScript land, things are decoupled, and you need to do too much setup/configuration. You must build your testing tools to integrate the different parts to have a sound testing story. Writing tests should be frictionless so that you can focus only on testing logic.

The second friction point in testing is speed - If it's faster to run the test, I'll do it more often. Having to wait makes you test less frequently.

The best tooling investment I ever made is integrating the vim-test and running the current test from my editor with a simple shortcut. 😇

As we talk about friction, often, people don't know what to test or how to test something. This adds friction and makes people not to write tests.

When I design systems, removing this friction is very important for me.

Here are my tips and "rules" for writing tests, which have helped me spend less time deliberating over whether to write a test and encouraged me to just do it.

Today, I’m going to cover

• Terms

• The Shape of a good test

• How to deal with brittle tests

• Ruby on Rails testing tips

• End-to-End (E2E) testing tips

• JavaScript testing tips

So, lets get started 👉

Terms

Let's start with some terms I'll be using. In the realm of automated testing (as in everything in software engineering), terms are all messed up.

SUD - a system under test; this is they mode of the system while it executes tests. Often is named “test environment”.

E2E tests, known as end-to-end tests or feature specs in Ruby land. These are tests which verify a full feature from start to finish..

Example: Login, fill in a form, and verify the form is stored in the database and email is sent. These often involve running tests in actual or simulated browsers.

In the Ruby, we often use Capybara, while in the JavaScript world, tools like  Cypress are common.

Smoke tests are tests that verify code runs without an error but don't care much if the result of code execution is what is expected from the program. They just make sure nothing has blown up.

They are a good place to start when you don’t have tests or you do major rewrites.

Brittle test - test that fails "randomly". The most frustrating thing with automated tests. Tests that fail time to time. I have whole section in this post about them. 😬

The Shape of a good test

Every test you write should have four phases.

1. Setup - Put SUD in a constant state, set data needed for the test

2. Action - Perform the action we want to verify is doing what we expect it to do

3. Verify - Verify that the result of the action is what we expect it to be

4. Teardown - Puts SUD back to its initial state. This is often done automatically by testing libraries. Example: Resetting the database of all created records during the test.

Example

// test suit
describe(emojiForCountryCode.name, () => {
  // test case
  it('returns the country emoji for given country', () => {
    // no setup, because test doesn't depend on anything
    // action
    const value = emojiForCountryCode('BG');
    // verify
    expect(value).toEqual('🇧🇬');
    // no teardown was needed
  });
});

Here is a similar example in Ruby:

describe Post do 
  describe '#state' do
    it 'is scheduled when scheduled at a time in the future' do
      # setup - create post
      post = create :post, scheduled_at: 1.day.from_now
      
      # action and verify 
      # written on single line, because more readable
      expect(post.state).to eq 'scheduled'
      # teardown - cleans the database, done automatically
    end
  end
end

All my tests in every language and testing framework follow this structure.

It is fine to have multiple assertions in the `verify` step. The goal is not to have `action verify action verify`, but not a single assertion.

So, the following is fine:

describe Post do 
  describe '#schedule' do
    it 'marks post as scheduled' do
      # setup - create post
      post = create :post

      # action 
      post.schedule_to 1.day.from_now
      
      # verify the action by multiple assertions
      expect(post.state).to eq 'scheduled'
      expect(post.scheduled_at).to eq 1.day.from_now
    end
  end
end

There is one group of tests where I don't follow the four phases of testing, - the E2E tests. There, I do multiple actions and verify that the action is completed before continuing to the main step.

setup - create an unpublished blog post

action - visit post edit page
verify - the post is shown on the edit page

action - select new "published_at" date from date picker
verify - the correct date selected from the date picker

action - submit the form
verify - no errors and a success message is shown
verify - post's "published_at" is changed in the database

action - go to the index page
verify - the post is shown on the page

This is because E2E often fails "randomly," and we want to know where exactly something failed. In my example - imagine if I had a custom date picker and the JS broke for it. I didn't test the picker to set the correct value. Then, I get an error that the post has not been published. How much time am I going to lose trying to find this error?

Another thing I do with E2E tests, which is slightly controversial, is use a single test scenario for multiple related flows. This is because setup/teardown steps in E2E take too much time.

So, what I do is to test multiple things in E2E.

# I like to name my E2E tests like this
test "comment > create > edit > destroy" do
  # test create comment with empty form
  # test create a comment with fill-in form
  # test edit of comment
  # test destroy of comment
end

Dealing with brittle tests

This is one of the most frustrating experiences you can have as a developer. Some test randomly fails in CI. You have worked hard on your PR, and when you merge it into the main branch, the test randomly fails. 😡

In my experience, the main reasons for a test to fail "randomly" are:

1/ Time

Often, this is because there is some dependency on time in our test or code.

For example, we expect 30.days.ago, the month will be changed. However, if the month is 31 days or 28 days. A good practice is to freeze time with something like Timecop or isolate current time as a dependency.

Issues can arise when we compare time, too.

Example:

now = Time.now
post = create :post
expect(post.created_at).to eq now // might be 1ms difference 

2/ Leakage of state between tests

In E2E tests running a browser, those can be - cookies, local stage, etc.

I once had a randomly failing E2E test because the previous tests set something in `localStorage`, and the current test was "assuming" localStorage was empty. 🤦‍♂️

My tests in CI usually run in random order, so I had this failure when those two tests were run one after the other.

When you debug failing tests in CI with random order running, make sure locally you. Run tests in the same order, bypassing the random test order seed number.

Similar issues can arise when there are leftovers from the previous run

Fixing this is to make sure `setup` / `teardown` are strict.

E2E tests often suffer from those issues because cleaning after a test like this is often very complicated.

3/ Async code

This also affects E2E tests. In general E2E test tend to be the brittle ones 🤷‍♂️.

Example: click a button and then assert something in database changes. However, you assert that the request is still being processed.

Fix 1: Use wait_for_ajax utility like this one.
Fix 2: Fix before verifying in the database; check if page state is changed (this breaks with optimistic UI changes)

I usually do both things, just in case.

Another example: Animation is running, so the element is not visible on the page

Fix 1: Disable animation
Fix 2: Have timeouts on your DOM assertions

4/ Database order

You make a query and expect post1, post2, and post3, but you get post2, post1, and post3.

The fix is to have an explicit order set in your query or to sort before verifying if you don't care about the order.

5/ Network

If your tests make external network requests, you will see “random” failures. The solution is to block all ongoing network requests in the SUD system.

Don't forget to turn off this in E2E browser tests as well.

Ruby on Rails testing tips

I prefer RSpec over mini-test. I can write a whole post about it.

Most of my tips can be reused with other libraries as well.

What should have a test in Ruby on Rails project?

• Always test

  • The "Active" part of Rails - ActiveJob, ActiveRecord, ActiveModel

  • GraphQL mutations and resolvers classes

    • If the GraphQL type method gets complex, move to the resolver and test resolver

  • Utilities, Service, Search, Form, and Value objects.

• Don't test

  • The "Action" part of Rails - ActionController, ActionViews, ActonMailer, …

  • ViewComponents (for those previews are sufficient, imho)

  • DLS based code with something like ActiveAdmin

    • if you have complicated logic, move it out and test it

  • private methods

    • private objects who are already tested by the interface of the object using them

• Write previews

  • Action Mailer Previews

  • View Components Previews

  • You can write a smoke test that opens a browse, opens all preview pages, and makes sure nothing blows up 💥

Tips for handling context/describe/it nesting

Use .class_method for class methods and #instance_method instance method naming. It helps a lot when when reading the output.

describe User do
  # test class method User.find_and_authorize
  describe ".find_and_authorize"

  # test instance method User.new.show_cookie_banner?
  describe "#show_cookie_banner?"
end

Don't have a single `it` in multiple `context`/`describe`

describe Object do
  describe '.class_method' do
    context 'a' do 
      context 'b' do 
        it 'does something'
       end
       
       context 'c' do 
        it 'does something'
       end
    end
  end
end

# convert to

describe Object do
  describe '.class_method' do
    it 'does something when a and b'
    it 'does something when a and c'
  end
end

It is fine to have describe for each method of object

describe Object do
  describe '.class_method' do
    it 'does something'
  end

  describe '#instance_method' do
    it 'does something as well'
  end
end

Tips for using "let"

Avoid overusing “let”. The key “overusing”, it is fine to it time to time. These are my rules:

• Use let when the object global for test and doesn't need configuration like user or account. Basically objects you don’t care about much

• If any of the let objects need to be modified in “setup” phase, replace let with factory method that creates whats needed and have params.

• If you have more than 2 lets (and especially if they are calling each other), consider inlining them and using factory methods

• Don’t overwrite let between describe blocks

• Don't ever use let! it creates a lot of mystery guest issues

Here and here are good articles illustrating issues with “let”, so use with cation.

General RSpec tips

• Use the expect(value).to eq true syntax.

• Use have_attributes matcher for asserting multiple attributes per object

• Never use subject - it is very implicit and makes tests hard to follow.

• Define custom matches for common assertions, especially for supporting domain functionality.

  • Example: expect(user).to have_received_notification_about(post)

• Use shared examples if you want to verify object from a certain type behaves as an object of this type; good example are concerns behaviors

describe Post do
  it_behaves_like 'votable', factory: post
end

Tips for using Factories

I like using FactoryBot, which creates records for testing.

• Every database model should have a factory defined for

• Factories should be bare minimal (article)

• Use traits to create variants of factories (docs)

  • If you notice in a lot of tests, you do orchestration of 2-3 records to set factories → extract trait

• Include use “config.include FactoryBot::Syntax::Methods “ in your configration

  • this gives “build” and “create” factory creation methods

One big issue with factories is that they often create a lot of background records for a single record.

An example is a system with products and categories. Creating a simple product will also create a category, maybe users, an account where this user belongs... etc. You get the point.

I have found a couple of strategies to deal with this.

1. Use “build” instead “create” where possible

2. Define the minimum amount of association in default factories. Everything that the database allow nullable columns

3. Re-use records from similar relationships.

FactoryBot.define do
  factory :money_transfer do
     association :user
     association :building
     association :source, factory: :withdraw
  end
end

This will create 1 user, 1 building, 1 withdraw and 1 user and 1 building for the withdraw: Total: 5

FactoryBot.define do
  factory :money_transfer do
     association :source, factory: :withdraw
     user { source.user }
     building { source.building }
  end
end

This will create the minimum: 1 user, 1 building, 1 withdraw.

End-to-End (E2E) testing tips

I find a lot of value in those tests.

E2E tests are the best storytellers of what your system actually does.

Often, system breaks and bugs spear in the integration between layers. So having a test that tests a feature from UI to through the backend to the database and email delivery schedule is very useful.

My main goal of the E2E test is to verify

1. Layers of my system work.

2. Help me with big refactoring when I change a lot in database models and logic, but UI and flows should keep working

People don't write those tests because they are slow (to write and run) and often brittle.

Most of the problems are about tooling around writing feature tests.

Good tooling is your saver here. In almost every project I worked on in the last decade, I had a lot of tooling built around making E2E easier.

To have success with E2E tests, you need to optimize your tools for

1. speed of writing

2. easy debug when they break.

Most of my advice here will be about RSpec and Capybara. But again, a lot of my tips work with other technologies.

Tip 1: Proper configuration

RSpec.configure do |config|
  # resize the browser before every test
  # often, tests fail because UI pushes elements out of the viewport
  # if you need to test desktop / mobile sizes, you can use "it" tags  
  config.before(:each, type: :feature) do
    Capybara
      .current_session 
      .driver
      .browser.manage
      .window
      .resize_to(2_000, 2_000)
  end

  # when there is a failed test
  # 1. make and save screenshot of the page
  # 2. print page url and text
  # can't tell you how many hours of debugging those have saved me
  config.after(:each, type: :feature) do |example|
    if example.exception.present?
      timestamp = Time.zone.now.strftime('%Y_%m_%d-%H_%M_%S')
      filename = "#{timestamp}__#{example.full_description}.png"
      screenshot_path = Rails.root.join('tmp', 'screenshots', filename)

      puts "Saving screenshot: #{screenshot_path}"
      Capybara.page.save_screenshot(screenshot_path)

      if Capybara.page.current_url.present?
        puts ''
        puts 'PAGE URL: '
        puts Capybara.page.current_url
        puts 'PAGE TEXT: '
        puts Capybara.page.text
      end
    end
  end
end

Tip 2: Write the test interactively

When I need to write a complicated feature spec, I do the following:

1. Create an empty test case

2. Call the needed factory to set the database

3. Put a "debug" statement

4. Run the tests

5. Then, in the interactive console

   • I write the code step by step to verify the feature.

   • When I'm happy with the step code, I paste in the test

   • Exit when I'm happy with the result

6. Run the test with pasted code

7. Clean the test

8. Move to next task

Tip 3: Use data attributes for matching test elements

// instead of this
element.find('.description button.expand-button').simulate('click');

// write this
element.find('[data-test="test"]').simulate('click');

This clarifies what is used for testing and which is part of UI. In this way, a simple UI redesign doesn't break the test.

Tip 4: Wait for ajax

The big cause for brittle tests is to have a UI action that makes an Ajax call, where you check for Ajax call results before the call has been processed.

Because of this, in all my projects, I have a method named wait_for_ajax; it is always custom for every application.

It waits for ongoing Ajax calls to be done before continuing the test.

Here is a wait_for_ajax, I used with React application using GraphQL.

Tip 5: Write helpers for custom operations

In E2E, you often have to perform a group of related operations. It is useful to have shared helpers for those.

For example, if you have a fancy calendar picker. Every time you use it, you will have to write the following:

- click data-test="calendar-picker"
- enter the correct year in a text input
- pick a correct month from month picker
- click on the correct day
- click the "ok" button to save

Having a single helper function makes a lot more sense, so when you change "ok" with "done", you can do it in one place.

select_calendar_picker(date)

Here are some other examples for grouping operations:

login_as(user)
enter_comment_with(text)
submit_comment_with(text)
close_modal()
enter_in_from(values)

You can hide your `verify` checks in those helpers.

This is essential to have a maintainable E2E test suite. Ideally, a good test should read like a story, and its implementation details would be hidden.

One of my favorite helpers is `submit_form` (I write it customarily on every project because forms and custom form elements differ between projects):

```ruby

# regular capybara with helpers for custom inputs
fill_in 'Name', 'Rado'
fill_in_custom_calendar_picker 'Date label', date
check 'Verify'
select_from_custom_select 'Select', 'value'

# helper that hides all forms interactions
submit_form(
  name: 'Rado',
  date: date,
  verify: true,
  select: 'value'
)
```

Here is an example of an E2E test from  Angry Building. It tests the CRUD of our issue-type module. 

feature "Issue Module" do
  scenario 'types > new > list > update > destroy' do
    building = create :building

    sign_in_operator(building)

    visit account_issues_path(building.account)

    # test: create issue type
    click_on I18n.t(:page_title_issue_types)

    click_on I18n.t(:action_new_issue_type)

    submit_form

    expect_form_errors :blank

    # NOTICE this helper
    # it will raise error if there is validation error
    # name - is string
    # severity - is select
    # visible_in_app - is checkbox
    submit_form!(
      name: 'New Issue Type',
      severity: 'warning',
      visible_in_app: false,
    )

    expect_flash_message :create

    expect(page).to have_content 'New Issue Type'

    type = building.account.issue_types.first!

    expect(type).to have_attributes(
      name: 'New type',
      severity: 'warning',
      visible_in_app: false,
    )

    # test: update issue type
    click_on_edit

    submit_form(
      name: 'Updated Issue Type',
      severity: 'low',
      visible_in_app: true,
    )

    expect_flash_message :update

    expect(page).to have_content 'Updated Issue Type'

    type.reload

    expect(type).to have_attributes(
      name: 'Updated Issue Type',
      severity: 'low',
      visible_in_app: true,
    )

    # test: destroy issue type (without related issues)
    issue = create :issue, building: building, type: type

    click_on_destroy

    expect_to_be_destroyed type
    expect_not_to_be_destroyed issue
  end
end

JavaScript testing tips

Again, many of my tips (especially the E2E ones) apply for other environments.

As mentioned, testing JavaScript code is not as fun as testing Ruby. This is mostly due to tooling. Jest is good, but it can't compare with RSpec.

The code in most npm packages isn't designed with testing in mind. In JavaScript project’s, often you write a lot of glue code to connect many tiny packages.

How I test JavaScript depends on whether I use JavaScript for frontend or backend.

• If JavaScript is used for backend

  • I do a lot more automated testing with rules similar to what I do with Ruby.

  • I try to isolate and not test external dependencies.

• If JavaScript is used for frontend

  • I mostly test pure utility function

    • The heavy lift of testing is done vie E2E tests

  • In React, I sometimes just test hooks

    • I do "Extract hook refactoring" and then test the hook (video, code)

On tip: when testing with JavaScript, don't hard code names of functions:

// if you change the function name you don't need to change the test
// bonus, you know naming your function works.
describe(emojiForCountryCode.name, () => {
  // ... 
});

If you use TypeScript, do you still need tests? 🤔

TypeScript is like a smoke test, making sure your functions are called with the functions, the right arguments, and components are integrated. It helps a lot with refactoring.

It is still useful to have tests for more complicated logic and E2E tests

Conclusion

I can write a lot more on the subject of testing. However, this is getting a bit too log. 😅

People don't write automated tests due to a lack of tooling and not knowing how to write tests. I hope that these tips have helped you on both fronts.

If you have any questions or comments, you can ping me on Threads, LinkedIn, Mastodon, Twitter or just leave a comment below 📭