RFC - Integration Tests & End to End tests
While this integration test architecture has some benefits:
- uses ruby code
- Mocking data with
fixtures
andfactoyBot
- Direct access to models/DB
- Works naturally with rails
it defiantly has some cons that affect us:
- Too slow
- Debugging is difficult and frustrating
- Random test failures
- Part of the integrations tests are testing some behavior from unit perspective (due to abscent of tools like jest)
- The current architecture seems to be incompatible with react, some tests are failing due to timeout errors, bad handling with promises, etc.
- tests are written in ruby, but communicate mostly with client events
- Gradually moving to another js driver, which may solves some of the issues above
- Replacing capybara for react pages with a more dedicated tool
- Gradually migrate capybara to a different tool
- Suggest any ? :)
On my searching for a modern testing tool, a tool named cypress took my attention the most
an open source project for full browser E2E tests
- integrates with CI such as jenkins and travis
- recording tests - automatically record failing test with a video and a snapshot
- Optional dashbaord - Cypress could be used as a sass with a dashboard, free for open source
- great for debugging tests
- adds a local tool for testing and debugging
- tests are written in javascript - communicate better with the client
- running tests concurrently
- No "out of the box" server side communication
- Different envieromet, javascript based
- ATM only chrome and electron are supported (supprting firefox and IE are developed)
For the demonstration I've created a travis configuration with foreman test environment and cypress
The new react based layout integration test
import { FOREMAN_ADDR } from './config';
describe('Layout', () => {
beforeEach(() => {
cy.visit(FOREMAN_ADDR);
});
it('hover all menu items', () => {
cy.get('.fa-tachometer').trigger('mouseover');
cy.contains('Dashboard').should('be.visible');
cy.get('.fa-server').trigger('mouseover');
cy.contains('All Hosts').should('be.visible');
cy.get('.fa-wrench').trigger('mouseover');
cy.contains('Host Group').should('be.visible');
cy.get('.pficon-network').trigger('mouseover');
cy.contains('Smart Proxies').should('be.visible');
cy.get('.fa-cog').trigger('mouseover');
cy.contains('Users').should('be.visible');
});
it('taxonomies dropdown', () => {
cy.contains('Any Organization').click();
cy.contains('Manage Organizations').should('be.visible');
cy.contains('Any Location').click();
cy.contains('Manage Locations').should('be.visible');
});
it('notfication drawer', () => {
cy.get('#notifications_container').click();
cy.contains('No Notifications Available').should('be.visible');
});
it('user dropdwon', () => {
cy.get('#account_menu').click();
cy.contains('My Account').should('be.visible');
});
it('mobile view', () => {
cy.viewport(550, 750);
cy.contains('Toggle navigation').click();
cy.get('.visible-xs-block .fa-user').click();
cy.contains('My Account').should('be.visible');
});
});
Migrate integration tests, even gradually, might be an overkill, even with these great advantages. however we can take these, and create an E2E architecture for foreman (within production environment). With it we can gain great benefits, for example, discovering js runtime errors, which happen after webpack compilation. after all, this will increase foreman builds stability by far.