Skip to content

Instantly share code, notes, and snippets.

@hon2a
Last active March 2, 2018 15:14
Show Gist options
  • Save hon2a/9d0f624c86560c59f5581eceb8283872 to your computer and use it in GitHub Desktop.
Save hon2a/9d0f624c86560c59f5581eceb8283872 to your computer and use it in GitHub Desktop.
Testing with Nightwatch

Testing with Nightwatch: The Bad and the Ugly

Nightwatch aims to provide an easy-to-use wrapper around Selenium WebDriver. Even though at the time of writing it's quite outdated, compared to other current alternatives, but it's still OK for simple cases. However, trying to build a large, maintainable set of test suites for modestly sized web app uncovers quite a few shortcomings. This text aims to list them and for each one describe both the issue and some way to work around it.

Note that this text is meant to supplement Nightwatch Developer Guide, not replace it.

Aside from the specific problems listed below, a general drawback of using Nightwatch is the way it is configured. Extensions (e.g. page objects or custom commands) are fed to it through paths, with filenames being used as names of the entities. This obfuscates all dependencies and makes it hard to distribute tooling into separate packages.

Page Objects

The main vehicle for abstractions in Nightwatch are so-called "page objects" and their sections. A page object is meant to encapsulate API for controlling a particular web page. It can have inner sections to separately encapsulate controllers for some clearly delimited parts of the page. Page objects and sections are also where most of the issues with Nightwatch lie - all of the problems described below are in some way tied to page objects.

Nightwatch API Parts

Nightwatch's API is separated into four groups: BDD-style assertions, regular assertions, Nightwatch commands, and WebDriver commands. While this separation may seem superficial at first glance, it's actually quite important. The important division is into the two following categories:

  • high-level (simple) commands - BDD-style assertions and Nightwatch commands - these have simpler contracts and work in context of page objects
  • low-level commands - regular assertions and WebDriver commands - these have contracts based on external specs (WebDriver, NodeJS assertions) and work only as globals

Low-level commands impose severe limitations, so it's best to use them only when absolutely necessary. Additionally, only the high-level commands are provided as methods on page objects and their sections.

Page Objects vs. Sections

Page object sections are just like page objects themselves except for a single important differerence - they have an additional selector property to target the section's container element. Selectors of elements defined inside a section are applied in context of section's container when passed to high-level commands executed on the section.

When testing a web application rather than a web page, page objects are best used only as containers for sections, so that the controller APIs are restricted to the parts of the application they control and they're similarly reusable.

Selectors

Nightwatch allows choosing between two "locate strategies" for element selection - css selector and xpath. There's a major difference in how the choice is made between WebDriver commands and the rest of the API. Those of the former that perform element selection accept locateStrategy as the first argument. There are only a few of these as the element selection is separated from other operations, which use the element IDs retrieved by the few selection methods.

section.element(
  'xpath',
  '//input[name="useBetterApi"]',
  ({ value: { ELEMENT } }) => browser.elementIdSelected(
    ELEMENT,
    ({ value: isSelected }) => console.log(isSelected)
  )
)

The rest of the API shares a global strategy selection with a configurable default value and two global methods for switching between the strategies - useXpath and useCss. The selection being both global and private makes using this quite awkward and brittle.

BDD-style assertions support passing locateStrategy as the second argument (though this is undocumented). If the second argument is not supplied, they use the global strategy as well.

Page Object Elements

Page objects and sections support defining elements to serve as aliases for selectors. The definitions contain the locateStrategy tied to each selector and ignore the global selection, so using elements can alleviate the issues with it. Unfortunately the definitions have to be static, so any dynamic selection, e.g. selecting items from large set by their labels, has to happen directly through the commands, without elements. Still, it's best to use elements when possible to simplify command code.

Best Practice: Locate Strategy

In our codebase, we resolved the issue of global private strategy by agreeing on a common default (css selector) and a usingXpath helper that switches to XPath just while running the commands passed to it.

const selector = `//a[.="${someLabel}"]`;
section.usingXpath(
  () => browser
    .waitForElementVisible(selector)
    .click(selector)
);

This replaces useCss and useXpath completely.

Best Practice: Selectors

Element selectors should be robust enough to survive through minor changes to app layout. To this end, they need to find just the right kind and amount of specificity. The following guidelines help achieve this.

  1. CSS selectors should be favored over XPath. They're easier to read and allow clean selection by class name, the most common mode of selection. When using XPath for class selection, contains(@class, "...") is the right tool, not exact match (@class="...").
  2. Tag names should only be used when necessary and only for meaningful tags, e.g. button or input, not div or span.
    '.Dialog_container button[type=submit]' // good
    'div.Dialog_container h1' // bad, even the heading should be selected by a class if possible
  3. Exact markup structure should only be used when necessary, e.g. when component nesting is possible.
    '.App_main > .HeaderLayout_body', // good, there might be another `HeaderLayout` inside
    '.App_main > div > .App_sidebar' // bad, needlessly specific structure

Note that these guidelines heavily depend on application markup. In our codebase, application components have unobfuscated class names with component-specific unique prefixes, making class names the ideal method of element selection.

XPath Selectors in Page Object Sections

When using XPath selectors in page object section elements, selectors need to begin with . to be executed in context of the section's root element, e.g. .//button[@type=submit]. Without that the selectors are executed globally, with no regard to the section context. While this can be detrimental when applied by mistake, it can be useful to break out of context when trying to select content rendered separately from the main application body but tied in functionality to the section.

const userManagement = {
  sections: {
    userSettings: {
      selector: '.userSettings',
      sections: {
        // the drop-down is semantically part of user settings but actually rendered outside the app
        userRolesDropdown: {
          // selector starting with '//' searches anywhere on the page, even outside the parent section
          selector: '//*[contains(@class, ".Dropdown_dropdown")]',
          elements: {
            // selector starting with './/' searches just inside the section
            adminRole: './/*[.="Admin"]'
          },
          // ...
        }
      },
      // ...
    }
  }
};

Selector Context in Page Object Sections

Only the selectors of elements defined in a page object section are applied in context of the section (with the exception of global XPath selectors as described above). Selectors passed directly to commands, even high-level ones, are executed globally. This makes dynamic selection in context of page object sections impossible, leaving the whole concept of sections severely lacking. Getting around this limitation requires some serious h4cking (see below).

Custom Commands

Nightwatch's API can be extended through "custom commands". When supplied to Nightwatch, these are added both to the global API and to page objects and their sections. However, even when custom commands get called from page object sections, they retain the global API as their context unlike commands defined directly in page object section definitions. In other words, globally defined custom commands have no access to the page object section they are called from, so they are not extensions equivalent to existing high-level commands.

Best Practice: Command Context

Having local and global commands combined on page object sections can lead to confusion. To avoid this, "custom" (global) commands should be used only for actions that are by their nature incompatible with the page object section context and they should always be called from the global API.

// Selenium doesn't support native drag&drop yet, so we have an implementation that requires executing
// a script in the browser. Browser script execution is by its nature a context-less action.
browser.dragAndDrop('.section-container .source', '.section-container .target');
section.waitForElementVisible('.target .source');

Mass Element Selection

The provided set of element-related Nightwatch commands when compared to the WebDriver commands shows a gaping omission. The high-level commands only ever deal with single element selection. There's even a config variable for making the tests fail whenever a selector supplied to a high-level command finds more than one element. Combined with the aforementioned limitation on selector context, there are no tools provided for e.g. retrieving labels from a list of items and making assertions about the whole list. Finally, the limitation on custom command context prevents simply extending the set of high-level commands to include the mass selection tools.

Best Practice: Page Objects

The above issues combined mean that trying to architect scalable and maintainable end-to-end tests with Nightwatch (as is) is largely an exercise in futility. Getting around them is dealt with in the next section. Regardless of how the various hurdles are surmounted, there's a simple guideline to make the tests maintainable and scalable.

Tests should operate on the application in user terms through the page objects.

Page objects should completely encapsulate DOM manipulation, creating an abstraction buffer between the user requirements the tests are supposed to verify and the app implementation. This helps make the tests resilient and their guarantees clear. Proper encapsulation ensures that even large-scale refactoring of the application has mostly 1:1 impact on page objects and little to no impact on tests themselves.

Achieving a clean encapsulation of implementation-specific controller code means the following.

  1. The tests should only interact with the app through page objects. Custom commands can be helpful to effectuate common global actions, e.g. login, but even they should work through page objects.
  2. The only part of page objects or their sections considered public are the local commands. elements aliases are for private use inside their page object / section only.
const tests = {
  // good - test operates purely in user terms
  'Meaning of life should be correctly calculated'(browser) {
    const { calculator } = browser.page.meaningOfLife().section;
    calculator
      .waitForVisible()
      .startCalculation()
      .waitForCalculationToComplete()
      .expectResult(42);
  },
  // bad - test directly manipulates DOM, even though it does it through page object section
  'There should be cake'() {
    const { calculator } = browser.page.meaningOfLife().section;
    calculator
      .waitForElementVisible('@container')
      .click('@startCalculation')
      .waitForElementNotPresent('@calculating')
      .expect.element('@result').text.to.equal('42');
  }
}

Operations on Absent Elements

In a modern web application, most of the actions are asynchronous. Oftentimes the needed time is so small that a user doesn't even notice the delay. Unfortunately, automated tests are much faster than human operators, so they can uncover these delays. A lot of Nightwatch commands already deals with this by waiting for configured time before declaring an element absent. Some of them don't, however, which can make tests using them fail unpredictably. waitForElementVisible may be used before such commands to deal with the issue.

The most commonly used command that doesn't wait for element to appear is click. In addition to the possible momentary absence of the target element, the target can also be disabled, making the command pass without warning and leading to later failures with unclear cause. The best practice in our codebase is using our helper waitForAndClick instead. But as it needs to be a full-fledged equivalent to Nightwatch commands (executed in section context), it depends on the h4cks described below.

Hooks, Failures, and Clean-up

All's well and good as long as your test suites can just end on failure, wherever it may happen, which is the default behavior provided by Nightwatch. However, if your tests require clean-up that is executed even on failure, e.g. because they need to modify a shared environment, there are a few things to know and consider.

To even start, end_session_on_fail needs to be turned off in Nightwatch test settings, so that it's still possible to communicate with the browser after a test case fail. (Not to be confused with skip_testcases_on_fail, which makes the test suite go on after a failed case when turned off.)

Being able to interact with the browser from the after hook, where the clean-up needs to be located, is a good start. Unfortunately it's also a step on the way to mysterious test suite failures, because when a test suite fails in an after hook, Nightwatch doesn't report it properly as a test suite failure. This leads to Nightwatch declaring that all tests passed while returning a failure exit code. (When a suite fails in a before hook, it's reported as a failure of the first test case in the suite.) Additionally, turning off end_session_on_fail means that the session needs to be ended manually after the clean-up is done. If the clean-up fails, the session can remain open, leaving the browser instance hanging and the driver possibly unresponsive.

So if making do without clean-up is not an option, it should at least be implemented in a way that makes it impossible to fail. This requires using WebDriver commands directly (or their proxies described at the end of this text) to be able to look for elements without failing automatically when they're not found.

H4cking Nightwatch

Page object functionality limitations outlined above severely hamper the effort of making a Nightwatch-based test suite maintainable. This section describes the tooling we've created to work around them.

Page Object Extensions

Nightwatch offers no way to hook directly into its page object instantialization process, so the only way to modify their functionality is through monkey-patching them after they're created. This requires tests to no longer create page objects directly and instead access them through our patcher. (See getPageObjects.js.)

By default, we aim to improve two aspects of page object / section functionality:

  1. High-level commands that select elements should use provided selectors in context of the page object / section they're called on.
  2. It should be possible to add more commands to all page objects / sections to be executed in context of the page object / section they're called on.

Use Selectors in Section Context

Since the native high-level commands only work in section context when used with aliases defined in elements, making the same apply to raw selectors passed to them requires registering new elements for those selectors. And as Nightwatch doesn't provide access to element instantializer, we have to clone existing elements. (See useSelectorsInContext.js.) This imposes the following important restriction:

All page objects and sections need to define at least one element in elements.

Additionally, as the current value of the global locateStrategy is inaccessible, selectors need to be assigned locateStrategy automatically, based on their format. The assigner assumes that all XPath selectors should start either with ./ or with /. As this is just a h4ck, possibly temporary, care should be taken not to exploit this and still write the page object section command code with the assumption that selectors use the global locateStrategy.

const section = {
  elements: {
    submit: {
      selector: './/button[@type="submit"]',
      locateStrategy: 'xpath'
    }
  },
  commands: [{
    submit() {
      return this
        // good - explicit `locateStrategy` switch
        .usingXpath(
          () => this.waitForElementNotPresent('.//*[contains(@class, "processing")]')
        )
        // good - `elements` have `locateStrategy` baked in
        .waitForAndClick('@submit')
        // bad - even though this works now, it'd stop if we ever removed the h4ck
        .waitForElementNotPresent('.//*[contains(@class, "Dialog_container")]');
    }
  }]
};

Add Commands to All Page Objects and Sections

There's no way to push section context to global commands, so generic local commands need to be added to the page objects / sections during the monkey-patching. To that end, local commands should be defined separately, unbeknownst to Nightwatch, and fed to the getPageObjects maker. Global commands can still be used but only for logic inherently unconnected to any page object / section context.

Multi-element Operations

Now that we're able to properly extend the set of omnipresent Nightwatch commands with our own, we can fill in the missing multi-element operation commands. Making use of the WebDriver commands, we can implement the recursive element search ourselves and provide high-level variants of all of the element state getters for both single- and multi-element selection. To make them fully on par with existing Nightwatch commands, we also need to support element aliases in addition to raw selectors. (See webDriverElementCommands.js for details.)

section
  // `elementId` and `elementIds` are low-level function for contextual element selection
  // this is just an example, there exists an additional `elementsCount` command just for this
  // note that the low-level command `elementIds` doesn't support aliases and unwrap result
  .elementIds('.hamster', ({ value }) => expect(value.length).to.be.above(1))
  // `getText` could be used as well, but that doesn't unwrap the result
  .elementText('xpath', './/a[2]', linkText => expect(linkText).to.equal('Release Hamsters'))
  // this multi-element operation could not be otherwise performed in context of a page object section
  // note the support for aliases on high-level commands `element*` and `elements*`
  .elementsAttribute('@actions', 'title', tooltips => expect(tooltips).to.include('Observe the mayhem'));
import _ from 'lodash';
import * as baseCommands from '../commands';
import { useSelectorsInContext } from './useSelectorsInContext.js';
import { addCommandsToPageObject } from './addCommandsToPageObject.js';
export function createPageObjectGetters({ commands = baseCommands, recursive = true } = {}) {
const defaultEnhancer = _.flow(
_.partial(addCommandsToPageObject, _, commands, recursive),
_.partial(useSelectorsInContext, _, recursive)
);
const getters = {
/**
* Retrieves a requested page object or section and enhances it.
* Default enhancer adds custom commands and h4cks the object/section to use naked
* selectors in context of the object/section.
*/
getPageObject(api, path, enhance = defaultEnhancer) {
const [ pageObjectName, ...pathArray ] = _.toPath(path);
const pageObjectFactory = api.page[pageObjectName];
if (!_.isFunction(pageObjectFactory)) {
throw new Error(`Can't find a page object named "${pageObjectName}"!`);
}
const pageObject = api.page[pageObjectName]();
if (!pageObject) {
throw new Error(`Page object "${pageObjectName}" doesn't exist.`);
}
const wantedObjectOrSection = (pathArray.length > 0)
? _.get(pageObject, `section.${pathArray.join('.section.')}`)
: pageObject;
if (!wantedObjectOrSection) {
throw new Error(`Page object section at "${path}" doesn't exist!`);
}
// page objects don't have names by default; add them to help with debugging
pageObject.name = pageObject.name || pageObjectName;
return enhance(wantedObjectOrSection);
},
/**
* Retrieves page objects or sections and returns them in a plain object with last
* part of their paths used as keys.
*/
getPageObjects(api, paths, enhance = defaultEnhancer) {
return paths.reduce(
(acc, path) => {
return {
...acc,
[_.last(_.toPath(path))]: getters.getPageObject(api, path, enhance)
};
},
{}
);
}
};
return getters;
}
import _ from 'lodash';
import { registerElement } from './registerElement.js';
import { isXpathSelector } from '../isXpathSelector.js';
const NIGHTWATCH_COMMANDS = [
'clearValue',
'click',
'getAttribute',
'getCssProperty',
'getElementSize',
'getLocation',
'getLocationInView',
'getTagName',
'getText',
'getValue',
'isVisible',
'moveToElement',
'setValue',
'submitForm',
'waitForElementNotPresent',
'waitForElementNotVisible',
'waitForElementPresent',
'waitForElementVisible',
'expect.element'
];
function getLocateStrategy(selector) {
return isXpathSelector(selector) ? 'xpath' : 'css selector';
}
const appliedFlag = '__useSelectorsInContextApplied__';
function useSelectorInContext(method, pageObject) {
return function(selector, ...args) {
if (_.startsWith(selector, '@')) {
return method.call(this, selector, ...args);
}
const alias = `__${selector}`;
registerElement(pageObject, selector, getLocateStrategy(selector), alias);
return method.call(this, `@${alias}`, ...args);
};
}
/**
* Enhances all page object's methods to apply selectors the same way the pre-defined `elements`
* are used - in context of that page object (and not globally).
*/
export function useSelectorsInContext(pageObject, recursive = false) {
if (!pageObject.elements && !pageObject.section) {
throw new Error('`useSelectorsInContext` can only be used on a page object!');
}
if (pageObject[appliedFlag]) {
return pageObject;
}
NIGHTWATCH_COMMANDS.forEach(
path => _.set(pageObject, path, useSelectorInContext(_.get(pageObject, path), pageObject))
);
pageObject[appliedFlag] = true;
if (recursive) {
_.forEach(pageObject.section, section => useSelectorsInContext(section, recursive));
}
return pageObject;
}
export function usingXpath(...functions) {
functions.forEach(fn => {
this.api.useXpath();
fn();
this.api.useCss();
});
return this;
}
export function waitForAndClick(selector) {
this.waitForElementVisible(selector)
.expect.element(selector).to.be.enabled;
return this.click(selector);
}
import _ from 'lodash';
import { breakUpCommandArgs, handleCommandResult } from '../elementCommandUtils.js';
const completeCommandName = shortName => `elementId${_.upperFirst(shortName)}`;
function getElementIdRecursive(api, sections, callback, contextElementId = void 0, path = []) {
if (!sections.length) {
// recursion tail (empty sections means we've retrieved the leaf element ID)
callback(contextElementId);
return;
}
const { name, selector, locateStrategy } = sections[0];
const currentPath = [ ...path, name ];
const handleResult = ({ status, value }) => {
if (status !== 0) {
throw new Error(`Can't find element "${selector}" in "${currentPath.join(' > ')}"!`);
}
getElementIdRecursive(api, sections.slice(1), callback, value.ELEMENT, currentPath);
};
if (contextElementId) {
// continue recursion (already got context, but there are still more sections in the path)
api.elementIdElement(contextElementId, locateStrategy, selector, handleResult);
} else {
// recursion head (no ID means call from outside)
api.element(locateStrategy, selector, handleResult);
}
}
function useWebDriverElementCommand(commandName, ...args) {
const fullCommandName = completeCommandName(commandName);
if (!this.api[fullCommandName]) {
throw new Error(`Web driver protocol command "${fullCommandName}" doesn't exist!`);
}
const sections = [];
let section = this;
// collect all page object sections in the path (not the root page object, those don't have selectors)
while (section.parent) {
sections.unshift(section);
section = section.parent;
}
if (!sections.length) {
// context is a root page object (not a section) -> call the non-contextual version of the command directly
this.api[commandName](...args);
} else {
// context is a page object section -> find the section container and call the command on it
getElementIdRecursive(this.api, sections, id => this.api[fullCommandName](id, ...args));
}
return this;
}
export function elementId(...args) {
return useWebDriverElementCommand.call(this, 'element', ...args);
}
export function elementIds(...args) {
return useWebDriverElementCommand.call(this, 'elements', ...args);
}
const commandNames = [
'attribute',
'cssProperty',
'displayed',
'enabled',
'name',
'selected',
'size',
'text',
'value',
'location',
'locationInView'
].map(completeCommandName);
const [
elementAttribute,
elementCssProperty,
elementDisplayed,
elementEnabled,
elementName,
elementSelected,
elementSize,
elementText,
elementValue,
elementLocation,
elementLocationInView
] = commandNames.map(commandName =>
function(...args) {
const [ locateStrategy, selector, middleArgs, callback ] = breakUpCommandArgs(this, args);
const handleResult = handleCommandResult(selector);
return elementId.call(
this,
locateStrategy,
selector,
handleResult(({ ELEMENT }) => this.api[commandName](ELEMENT, ...middleArgs, ({ value }) => callback(value)))
);
}
);
const [
elementsAttribute,
elementsCssProperty,
elementsDisplayed,
elementsEnabled,
elementsName,
elementsSelected,
elementsSize,
elementsText,
elementsValue,
elementsLocation,
elementsLocationInView
] = commandNames.map(commandName =>
function(...args) {
const [ locateStrategy, selector, middleArgs, callback ] = breakUpCommandArgs(this, args);
const handleResult = handleCommandResult(selector);
return elementIds.call(
this,
locateStrategy,
selector,
handleResult(elementDefs => this.api.perform(done => {
const unresolved = _.range(elementDefs.length);
const values = [];
const endIfDone = () => {
if (!unresolved.length) {
callback(values);
done();
}
};
endIfDone();
const handleValue = (value, idx) => {
values[idx] = value;
unresolved.splice(unresolved.indexOf(idx), 1);
endIfDone();
};
elementDefs.forEach(
({ ELEMENT }, idx) => this.api[commandName](ELEMENT, ...middleArgs, ({ value }) => handleValue(value, idx))
);
}))
);
}
);
export {
elementAttribute,
elementsAttribute,
elementCssProperty,
elementsCssProperty,
elementDisplayed,
elementsDisplayed,
elementEnabled,
elementsEnabled,
elementName,
elementsName,
elementSelected,
elementsSelected,
elementSize,
elementsSize,
elementText,
elementsText,
elementValue,
elementsValue,
elementLocation,
elementsLocation,
elementLocationInView,
elementsLocationInView
};
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment