- Collects points from several sources, like Kent Dodds and DHH
- More focus on integration tests as a sweet spot of confidence/coverage and time
- Avoid architecture and maintenance overhead of unit tests
David Heinemeier Hansson, RailsConf 2014
- Committed to building information systems
- Doesn't align with computer "science"
- Reading code is like reading old French poetry
- More like "computer pseudoscience", and diet fads
- TDD is the most popular software diet
- "Test at a courser level of granularity", Coplien
- Unit-test-driven design destroys system architecure
- System tests are more meaningful
- Aim for regression tests that allow change w/o breakage
- Kent Beck: test as little as possible to reach confidence
- Embrace software "writing" instead of "engineering"
- Engineering build son hard sciences and metrics
- Writing focuses on clarity
- Develop an eye through practice, similar to any creative effort
- Rewrite drafts for clarity (aka refactor), backed by tests
- "Delete" key is the most powerful tool
Google Testing Blog
- E2E tests are appealing as user simulation
- But frequently brittle, hard to find root cause
- To evaluate any testing strategy, evaluate how it enables developers to fix and prevent bugs
- Feedback loop: fast, reliable, isolates failures
- Unit tests: a small piece of the product in isolation
- E2E tests: entire built product; hard to find root cause
- Integration tests: behavior of small group of units as a whole
- 70% unit tests, 20% integration tests, and 10% end-to-end tests
- Avoid inverted pyramid, hourglass
- Test-driven development != Test-first development
- "Units" of functionality in isolation
- "Integration" of units
- Fully-integrated "system" for "end-to-end" "acceptance"
- Avoid UI complexity with "subcutaneous" API
- Test early and often, w/ a short feedback loop
- Keep tests fast, readable, and maintainable
- Write failing tests for bugs
- Test one behavior per test, instead of a specific API
- Test one thing: view, model, form
- Avoid complicated utilities and DRY
- Test for graceful failure
- Tests enable smoother upgrades
- Watch for incremental coverage changes, instead of aiming for 100%
- Code Coverage Done Right | Codecov
James Bennett, Django core team
- Use
unittest
, because that's what Django uses, and has less magic thanpytest
- "Standalone" mode for reusable apps
- Use
tox
- Write meaningful tests first, then use coverage to detect changes
PYTHONWARNINGS=module::DeprecationWarning
- See https://github.com/ubernostrum/pwned-passwords-django for example
PyCon 2017
- Only humans can test for correctness (but automation helps)
- Automated tests prevent change, for better or worse
- Define goals and situation to determine testing strategy
Sandi Metz, Rails Conf 2013
- We hate our tests because they're slow, fragile, and expensive
- Unit tests should be thorough, stable, fast, and few
- Think in terms of objects and messages
- Query: Return something, change nothing
- Command: Return nothing, change something
- Test every part of an object's interface once
- Trust an object's collaborators
- Mocks must stay in sync with real API; use libraries to do this automagically
Message | Query | Command |
---|---|---|
Incoming | Assert result | Assert direct public side effects |
Self | Ignore | Ignore |
Outgoing | Ignore | Expect to send |
Justin Crown, PyCon 2018
- Slides
- No compelling reason not to
- Test what you touch: start with current, then desired, behavior
- Favor functional/integration tests
- Change code cautiously
- Okay to fix symptoms instead of root cause
- Write tests to take a break, refresh
- Mock out network with pytest-socket, etc.
- TODO: Strategy for new features
Andrew Knight, PyCon 2018
- https://github.com/behave/behave for Behavior-Driven Development, Acceptance Testing
- Start writing features
- Screenplay pattern (vs. Page object) for web automation
- Aspect-oriented programming
- BDD | Automation Panda
- Python Testing 101: pytest-bdd | Automation Panda
Andrew Knight, PyGotham 2018
- Review slides
- TODO: Screenplay pattern
- Consider ROI of automation; sometimes manual is okay (but still spec)
- reportportal.io
- BDD encourages descriptive, reusable steps
- BDD helps separate test cases from test code
Gary Bernhardt, PyCon 2012
- Tests should prevent regression, enable refactoring, and improve design
- System tests == Functional/Integration/Not-Unit tests
- System tests don't isolate errors, frequently break multiple at a time
- System tests prevent regression, but don't enable refactoring or improve design, because they're slow
- Use system tests for boundaries, unit tests for behavior
- Aim for 90/10% unit/system tests for object/logic-heavy web apps w/ fewer boundaries
- Try writing tests from the ground up, by starting with
assert
- Avoid negative assertions, e.g.
assert 'foo' not in bar
- Fail at testing w/ lots of Selenium, big "unit" tests, unit tests for legacy code
- Unit testing legacy code solidifies it, makes it hard to refactor
- See "Working Effectively with Legacy Code" for decomposition guidance
- Consider using service classes for model behaviors, instead of "fat models", to decouple from database
Augie Fackler & Nathaniel Manista, PyCon 2012
- System built of stateless modular components
- State managed via messages, persisted to database
- Started w/ poor coverage and manual verification
- Added tests for new/changed code, but easily bitten by untested code
- Lots of devs w/ duplicate mocks that got out of sync w/ API
- Use one well-tested mock (if any)
- Use inspect (or autospec?) to validate mock signatures
- Use system tests for gaps in unit tests
- Use system tests for user stories, not corner cases
- Aim for a collection of authoritative, narrow, isolated fakes
- No "has been called X times with Y"
- TODO: Clarify distinction between fakes, mocks, and stubs
- Use mocks for services, not business logic
- Test interface, not implementation
- Design for love and tests
- Use explicit dependency injections (instead of defaults)
- Separate storage from behavior using free functions
- Don't write unit tests coupled to complex implementations
- TODO: Tell, don't ask
- Consider ABC's/interfaces for mocks
- Write unit tests for behavior and pure functions, not "assembly" code
- Prefer fakes, not mocks, for code in your control
SF Django, Jan 2019
- Slides
- Example repo w/ "real" app
- Nice before/after of Django's
TestCase
vs. pytest - Good overview of non-TDD process
- Interesting plugins:
- pytest-dev/pytest-cov: Coverage plugin for pytest.
- pytest-dev/pytest-mock: Thin-wrapper around the mock package for easier use with py.test
- getsentry/responses: A utility for mocking out the Python Requests library.
- getsentry/pytest-responses: py.test integration for responses
- pytest-dev/pytest-selenium: Plugin for running Selenium with pytest
- spulec/freezegun: Let your Python tests travel through time