How We Test Our Software
We run automated tests on services exclusively in isolation because orchestrating service ecosystems for testing is complicated, expensive, brittle, and diffuses the responsibility to design and uphold durable contracts at the application boundary.
We build lightweight fakes of all collaborator services (see WebValve) into each application that run by default in local dev and test mode and provide sufficiently realistic responses via either stateless or stateful fakes to exercise the full functionality of the current application. This allows us to easily work on one app at a time, get a sense for the collaborating services’ behavior by glancing at their fake implementations when desired, and ignore tangential service calls when writing unrelated tests.
We build stateful fakes’ state into the application’s primary database because this ensures test consistency and state rollback.
We write API contract tests in the repo of an API provider, owned by the consuming app team so that the consuming team can ensure that the invariants that matter to them will hold under change.
We run CI over our pull request and deployment branch test suites and don’t merge software with failing suites because it helps ensure that new code doesn’t break things at a distance.
We run automated smoke tests of key services and integration points in production because testing environments are inherently artificial and isolation testing services, while providing strong regression protection when tests are good, isn’t a panacea.
We deploy to one-off ecosystems to validate riskier changes, test in isolation, and demo our code to potential customers. We rely on our lightweight fakes to deploy any combination of applications to a distinct ecosystem. This gives us the chance to see the effects of larger changes, like runtime upgrades or new software integrations, in a way that a feature flag won’t permit. It also helps us build confidence in platform improvements while reducing risk.