On Testing

My title is Senior Software Engineer II. I work on Infrastructure. No mention of customers or testing in there, but I'm a tester. I write and execute tests to make sure my customers always get what they expect, and if something goes wrong *I* know before they do. It wasn't always that way.

Back in the day, there were different roles, and they kept to themselves. PMs talked to business leaders and customers and published the requirements. Devs wrote some code, hopefully related to the requirements, and tossed them over the wall to the testers, who tried to run it based on their understanding of the requirements, and told the developers it was broken. Lather, Rinse, Repeat. Eventually something got sent to customers, and then the patching, which went through the same cycle, began.

Now, not so much. There are lots of PM types, the walls between roles are much more porous, and the test org is largely gone. No STEs (Software Test Engineers), no SDETs (Software Development Engineer in Test), no SEs (Sustained Engineering). Instead we have lots of different kinds of tests and times/places to run them. There are Dev Tests, Unit tests, Functional Tests, Integration Tests, UI Tests, Simulations, Staging Environments, Black Box Tests, Chaos Monkeys, A/B experiments, and HealthChecks. So how do you know which ones to use and when?

That depends on where you are in the product life cycle and what you're trying to accomplish. Unit tests are great for making sure things do the right things when the inputs are defined and give the proper error when expected problems occur. They should be written early in the cycle and executed all the time to make sure unexpected changes don't sneak in. And of course, they should be completely deterministic, carrying all the data they need and not relying on any outside functionality.

At the other end of the cycle is black box testing and health checks. Assuming your system is deployed and running, use it like a customer would and make sure you get the correct responses. If you do this kind of testing regularly you stand a good chance of finding out there's a problem at the same time your customers do, possibly even sooner.

In between are all kinds of tests that focus on different things. Chaos monkeys for dealing with random problems. Integration tests to make sure your assumptions about another system are valid. Staging environments for medium scale testing. A/B experiments for large scale testing. 

Martin Fowler has a whole page of articles about testing and when and how to use them.