Empirical Software Engineering

What is software engineering? What is important to software engineering? What do we know about software engineering? How do we know what we know about software engineering?

Perhaps more importantly, what do we know we don’t know about software engineering? According to Hillel Wayne, 

Empirical Software Engineering is the study of what actually works in programming. Instead of trusting our instincts we collect data, run studies, and peer-review our results.

Seems reasonable. But what does it mean? Are shorter functions better or worse? Is there a language that is better to use than some other language? Is a microservice architecture better than an event driven one? For that matter, what does better mean?

Like many things, the answer is, it depends. It depends on context. It depends on the problem space. It depends on the constraints you have to deal with. There’s a great scene in Apollo 13 where they need to “make this, fit into the hole for this, using nothing but that”. Those are constraints. Doesn’t matter what the perfect answer might be. That’s the best answer, right now.

And in Hillel’s talk, he talks about the definition of better a lot.  And not just what better means, but how do you know if you’re right, and if the cause you’re positing is really the cause? And qualitative vs. quantitative research. And the validity of the finding itself.

I’ll let you all watch the talk yourself for the details, but here’s a few of the biggest takeaways.

The best metric for the number of bugs in a code base is the number of lines of code. Pretty obvious. But also not very helpful. It’s not very actionable.

Org structure has a big impact. The higher number of different people/teams/groups adding changing code in a module, the higher the number of bugs. But it’s a balance. Silos and gatekeepers may have a small positive impact on the number of bugs, but they have a large impact on velocity.

Finally, the things that are most impactful to development are Stress and Sleep. Missing a little bit of sleep on one day makes you less productive. Missing a few hours a night is like missing a whole night’s sleep. And even worse, along with that reduction in productivity and effectiveness, comes an inability to notice that reduction. Which means, not only do you make mistakes, you can’t tell you’re making mistakes. And you can’t do anything about them.

So whatever else you do, make sure you get enough sleep.

And on a mostly unrelated note, take a look at the slide deck that goes with the talk. That’s a really interesting way to do a talk. Very minimalist. So you focus on the presenter. Which is often a good thing. But if you consider a presentation as two channels of information (spoken and written), it basically leaves the second channel unused. But that’s a topic for another day.