Recent Posts (page 23 / 65)

Testing And Failure

"Testing leads to failure, and failure leads to understanding." - Burt Rutan

I’m an airplane guy. As such, I’ve been aware of Burt Rutan for 40+ years. I even once helped a friend build a VariEze. Way back before there was a Mark Zuckerberg, let alone Facebook, Rutan was experimenting and failing forward. Lots of iterative development. On paper, in wind tunnels, in the air, and in space.

As impressive as SpaceShipOne was, I think Voyager was even more impressive. 9 days in flight. 26K miles. Mostly one engine. All on a single tank of gas. That’s an endurance record.

To get there took a lot of understanding of the systems involved. Of course the aircraft itself, but not just that. Also the engines. The weather. The support systems. The pilots. The question is, how do you get that understanding? You get it from testing. And fixing the problems. But not just fixing the specific problem, but deeply understanding what the problem was and making sure there weren’t similar problems later.

You can (should) approach software development the same way. You know what you want to happen when everything goes right. And you write things so you get the result you want. And you do your best to consider other possible situations. And make sure that they are at least handled gracefully, even if you can’t do what you originally planned.

Then you start testing, You find the edge of the envelope. More load, more traffic, less time to respond, strange data, whatever your inputs and environments are. And something breaks. So of course, you fix it. But you don’t just fix it. You figure out why you got into that situation and how to make sure you never end up there again. Which means not just understanding the thing you’re writing, but the environment it operates in. And the people and systems upstream that contribute to the environment. And the people and systems downstream who’s environment you are a part of.

Because without that understanding all you’re doing is going fast and breaking things. You’re not actually getting anything done

Rocket Science

Almost 120 years ago Konstantin Tsiolkovsky published his rocket equation:

That’s rocket science. Really. That’s all there is to it. One equation. For any given rocket the change in velocity (speed in a direction) is equal to the rocket exhaust velocity multiplied by the natural log of the ratio of the initial mass to the final mass.

Sure there are a lot of nuances, and any good physics teacher will tell you at the end of class that it isn’t quite as simple as it’s made out to be, but to a first (and probably second and third) approximation, that’s all there is to the science behind rockets.

Which is kind of funny actually. Because when we want to say something isn’t that hard we say that it’s not rocket science. A person needs a certain amount of food, water, and oxygen each day. We know what those numbers are. With that, the relative motions of the Earth and the Moon, and the rocket equation we can figure out just what we need to build to get from one to the other and back on a certain schedule. It’s just math. And not an integral in sight.

No, it’s not the science that’s hard. There are lots of things that are harder than rocket science. Like rocket technology. Just because you know how much fuel you need and how fast you need to throw it overboard doesn't mean you can build the rocket. You need the right materials. You need the right mechanical systems. The right electrical systems. All kinds of implementations of things.

And to make things harder, they’re all interconnected. The more fuel, the heavier your rocket needs to be. Which means you need more fuel. At higher pressures. And the trip is longer. So you need bigger batteries. Or lower power electronics. And so on.

And as complicated as that is, there’s an even harder problem to solve. How do you get the entire system to work together? It takes 1000’s of people years to design, build, test, and fly one of those rockets. As impressive as a Saturn V rocket is (and if you’ve ever been close to one, they are impressive), the management of the process is even more impressive. Getting that many people to work on more or less the right things, at more or less the right time, in more or less the right order. Repeatably.

Software is like that too. Computers are inherently simple. A logic gate is open or closed. 0 or 1. That’s it. Throw in some simple boolean logic. AND, OR, XOR, and NOT. Add a few simple instructions. ADD, SUB, LOAD, and STORE. It’s only when you have billions of them working together, doing all the (relatively) simple things that all the different people have gotten them to do. At the right time. In the right order. With the right feedback loops. That’s when you get a database, a search engine, or a control system that can get a rocket from the Earth to the Moon.

So the next time someone tells you that what you’re doing isn’t rocket science and shouldn’t be that hard, remind them that it’s not the science part that’s hard. It’s the engineering, the turning of science into reality, and doing it with others that’s hard.

Global Data(base)

A database is a giant global variable that pervades your code. How, exactly, is that a good thing?

  -- Allen Holub

Well. That’s a pretty bold statement. Let’s break it down and see how true it is.

A database is certainly a “global” variable. Or at least it is in the sense that any code with the correct permissions/connection string can connect to it and read/write data. That does make it global. Can’t really argue with that.

Especially if you only have one database. With one (widely) shared secret that lots of different things use to connect to that database. To read/write from the same table with a magic set of filters. Or lots of different tables that all have the same unrestricted permissions.

In that case your dB is a SPOF. SPOFs are almost never a good thing. SPOFs are rarely completely unavoidable, but can be more expensive (however you define that) to eliminate the SPOF than it’s worth to avoid, so you live with them, even if it’s not a good thing. so that part is true too.

Which leaves us with the middle part, pervades your code. That’s an interesting one. And about as subjective as you want to make it. Liberally sprinkling your codebase with code (not function calls) that opens a connection to a database, reads/writes some data, then closes the connection and moves on is almost certainly a bad thing. Can’t really argue with that.

So databases are bad and we should just stop using them, right? Of course not.

It’s about using the right tool for the job, and using the right way. If you have one service/tool that needs to deal with a few hundred kilobytes of inter-related data the right tool is probably a clear data model with an API designed for how the data is actually used. Behind that data model have an in-memory set of maps and a little bit of logic to connect them. If you need to get really fancy add some kind of write-through mode so any written changes get persisted for next time you start up. No (formal) database involved.

What if a bunch of things need access to that data? Put it in a (micro?) service. Same API, but now accessible from lots of things at the same time, regardless of what process they’re in. Still no formal dB involved.

18 months later and things are going so well that you’ve got 100s of GB of active interrelated data. Read/Write rates are up. You can’t afford to lose changes, and the ability to undo a set of changes as a unit is now important. So you start to build those changes in. Then you get smart, and instead of reinventing the wheel you put your data in a traditional database.

So was Allen right? It depends. Did you need a 256 core MS-SQL database with 512 GB of RAM and 2 PB of SSD to start out with? Of course not. You didn’t even need a simple AWS RDS. So it was correct.

On the other hand, a database is just a collection of data you access via computer, so pretty much everything is a database. So in the most fundamental sense, he was completely wrong. You already have a database, because storing and accessing data is what programs do.

Like everything though, context matters. It was a tweet, it was written for retweetability and impact. As a conversation starter it was effective. So next time you go to create a new AWS RDS, think about if you need that much database yet. You still need to do all the other hard parts of design, but do you need that bit of complexity? I can’t tell you for sure, but I do know it’s worth thinking about.

Lone Wolf - Hard Nope

Sometimes I read something and wonder if I’m living in the same universe as the author. Then I think about it some more and realize I’m not. Consider this article about How a Single Freelancer Outperforms a Whole Team. At first glance it makes sense. From the Mythical Man-Month to convoy speeds, a single person can be shown to be individually efficient than a team.

But, as I have been known to say, “What are you really trying to do here?” Are you trying to ensure that every individual is maximally busy? Rack up as many billable hours as possible? Ship something and never touch it again? Have the ability to chase after the “fad of the moment”? In that case, go ahead and lone-wolf it.

In my world, on the other hand, the goal is to add value, over time, in support of a larger goal. Those are very different goals. Which leads to two different answers. One optimized for the one wolf, one for the customer.

Yes, a single ship is (generally) faster than a convoy. A single rider is (generally) faster than a group. But consider the whole environment. In WWII convoys lost about 4% of their tonnage, while ships travelling outside convoys lost 20%. Overall there’s no question which method got more cargo safely across the Atlantic. For any given trip individual ships were less likely to be detected, but over multiple trips that changed significantly. It’s the same for developers. On any given task one experienced person is (generally) faster than a group. But over multiple projects, especially when you add in support/keeping the lights on, the overhead work will more than eliminate any gains.

The other, and more significant, way that article lives in a different universe than I do is around uncertainty. When you’re a contractor you either bid a fixed price/fixed result contract, or you go with a time and materials bid. In the first case you decide what you’re going to do up front, whether you understand the problem or not, then you do that thing. After you’re done you leave. Or get another contract to redo it based on what you learned. In the second, there’s no incentive to get it right. In fact there’s an incentive to keep going as long as possible. Either way, the lone-wolf contractor doesn’t care about the uncertainty.

I, on the other hand, care a lot about that uncertainty. In fact, dealing with that uncertainty and figuring out how to turn ambiguity into value is a big part of the job. To go with that responsibility is the autonomy to make the choices that add the most value. In conjunction with other people going through the same process. 

Which leads to my biggest issue with that article. It implies/assumes that nothing anyone else does is impacted by the work the lone wolf does. That the lone wolf is a single point of failure during development and forever after as support. Even more critically, it assumes that nothing anyone else does has any impact on what the lone wolf does, or should be doing. 

And that’s not the world I live in. Or want to live in.

Design By Example

Yesterday I talked about Design by Counter-Example. What about the opposite? You know what you want to do (the user story), but you don’t know how to do it, or what the API and internals should look like. How might that work?

You’d start with a real world problem. A user story that describes a way to make the user’s world better. Better by adding some value. It might be by eliminating manual work, reducing the number of steps, or some whole new way of approaching the problem that solves a user’s problem.

You’d work with the user, discussing options and priorities. You might go to the whiteboard and draw boxes and arrows. Maybe use Visio or Google Draw to make a shareable version. Whatever it takes to understand the domain and how you’re adding value.

Then you think about it a little. And come up with some pseudo-code that shows how you envision adding value to the user. It doesn’t work of course, because none of it really exists, but you get some more feedback from the user and can feel pretty confident that it really will add value.

Finally (really, it’s only been hours, maybe a day or so), you go to your favorite IDE and turn that pseudo-code into real code, but still making calls to non-existent functions and APIs. So you start to fill in those gaps.

You start simple. At first you just blindly return some happy path answers. Things check out, so you continue. Then maybe you add some input and state validation. You check with your user and make sure you’re still adding value. You’ve learned a little, so maybe you tweak the example code a little.

You keep adding depth and detail until all of the examples work and you release the new benefit to the user and they’re happy.

That’s test driven development design by example.

Design By Counter-Example

Every once in a while I come across a coworker who knows exactly what they don’t want. Whether it’s a design doc, slide deck, email, error message, icon, or whatever you’ve come up with isn’t it. They could tell you that you were close, but it's not what they're looking for. The level of detail wasn't right. or the constraints were at the wrong level. That the color or saturation was off. They could be very clear that there was a problem.

Unfortunately, they weren’t nearly as clear about the converse. They didn't know what the problem was. Now I understand that not everyone can describe colors, write the perfect error message, or come up with the exact sound bite that everyone will remember for years, but knowing that didn’t make my job easier. I wanted the right feedback at the right time. Unfortunately, all I got was “That’s not what I’m looking for.”

Which left me in the position of needing to do a binary search for the right answer. Pick a solution. Get mostly non-directional feedback, with just the tiniest amount of direction. Then I’d take a medium sized step in that direction. And get the same kind of feedback. And then take step N+1. Either further in the direction I was going, or half-way back to step N-1. Lather. Rinse. Repeat. Until I got close enough to the solution, or we both got tired of trying and accepted where it was.

While it worked, after a fashion, it wasn’t efficient. Or fun. Or scaleable. So how can we do better? Even in the case where we don’t know the right answer. When you have to creep up on the right answer because you really don’t know where it is yourself, so you can’t describe it to anyone. You just know what it isn’t.

First, and foremost, when giving feedback, it’s ok to start with what needs to be changed. But instead of stopping at “That’s not it”, continue on with what it is like, or more like. And add why you want the change. Instead of saying “There are too many words on the slide” and leaving it at that, add the goal. Something like “I want the audience to be able to focus on the one important thing on the slide and not be distracted.” Now that’s the kind of actionable feedback I like.

Which, when you think about it, is the kind of goal/value oriented management you want in general.

Complex Vs. Complicated

According to dictionary.com, complex is:

adjective

• composed of many interconnected parts; compound; composite:
  a complex highway system.
• characterized by a very complicated or involved arrangement of parts, units, etc.:
  complex machinery.
• so complicated or intricate as to be hard to understand or deal with:
  a complex problem.

While  complicated is:

adjective

• composed of elaborately interconnected parts; complex:
  complicated apparatus for measuring brain functions.
• difficult to analyze, understand, explain, etc.:
  a complicated problem.

Seems pretty similar, right? In fact, 2 of the 3 definitions of complex include the word complicated. And the thesaurus lists them as synonyms for each other. So they’re interchangeable, right?

Sometimes, maybe even often, but not always. One of the biggest areas where they’re not interchangeable is in mapping management processes to use cases. And that’s where the Cynefin framework comes in.

As you can see right there in the chart, the top two use cases are Complex and Complicated. Because how you manage them is very different.

Building a car is complicated. But building a car doesn’t have a lot of trade-offs and decision making (that’s designing a car). Lots and lots of moving parts (literally). And critical paths. Deep dependency chains that might not be obvious at first glance. Getting them out of order means lots of rework and delays. Because we’ve done it so many times and the interdependencies (not dependencies) between steps is very minimal, you can write down the steps, in a specified order, and expect that the process will pretty much follow those steps.

On the other hand, Software development is often complex. It’s the addition of the first part of the definition, the interconnectedness that makes it complex. That fact that it’s recursive. And that you don’t have full knowledge going in. Your choice of domains informs the design of the domain’s API and the act of building that API informs your choice of domains, which changes your API.

Or, working with your customer on what they’re really trying to do, you build a ubiquitous language to describe the work and the expected value. Then you start to implement things based on that understanding. But your language is not perfect, so you learn more about what they really mean as you work. So what you’re building changes. Which clarifies the langage. Which changes what you’re building some more.

So the key is pick the right tool for the job. Using the wrong strategy isn't just sub-optimal, it can make things worse.

And to make software development more challenging, different parts of the solution you’re building land in different quadrants, so it’s not one size fits all. You need to find the right strategy for the different parts of the job and make them work together.

More Than Cruft

I’ve talked about Tech Debt before. The idea that, like with earnings, you can trade future work and understanding for earlier delivery of a product, then pay it back later. It made sense when I first heard about it, and it makes sense now.

What’s important though, is to do both parts. Not just pushing work or learning into the future to speed up delivery. But what does that really mean? It means being an owner and thinking not just of delivery dates, but of future developers, which includes you. You’re not just making life harder for some faceless future person, you’re making life harder for yourself. So it’s in your own best interest to do it knowingly.

It means doing things with the best information and understanding you have at the time, then, as you learn, do the important things, the critical things, that you put off. Making sure the code reflects your current understanding, at all times. You made the choice and bought the time. Maybe it was for a demo or a holiday ship date, but you made the choice. Now live up to the promise you made to yourself and your customers. Take the thing you got by releasing sooner, the capital (feedback, experience, brand recognition, investment, whatever) and put it to use. Update the user interface after watching your customers use it. Refactor your domains now that you know what they really are. Add error handling and the operational domain outside the narrow happy path you just implemented.

While I’ve known Ward Cunningham came up with the idea and I recently came across this description of the situation where he first started using the term. At the time he was using Smalltalk and talking about objects and hierarchies. We don’t use Smalltalk, and where he talks about objects he’s talking about domains and domain driven design, not objects as defined by a particular language. And doing things that way, keeping your domains clear and bounded, is what keeps your debt from burying you. It’s what allows you to keep the promise you made to yourself when you took on the debt in the first place. It’s about keeping things clean enough so that you can be agile enough to use the learnings you took on the debt for in the first place.

And if that isn’t clear enough, here’s Ward himself talking about it and a transcript if you prefer.

Blink Estimation

Speaking of Tacit Knowledge, have you heard of Blink Estimation? Take a medium sized group of people with significant tacit knowledge in a domain. Explain a problem/opportunity and give them some time to ask questions and explore the shape of the problem. Then, ask them for a rough order of magnitude of the level of effort required to implement the solution. If your group comes up with similar answers then you’ve got your estimate. If they don’t, work with the outliers to understand why their answers were so different. At least one group missed something, possibly both.

It’s surprising how often this works. There are lots of caveats and biases to watch out for, but if you can manage those you can get a good answer. The biggest of those caveats is that tacit knowledge can be incredibly domain specific. Which means the folks you ask need to be well versed in something close enough to what you’re asking them about.

And not just on the surface. All software development is not the same. Asking someone with deep tacit knowledge about IOS apps for an estimate on distributed parallel processing is unlikely to give you a good answer. Both problems involve computers, but they're not the same. Consider bicycles and unicycles. In the US somewhere around 6% of people say they don’t know how to ride a bike. Unicycles are half the complexity (1 wheel vs 2), so even fewer people don’t know how to ride a unicycle, right? Of course not. On the surface they’re the same, but in reality they’re mostly unrelated.

Then there are the cognitive biases. Not surprisingly, if the experts have been working on the project and know it was supposed to take a year and they’re 6 months in, that fact will color their answer. So will how long it took them to finish the last task they worked on. Just mentioning an expected duration will bias the results. Even a bad commute into the office that morning can bias the results.

And it won’t work if the problem space is new. We can estimate how long it will take to build a single family ranch house in the suburbs because we’ve got thousands (millions) of examples to look at and get that tacit knowledge from. Building the first habitat on Mars, from local material, will not be nearly as predictable.

How can such a shallow evaluation be better than a deeper one? Of course having the right tacit knowledge is key. But there’s more to it than that. Here’s a question for you. How long is the coast of England? It depends on how small you break the problem down. I suppose if you get down to plank units there’s a limit, but the closer you look the longer the coastline gets. Software can be like that too. The deeper you design, the more work you find. And of course, no task takes less than a day, so the estimate blooms the harder you look. Which seems counterintuitive, but is reality.

We’re doing lots of new things, so it may not be very applicable, but just for grins, next time you start a project, get a few folks together, think deeply about the size and shape of the problem, (but not too deeply) and make an estimate. Then see how close you come. Do it a few times. Keep track of the results. Regardless of the outcome it will likely make you blink.

Tacit Knowledge

When I first heard about tacit knowledge and started looking into it, I thought it was the same thing as tribal knowledge. You know, those things that aren’t written down, never get explained, and you just need to know to be successful. Turns out that’s not the case. On the surface they seem similar in that in both cases you aren’t told the knowledge you’re looking for. The difference is that tacit knowledge is the kind of knowledge that can’t be explained with words (at least in an amount of time that is useful), not just something that hasn’t been explained.

One classic example is riding a bicycle. Think about it. How would you explain to someone how to ride a bike? You could tell them all about the physics of flywheels and precession, but understanding the physics won’t teach you the muscle and balance feedback you need. Comparing it to standing on one foot or slack-lining doesn’t really help either, but those are comparisons, not explanations or instructions.

The next thing I thought of was that ok, this is just experiential learning. You know, learn by doing instead of learning by rote by watching, but again, not really. I’ll be the first to admit that I’m an experiential learner. I learn best by trying and watching what happens when I interact with something. But I can learn by listening to others, or just watching them do it. Learning styles are a property of the learner, not the subject.

So what is tacit knowledge? To me, it’s the knowledge you get by doing something, then having to live with the results. In software it’s the kind of knowledge that leads to a feeling that something is the right (or wrong) design. A code or design smell. You look at it and say “that’s not right.” Then you think about it and you come up with some reasons. But the reasons come after the feeling and justify it, not precede it. And that’s what makes teaching and learning tacit knowledge so hard.

Consider the expert system (really a knowledge based system). Find a few experts, look at a bunch of their decisions, and ask them why. Keep asking why. Build a sufficiently complex tree that you can use to arrive at the same conclusions. Sounds simple. People tried it and it’s not. Take a simple case. Is that function too long? It should fit on a page. It should encapsulate a single action. It should be replaceable without changing the results. It shouldn’t have too many branches. Unless it's a complicated topic. Unless it needs that many steps. Unless the combination of possible conditions is that large. Unless there are no other functions. Unless breaking it up makes it harder to reason about.

So how do you learn tacit knowledge? It’s not just repetition. Repeating the same thing over and over again will make you faster at it, but not better, and how do you know if you’re correct anyway? So you do it and look at the results. Repetition and introspection. That helps. Now you know what you did, and you can hopefully see the flaws, but that doesn’t help you get better.

For that you need to add something else. Feedback. The completion of repetition with introspection and feedback is one of the best ways to build tacit knowledge. So how do you get that in software? Of course you have to do it. You need to write the code, or do the design. Over and over again, That’s the repetition.

You need to know if it works. You test what you’ve done, and see the results. And you change things until you get the answers you think you want. That’s the introspection.

And finally, you need to share what you’ve done and get feedback, It could be a diff/PR. It could be a design doc. It could be an architectural kata. Whatever the mechanism, you need that external feedback to make sure your biases, conscious and unconscious, don’t get in the way.