Recent Posts (page 19 / 67)

OODA and Systems Thinking

Lately I’ve been thinking about systems thinking a lot. The difference between complicated systems and complex systems. How you approach them and how you work with them. You can control a complicated system. It might take a while to find the right lever to pull or knob to turn, but once you figure it out you can control it pretty tightly with a feedback loop. Because a complicated system’s responses are (generally) linear and predictable.

Complex systems are different. You work with a complex system. You don’t control it. You can characterize it’s responses to small changes in constrained environments, but they’re hard to generalize about. Because complex systems have feedback loops in them already. And those feedback loops aren’t linear. They can be exponential. They have inflection points. They can change direction. They’re dependent not only on the initial conditions, and the change applied, but also on the rate of change.

Or to put it more concretely, with a complicated system, you know how the system will react to a change, and a person with lots of experience with the system has a high likelihood of knowing what to change,and how much to change it, to get a desired response. On the other hand, with a complicated system, having lots of experience with a given set of conditions and making small changes doesn’t give you high confidence about what will happen if you make large changes, nor does it tell you how small changes will impact the system if you start from a different state.

A complicated system, no matter how Rube Goldberg it is, can be boiled down to a series of runbooks, checklists, or decision trees. If this, then that. After A, always B. When the next step doesn’t happen you know exactly where to look to find the problem, and the solution is usually obvious. Executing the fix might be hard, but knowing what it is, not so much.

For complex systems it’s different. You might have an idea about how the system will respond to a change, but unless you’ve seen the exact same situation before then you don’t really know what will happen. That’s where the OODA loop comes in. I first learned about the OODA loop from fighter pilots back in my flight simulator days, but it really applies to any situation where you can’t predict exactly what the response to an input will be. 

OODA is short for Observe, Orient, Decide, and Act.

But that simple loop just glosses over the critical details of the OODA loop. In its full form the OODA loop is a complex system in of itself, 

That’s because it takes a complex system to deal with a complex system. OODA is all about interacting with the environment, understanding how things are changing, due to its own inherent tendencies and reacting to your actions, and then modifying those actions to drive the system to the state you’re looking for.

Luckily, while OODA is a complex system, it’s really not that complicated of a system. While it has lots of feedback and feedforward loops, the process itself is straightforward. Where it gets hard is in the observe and orient steps. It takes effort and intellectual honesty to avoid biases. To see things as the are and as they are evolving, not how we want or expect them to.

So next time you run into a complex system that you’re trying to manage, Observe, Orient, Decide, and Act. And repeat as necessary.

Scrape To Taste

our system of make-and-inspect, which if applied to making toast would be expressed: “You burn, I’ll scrape.”

-- W. Edwards Deming

After 2 weeks at sea, just a quick thought on testing and quality. You can’t test quality into a product or system. I think testing is critical. Unit tests, integration tests, exploratory tests, visual tests, and sometimes, fun-ness tests. But as important as they are, all you can do with after the fact testing is make sure that you don’t inflict known defects on your users. That’s critical, and we need to make sure we do enough of it, but we can’t rely on it.

I’m not sure when Deming first said it, but I’ve seen it quoted as far back as 1991. If you want consistent quality output you need to design the system to produce it, not just throw out things that don’t meet the requirements. If you want to consistently and quickly deliver a quality product you need a culture that values that quality. Otherwise you end up moving faster and delivering value later.

Better Code

Ran into a really interesting video the other day by Woody Zuill (of Mob Programming) and Llewellyn Falco. Lots of good things to learn there. I’ll leave it to you to watch for yourself. It’s the meta point that I find really interesting.

The meta point is that you don’t need to understand code to refactor it, and that by refactoring you gain understanding. Then, with understanding, it’s easier, and overall faster, to do the thing you want to do.

I comes down to what Martin Fowler said many years ago:

“Any fool can write code that a computer can understand. Good programmers write code that humans can understand.”

We’ve all been there. We run into code, often code we wrote ourselves, that for any number of good reasons, like lack of understanding, shifting requirements, or tight deadlines, that isn’t as understandable in retrospect as we thought it was when it was written. That’s when you’ve got a choice.

You can beat the code into submission or you can use all of the tools at your disposal to make the code understandable and then make the change. It might make things take a little longer, but probably not. It will make it easier to move forward next time. Because that’s what we’re really trying to do. Reach the finish line sooner, not go as fast as possible right now.

As we head into the holiday weekend, I’m thankful for all of you who put in that little bit of extra effort in the past or will do it in the future to help us get to the finish line (and make my life easier) that much sooner.

Happy Turkey Day and see you all in 2.5 weeks.

Walking Through The Code

I’ve been thinking about code quality a bunch lately and the kinds of metrics that exist for how “clean” code is. Things like cyclomatic complexity, maintainability index, nesting depth, method length, and Halstead volume. They all provide some info, but they’ve also got issues. While the source code is the ultimate source of truth for what is, it’s by no means the full story. It can’t tell you what problem it’s solving, why it is the way it is, why it isn’t some other way, or what the foreseen but not yet handled problems are. And it’s particularly bad at helping someone understand the gestalt of it.

That’s where documentation comes in. The typical README.md can be good for the why’s. Why is it this way? Why isn’t it that way? Why are the domains split the way they are. There’s also usually some background on the problem being solved. Maybe even something about how the component can or should fit into the bigger picture. These are all great things and we should write them down for all of our components.

One thing I rarely see though, and wish I saw more of, is a walkthrough. How are control and data flow embodied by the actual code. Combining the details of class definition and control logic with the high level design by following a specific call or command from the user’s request until the work is done. You’ve done this as part of debugging. Before you can understand what went wrong with a bug, you had to step through the code to understand how the happy path worked. Only then can you try to figure out where things went wrong and do something about it.

You’ve certainly done this by yourself, and probably sometimes with someone more familiar with the code. It’s almost always easier to find your way through the code with a guide. The guide can help you stay on the main path and point out interesting and relevant things nearby along the way. They can provide context along the way that will help you understand the specific situation you’re looking in. The problem is, there might not be a guide available when you need one.

That’s where the walkthrough comes in. It’s not just a list of facts and details. Just like a museum self-guided tour, the walkthrough gives you an overview and context but lets you dig in to the thing that is most important to you. Which makes things easier for you the maintainer. Even if you were the one who wrote the walkthrough in the first place.

Ignorance As A Service

Extreme Programming has been around for a while. It brought with it the concept of pair programming. 2 (or more) developers working together on one keyboard to complete a single task. Sounds crazy, no? In practice it turns out not to be. Which leads to the next question. How long should a pair program together?

I came across this paper from 2005 that describes one company’s results from experimenting with how much time there should be between pair changes. Very surprisingly, their results showed that the best results came with 90 minute pairing sessions. The authors go into some detail about why that is. One of the biggest factors they found that made that important was that there was always one member of the pair who was not deeply familiar with the code being written.

That seems counter-intuitive. Making sure that part of the pair doesn’t know what’s going on makes it better? What they concluded was that it was because of the benefit of the Beginner’s Mind. Not just learning about something, but cultivating an active curiosity and openness to new things. This leads to fast and deep knowledge transfer. It also leads to new insights. Because the beginner comes without fixed/preconceived ideas, they ask questions that can lead to new avenues of thought.

But it goes beyond that. Before an “expert” can explain something, they need to understand it. If you don’t understand something you can describe it. What it is. How it responds to known inputs. Where it’s known to fall over. But to explain it, you need to deeply understand it. It’s that deep understanding that leads to clear explanations. More importantly, it also leads to simplification. To removing the cruft and noise that gets in the way of doing the one thing well and truly. To building a system that is easy to use, hard to misuse, and powerful enough to meet today’s and tomorrow’s needs.

Something to think about. And if you need to apply a beginner’s mind to your problem feel free to ask.

More WAT?

Avoiding the Wat is important. It’s especially important when you’re designing APIs. APIs are for life. People will be using them long after you’ve stopped thinking about them and using them in ways you didn’t originally intend. When it comes to language design the stakes are even higher.

Consider the JavasScript language. It’s pretty widely used, so choices in the language design there are pretty impactful. JS has a lot of built in functionality. Among them are the in operator and the includes() method for arrays.

According to Mozilla, the in operator returns true if the specified property is in the specified object or its prototype chain, otherwise false, and the includes() method determines whether an array includes a certain value among its entries, returning true or false as appropriate. Both of those seem completely reasonable by themselves. The Wat? comes when you use them together.

Consider this little snippet of code:

let x = [1, 2, 3]
console.log("x: " + x + "\n\n")
console.log("1 in x        : " + (1 in x))
console.log("x.includes(1) : " + x.includes(1) + "\n")
console.log("2 in x        : " + (2 in x))
console.log("x.includes(2) : " + x.includes(2) + "\n")
console.log("3 in x        : " + (3 in x))
console.log("x.includes(3) : " + x.includes(3) + "\n")
console.log("4 in x        : " + (4 in x))
console.log("x.includes(4) : " + x.includes(4) + "\n")
console.log("0 in x        : " + (0 in x))
console.log("x.includes(0) : " + x.includes(0) + "\n")

Run that in your favorite REPL and you’ll get

x: 1,2,3
1 in x        : true
x.includes(1) : true
2 in x        : true
x.includes(2) : true
3 in x        : false
x.includes(3) : true
4 in x        : false
x.includes(4) : false
0 in x        : true
x.includes(0) : false

At first glance the results for 1, 2, and 4 make complete sense, while the results for 0 and 3 seem wrong. Technically they’re not. Because the in operator operates on the keys of an object and includes() looks at the values. And JS has decided that the indexes into an array are it’s keys. So as I said, it’s technically correct, but WAT?

Code is read way more often than it’s written, and you never want to confuse the reader. Clear, unambiguous names are important. includes() is pretty clear and unambiguous. Are any of the elements of the array you’re checking? No problems there. It’s the in operator that makes me go WAT?

First of all, it operates on the keys of a map, not the values. Standard usage of English says that if you ask if an element is in a collection you’re talking about values, not keys, so the operator should be keys or something like that. That would go a long way to avoiding the WAT. The other problem is that the in operator treats arrays as maps, with the indices as keys. WAT? Sure, if you dig down into the implementation you can see why/how that works, but no. It’s not right. It’s not an emergent feature that adds customer value. It’s a footgun just waiting to go off.

ADRs and You

I’ve talked about Architectural Decision Records before. Today is the 10th anniversary of Michael Nygard’s original article on them so I wanted to mention them again. Not just to remind folks that we need to record our decisions (which we do), but also as a segue to what, and more importantly, who an architect is.

According to Martin Fowler, “Architecture is about the important stuff. Whatever that is.”. While not terribly specific, it’s a pretty good heuristic. Add in the reality that while we might know the problem we’re trying to solve at the beginning, and maybe even have a good idea of the initial approach, we don’t know what we don’t know, so the final answer, and its architecture, can't be fully known up front.

Which brings us to the concept of doorways. Every choice we make takes us through a door. We become aware of new opportunities and we lose sight of the opportunities we had on the other side of the door. Some of those choices are easily reversible, even after some time has passed. Those are two-way doors. Others are one-way doors. Those choices are much harder to change after the fact. Those decisions need more deliberation and a lot more documentation of the why behind the decision. As noted above though, we often don’t have the information and context needed upfront to make that decision, and you can’t always know what kind of door it’s going to be until you get there, long after the “design doc” has been written.

Regardless, that decision, especially if you’re going through a one-way door, is an architectural decision. By definition, the person making an architectural decision is an architect. That means that at one or more times in the development cycle, we’re all architects. And those architectural decisions need to be recorded. Because at some point, after the decision has been made and implemented, someone is going to have a question about why the decision was made the way it was. Because a new use case or edge case or requirement has come up and needs to be addressed. That person is going to need as much context as possible to decide what can be extended, what needs to changed, and what needs to be completely redone.

And now we’ve come full circle. We’re all architects and architectural decisions need to be recorded. The way to do that is with Architectural Decision Records.

PR Comments

I’ve talked about what makes a good commit message. They’re important markers left for future developers to understand what changed and why as they look back to understand why things are the way they are. They’re also important in the moment to the people who are reviewing the code. They explain the purpose, expected changes and interactions, and potentially, what won’t change.

That’s important information when you’re reviewing a PR for a bunch of reasons. One of them is that it informs what you comment on. Any issues you see during the review should be noted, but that doesn’t mean they’re PR comments. Let’s say there’s a PR on some library function and you notice during the review that there are some uses of the method that don’t handle the returned error properly. You should definitely make a note, probably a ticket for the right group(s), but those fixes most likely shouldn't be part of the library change PR (unless the change is to how errors are reported). On the other hand, if you notice that some of the new code is a lot like other code in the library, that should be a comment on the PR.

Which leads to another way that the commit message helps inform how you make the comment. Generally speaking, PR comments come in 2 flavors, blocking and non-blocking. Blocking comments are those that the reviewer feels need to be fixed AND re-reviewed before it’s OK to land the change. Pretty straight forward.

It’s the non-blocking ones that can be more interesting. They can range from inquisitive/informational to “this needs to be fixed but I trust the author to get it right”. The challenge then is to make sure the author of the PR (one of the readers of the comment) understands the commenter’s intent. The easiest way to do that is to be explicit. Identify what kind of comment it is and what the expected response (if any) is.

In a larger PR, or where reviewers with different focuses are looking it might be helpful to add some identifiers. Maybe the comment is security related, or is particularly important for a specific use case. For example, a comment related to calling the method in a CI process vs a production service or a CLI app. Noting the difference will help bring awareness to the use case and possibly uncover additional issues.

Another important thing to keep in mind is that although comments are attached to a specific line of code, they might apply to a block/loop/method/library/etc. That means, particularly for the larger scoped comments, a one line subject can provide some context and let the reader know what’s coming. And if you need to provide context, you’ll want to provide the detail.

Which gives us a nice template to use for PR comments. Something like:

<label> [optional identifiers]: <subject>
<details>

In practice, this might look something like

nit: Misspelled word in error message, “thier” should be “their”

Which would be a blocking change that is expected to be fixed before land, but doesn’t need re-review. Or maybe

Suggestion: This logic is roughly duplicated in methods XXX, YYY, and ZZZ
A future PR be considered should refactor the logic into one place

Indicating a future task to decide if there should be a refactoring, and then in that case a new PR should be made to refactor. The existing PR should NOT be delayed/extended to include the refactoring.

Check out Conventional Comments for more examples and suggested labels/identifiers.

Plans and Planning

We’re in the thick of planning season again, and at times like this it's helpful to remember something else Eisenhower said back in 1950, “Plans are worthless, but planning is essential.” That sounds paradoxical on the surface, but it’s a generalization of a line from von Moltke the Elder’s On Strategy, “no plan of operations extends with any certainty beyond the first contact with the main hostile force.” And given Eisenhower’s background, that’s not an unsuprising source.

Thinking about it from that context, it’s not that paradoxical. It’s not that plans, in and of themselves, are worthless. It’s that plans make all sorts of assumptions about how the thing you’re making plans about are going to respond to the plan. And like any other assumption, reality is often very different.

So we know that long term plans are going to be problematic. But that doesn’t mean that planning (or even building the plan) is wasted. To build a good plan you need a deep understanding of the situation. You need to understand the drivers, the constraints, and the goals. Not just what they are, but the relative priorities inside and across those groups. Which means that the output of the planning process is not just the plan itself, but also a shared understanding of the problem and solution space.

It’s that shared understanding that really provides the long term value. You start with the plan and the first step in the plan. Then, as reality diverges more and more from the plan, the entire team can work together, with that shared understanding of the solution space, towards a solution to the problem the team set out to solve.

Not necessarily the solution as envisioned in the original plan. Reality has provided direct and imperative feedback on the situation. The team needs take that into consideration and update the plan, and the end goal, accordingly.

Because plans are worthless, but planning is essential to solving the customer problem you set out to solve.

Cloth or Die

You’ve probably heard of Adam Savage, of Mythbusters and tested.com fame. Lots of fun stuff there. And lots to learn as well. Every once and a while he’ll need to cut something. Depending on what it is, he’ll reach for a different tool. Plasma cutter for steel plate, Pipe cutter for pipe. Scissors for paper, cloth, aluminum and sheet metal. 

Unsurprisingly, he doesn’t use the same scissors for all of them. Aluminum and sheet metal use one set, and they’re a lot beefier and have a much longer moment arm. One thing that might be surprising is that he doesn’t use the same scissors for cloth and paper. In fact, there are lots of kinds of scissors. Sure, they’re all hand operated and have moving blades, but they’re pretty different and some have specific uses. And apparently one of the mst damaging things you can do with a pair of scissors is cut paper. As he put it in his 20 minute video on scissors, on his cloth scissors he wrote “cloth or die” to make sure no one gets it wrong.

The same holds true in software. Consider the Collection. There are lots of different collection types, but in general a collection is a bag of stuff. It Depends. With enough extra code you can use almost any collection type to provide almost any functionality. But that doesn’t mean you should. What kind of collection you should use depends on what you want/need to do with the collection. Need really fast access to a statically indexed/ordered collection? Probably want an array. Or maybe a hash/map. Need to guarantee that there’s only one value for a given key? Try a set. Are there more reads or writes? Does it need to be sorted somehow? Do you need to maintain the order things were added? Are there memory constraints? All these things go into the decision making process.

So next time you need to maintain a collection of things make sure you know how you’re going to use it. And make sure you don’t earn the Wrong Scissors (de)merit badge.