Recent Posts (page 24 / 67)

Autonomy != Anarchy

Autonomy is more than a car or truck driving itself. According to dictionary.com it’s

independence or freedom, as of the will or one's actions

But what does it really mean? Especially to a team, working with other teams?

Autonomy is power. The power to decide, for yourself or your team, what the right answer is. The power to decide how to achieve that result, The power to decide how to present and share that result,

By that definition the difference between autonomy and anarchy isn’t that big. It could be argued that they’re the same thing. Certainly if you have anarchy then you have autonomy, But I’m pretty sure that anarchy isn’t the best way for a group of developers to get something done.

While autonomy can mean the freedom to choose to do whatever you want, in this context it doesn’t. It means understanding the end goals and constraints and having the freedom, the autonomy to act within those constraints to achieve the goals.

It also means that you have a bunch of responsibilities. Of course you’re responsible for meeting the agreements you’ve made. But you’re also responsible to communicate. To communicate status. To communicate when expectations change. To communicate when your needs change. Again, autonomy doesn’t mean do what you want when you want and don’t think about the consequences.

The key is to balance the power and the responsibility. To maintain alignment between your autonomous decisions and everyone else’s autonomous decisions and the shared goals. To understand that while you have the power to do what you want, you’re working towards a shared vision and what’s best for you isn’t necessarily best for the vision. It’s easy to confuse a personal maximum for a team maximum. Especially if all you look at is speed. Just because you’re moving as fast as you can doesn’t mean you’re adding the most value. 

Or, as Uncle Ben said, “With great power comes great responsibility.”

Perspective

You’ve probably seen this before, maybe you’re one of today’s lucky 10000.

What’s the number of the covered parking spot in this image?

When I first saw this I had no idea. Eventually I decided it had to be 128. Because pairwise, 16 - 6 = 10, 68 - 88 = -20, so x - 98 = 30. That’s a stretch, but it made some sense.

Of course, for parking spot numbers that makes no sense. Who would number parking spots like that? You’d never find your parking spot, and there would be duplicates. Really bad idea.

On the other hand, it’s all a matter of perspective. If you look at it this way, it seems pretty obvious.

In this case it’s clearly 87. And that makes sense from a real-world parking lot perspective as well.

So what do we learn from this? We learn that our perspective, our biases, can change how we solve problems. 

Back in my aerial imagery days we had to deal with lens distortion. We needed to minimize it to get the most accurate images possible. Lots of effort went into multi-part lens design and construction. Managing thermal deformation, both during lens grinding and after installation was important. Things had to be just right. And we still couldn’t get rid of the distortion.

So we cheated. Or rather we looked at the problem from the other direction. Given a perfect test image, in our case a grid of points, and an actual image, what kind of post-capture transformation did we need to do, at the pixel level, to get the correct image. We couldn’t calculate it before hand, but we could easily measure the deltas. So we did. Then we inverted the direction. And then we applied them to all of the raw images. Which removed (almost) all of the distortion.

So what are currently you doing at one end of a pipeline that you could do more easily at the other?

Inspect And Adapt

Agile. XP, Scrum, SAFe. Software development methods. Agile software development methods. They all have their trademarks. Processes and artifacts that you should do and produce that will lead you to high performance. Just follow the rules and it will happen for you.

There’s a basic assumption in there that may or may not be correct though. As Tolstoy put it, 

All happy (high performing) families (teams) are alike; each unhappy (low performing) family (team) is unhappy (low performing) in it’s own way.

It’s that assumption that drives the methods. All high performing teams are the same, so if you do those things you’ll be a high performing team.

But is that really true? It Depends. It depends on how you look at it. From the outside all the happy families/high performing teams look the same. They get things done. They exceed expectations. Meanwhile, at the other end of the spectrum you might see internal fights, distrust, lack of direction/purpose, sullenness, or just plain anger.

If you look a little deeper though, you might have some different insights. The highest performing teams have figured out what works best for them. The right balance of collaboration, autonomy, agency, and direction. It’s going to be different for every team. Because teams are made up of individuals. And every individual is different. And the combination of those different individuals is even more unique.

On the other hand, many (most? all?) low performing teams have a lot in common. The biggest is that they’re often not a team, but a collection of individuals. Without common purpose and goals. Without a shared vision. So while the specific, outwardly visible signs might be different for each of them, the internal issues that are the root cause of the low performance are the same.

It seems to me that if you want to end up at that high performing state, what it really comes down to is the way the team operates being agile. That means some variation of the “inspect and adapt” principle. Look at what you did. Think about what you want. Adjust things to be closer to what you want.

Not closer to some externally defined process/norm/ideal, but closer to what the team (and the customer) defines as ideal.

Obligations

When we talk about incident management there are lots of things to keep in mind, and priority is important. My formative years were in aviation so to keep the priorities in order I always think of it as aviate, navigate, communicate.

Something bad happened. First and foremost, keep the airplane flying, don’t make or let it get worse. Find someplace safe to land. Figure out how to mitigate the problem. Call ATC and declare an emergency and keep them informed. Open an incident and update your customers. That’s the priority, and in the heat of the moment it’s important to keep your priorities straight.

But that’s the priority order. Just because something is lower priority, it doesn’t mean it’s less important. In fact, over the long term the last part, communication, can be more important than the others. Because properly communicating the experience and following through on the learnings can keep the problem from happening again.

Consider this. One of the things that often happens during an incident is that a quick fix is put in place to alleviate the symptoms. You find that suddenly your service is taking twice as much memory and crashing. So as a mitigation you allocate 2x the memory and things are working again. You’re done, right. Time to head home and celebrate.

It might be time to celebrate, but you’re not done. That quick fix is a pressure cooker, quietly sitting there until the next time it explodes and takes out your service again. You might even be able to keep turning up the allocation a few times, but you haven’t solved the problem. It will come back and bite you later. At some point you’ll need more memory than you can get. What are you going to do then?

In reality, that quick, brittle fix is an obligation you’ve taken on. An obligation to yourself, your team, and the business. An obligation to take the time to fix things the right way. The forever fix that identifies why it happened and keeps it from happening again. Because incidents happen. What’s important is how we deal with them.

Keeping Your Balance

Macro vs Micro. Explicit vs Implicit. Small functions vs Big Functions. Up front design vs YAGNI. Like many things software, It Depends. It’s a question of balance.

But what are you balancing? In most cases, when you get down to it, what you’re balancing are the perceived/expected needs of the future against the needs of the moment. In conjunction with the known cost now compared to the potential cost later.

The needs of the moment are clear. Your customers are talking to you about them daily. You’re looking at metrics and error rates and other real-time feedback. You’ve committed to delivering something specific to solve a specific user problem.

Then there are future needs. Some are easier to predict/quantify than others. Someone, likely you, will need to maintain the code you write today. Usage will probably go up. Computers will get faster. Data rates will almost certainly increase. You can count on those things. You should design and code for those changes. It might cost a little more now, but it will pay off tomorrow.

Other future needs aren’t as easily knowable/quantifiable. Will you need microservices next year? Will your domains change, and if so, how? How will market changes impact business goals? What’s your vision and mission, how might they change, and what impact will that have on your plans? And if you don’t know the changes, how can you know the cost of those changes?

Which leaves you to balance the two timelines. You need to think enough about the macro/long timescale side of things to keep from painting yourself into a corner. You need to move quickly and solve today’s problems today. You can’t spend so much time thinking about the future that you don’t take care of the immediate. You can’t spend so much time working on the immediate that you never plan for the future.

You need to keep the two in balance.

Testing And Failure

"Testing leads to failure, and failure leads to understanding." - Burt Rutan

I’m an airplane guy. As such, I’ve been aware of Burt Rutan for 40+ years. I even once helped a friend build a VariEze. Way back before there was a Mark Zuckerberg, let alone Facebook, Rutan was experimenting and failing forward. Lots of iterative development. On paper, in wind tunnels, in the air, and in space.

As impressive as SpaceShipOne was, I think Voyager was even more impressive. 9 days in flight. 26K miles. Mostly one engine. All on a single tank of gas. That’s an endurance record.

To get there took a lot of understanding of the systems involved. Of course the aircraft itself, but not just that. Also the engines. The weather. The support systems. The pilots. The question is, how do you get that understanding? You get it from testing. And fixing the problems. But not just fixing the specific problem, but deeply understanding what the problem was and making sure there weren’t similar problems later.

You can (should) approach software development the same way. You know what you want to happen when everything goes right. And you write things so you get the result you want. And you do your best to consider other possible situations. And make sure that they are at least handled gracefully, even if you can’t do what you originally planned.

Then you start testing, You find the edge of the envelope. More load, more traffic, less time to respond, strange data, whatever your inputs and environments are. And something breaks. So of course, you fix it. But you don’t just fix it. You figure out why you got into that situation and how to make sure you never end up there again. Which means not just understanding the thing you’re writing, but the environment it operates in. And the people and systems upstream that contribute to the environment. And the people and systems downstream who’s environment you are a part of.

Because without that understanding all you’re doing is going fast and breaking things. You’re not actually getting anything done

Rocket Science

Almost 120 years ago Konstantin Tsiolkovsky published his rocket equation:

That’s rocket science. Really. That’s all there is to it. One equation. For any given rocket the change in velocity (speed in a direction) is equal to the rocket exhaust velocity multiplied by the natural log of the ratio of the initial mass to the final mass.

Sure there are a lot of nuances, and any good physics teacher will tell you at the end of class that it isn’t quite as simple as it’s made out to be, but to a first (and probably second and third) approximation, that’s all there is to the science behind rockets.

Which is kind of funny actually. Because when we want to say something isn’t that hard we say that it’s not rocket science. A person needs a certain amount of food, water, and oxygen each day. We know what those numbers are. With that, the relative motions of the Earth and the Moon, and the rocket equation we can figure out just what we need to build to get from one to the other and back on a certain schedule. It’s just math. And not an integral in sight.

No, it’s not the science that’s hard. There are lots of things that are harder than rocket science. Like rocket technology. Just because you know how much fuel you need and how fast you need to throw it overboard doesn't mean you can build the rocket. You need the right materials. You need the right mechanical systems. The right electrical systems. All kinds of implementations of things.

And to make things harder, they’re all interconnected. The more fuel, the heavier your rocket needs to be. Which means you need more fuel. At higher pressures. And the trip is longer. So you need bigger batteries. Or lower power electronics. And so on.

And as complicated as that is, there’s an even harder problem to solve. How do you get the entire system to work together? It takes 1000’s of people years to design, build, test, and fly one of those rockets. As impressive as a Saturn V rocket is (and if you’ve ever been close to one, they are impressive), the management of the process is even more impressive. Getting that many people to work on more or less the right things, at more or less the right time, in more or less the right order. Repeatably.

Software is like that too. Computers are inherently simple. A logic gate is open or closed. 0 or 1. That’s it. Throw in some simple boolean logic. AND, OR, XOR, and NOT. Add a few simple instructions. ADD, SUB, LOAD, and STORE. It’s only when you have billions of them working together, doing all the (relatively) simple things that all the different people have gotten them to do. At the right time. In the right order. With the right feedback loops. That’s when you get a database, a search engine, or a control system that can get a rocket from the Earth to the Moon.

So the next time someone tells you that what you’re doing isn’t rocket science and shouldn’t be that hard, remind them that it’s not the science part that’s hard. It’s the engineering, the turning of science into reality, and doing it with others that’s hard.

Global Data(base)

A database is a giant global variable that pervades your code. How, exactly, is that a good thing?

  -- Allen Holub

Well. That’s a pretty bold statement. Let’s break it down and see how true it is.

A database is certainly a “global” variable. Or at least it is in the sense that any code with the correct permissions/connection string can connect to it and read/write data. That does make it global. Can’t really argue with that.

Especially if you only have one database. With one (widely) shared secret that lots of different things use to connect to that database. To read/write from the same table with a magic set of filters. Or lots of different tables that all have the same unrestricted permissions.

In that case your dB is a SPOF. SPOFs are almost never a good thing. SPOFs are rarely completely unavoidable, but can be more expensive (however you define that) to eliminate the SPOF than it’s worth to avoid, so you live with them, even if it’s not a good thing. so that part is true too.

Which leaves us with the middle part, pervades your code. That’s an interesting one. And about as subjective as you want to make it. Liberally sprinkling your codebase with code (not function calls) that opens a connection to a database, reads/writes some data, then closes the connection and moves on is almost certainly a bad thing. Can’t really argue with that.

So databases are bad and we should just stop using them, right? Of course not.

It’s about using the right tool for the job, and using the right way. If you have one service/tool that needs to deal with a few hundred kilobytes of inter-related data the right tool is probably a clear data model with an API designed for how the data is actually used. Behind that data model have an in-memory set of maps and a little bit of logic to connect them. If you need to get really fancy add some kind of write-through mode so any written changes get persisted for next time you start up. No (formal) database involved.

What if a bunch of things need access to that data? Put it in a (micro?) service. Same API, but now accessible from lots of things at the same time, regardless of what process they’re in. Still no formal dB involved.

18 months later and things are going so well that you’ve got 100s of GB of active interrelated data. Read/Write rates are up. You can’t afford to lose changes, and the ability to undo a set of changes as a unit is now important. So you start to build those changes in. Then you get smart, and instead of reinventing the wheel you put your data in a traditional database.

So was Allen right? It depends. Did you need a 256 core MS-SQL database with 512 GB of RAM and 2 PB of SSD to start out with? Of course not. You didn’t even need a simple AWS RDS. So it was correct.

On the other hand, a database is just a collection of data you access via computer, so pretty much everything is a database. So in the most fundamental sense, he was completely wrong. You already have a database, because storing and accessing data is what programs do.

Like everything though, context matters. It was a tweet, it was written for retweetability and impact. As a conversation starter it was effective. So next time you go to create a new AWS RDS, think about if you need that much database yet. You still need to do all the other hard parts of design, but do you need that bit of complexity? I can’t tell you for sure, but I do know it’s worth thinking about.

Lone Wolf - Hard Nope

Sometimes I read something and wonder if I’m living in the same universe as the author. Then I think about it some more and realize I’m not. Consider this article about How a Single Freelancer Outperforms a Whole Team. At first glance it makes sense. From the Mythical Man-Month to convoy speeds, a single person can be shown to be individually efficient than a team.

But, as I have been known to say, “What are you really trying to do here?” Are you trying to ensure that every individual is maximally busy? Rack up as many billable hours as possible? Ship something and never touch it again? Have the ability to chase after the “fad of the moment”? In that case, go ahead and lone-wolf it.

In my world, on the other hand, the goal is to add value, over time, in support of a larger goal. Those are very different goals. Which leads to two different answers. One optimized for the one wolf, one for the customer.

Yes, a single ship is (generally) faster than a convoy. A single rider is (generally) faster than a group. But consider the whole environment. In WWII convoys lost about 4% of their tonnage, while ships travelling outside convoys lost 20%. Overall there’s no question which method got more cargo safely across the Atlantic. For any given trip individual ships were less likely to be detected, but over multiple trips that changed significantly. It’s the same for developers. On any given task one experienced person is (generally) faster than a group. But over multiple projects, especially when you add in support/keeping the lights on, the overhead work will more than eliminate any gains.

The other, and more significant, way that article lives in a different universe than I do is around uncertainty. When you’re a contractor you either bid a fixed price/fixed result contract, or you go with a time and materials bid. In the first case you decide what you’re going to do up front, whether you understand the problem or not, then you do that thing. After you’re done you leave. Or get another contract to redo it based on what you learned. In the second, there’s no incentive to get it right. In fact there’s an incentive to keep going as long as possible. Either way, the lone-wolf contractor doesn’t care about the uncertainty.

I, on the other hand, care a lot about that uncertainty. In fact, dealing with that uncertainty and figuring out how to turn ambiguity into value is a big part of the job. To go with that responsibility is the autonomy to make the choices that add the most value. In conjunction with other people going through the same process. 

Which leads to my biggest issue with that article. It implies/assumes that nothing anyone else does is impacted by the work the lone wolf does. That the lone wolf is a single point of failure during development and forever after as support. Even more critically, it assumes that nothing anyone else does has any impact on what the lone wolf does, or should be doing. 

And that’s not the world I live in. Or want to live in.

Design By Example

Yesterday I talked about Design by Counter-Example. What about the opposite? You know what you want to do (the user story), but you don’t know how to do it, or what the API and internals should look like. How might that work?

You’d start with a real world problem. A user story that describes a way to make the user’s world better. Better by adding some value. It might be by eliminating manual work, reducing the number of steps, or some whole new way of approaching the problem that solves a user’s problem.

You’d work with the user, discussing options and priorities. You might go to the whiteboard and draw boxes and arrows. Maybe use Visio or Google Draw to make a shareable version. Whatever it takes to understand the domain and how you’re adding value.

Then you think about it a little. And come up with some pseudo-code that shows how you envision adding value to the user. It doesn’t work of course, because none of it really exists, but you get some more feedback from the user and can feel pretty confident that it really will add value.

Finally (really, it’s only been hours, maybe a day or so), you go to your favorite IDE and turn that pseudo-code into real code, but still making calls to non-existent functions and APIs. So you start to fill in those gaps.

You start simple. At first you just blindly return some happy path answers. Things check out, so you continue. Then maybe you add some input and state validation. You check with your user and make sure you’re still adding value. You’ve learned a little, so maybe you tweak the example code a little.

You keep adding depth and detail until all of the examples work and you release the new benefit to the user and they’re happy.

That’s test driven development design by example.