Recent Posts (page 10 / 67)

Hello Mother

Speaking of change and staying the same, today is the anniversary of the mother of all demos.

54 years ago Douglas Engelbart got on stage at the ACM/IEEE Fall Joint Computer Conference and defined the environment we’re living in today.

In 90 minutes he showed, on one system, things that had never been seen together before. Things like graphical interfaces, windows, real-time collaboration, hypertext, mice, version control, video conferencing and more.

A lot of people say that Steve Jobs stole the ideas for the Mac from Xerox’s PARC, and that might be true, but the folks at PARC got their ideas from the Engelbart and the demo. Pretty much everything we take for granted about computers (and smartphones) can be traced right back to that demo. Everything from Amazon to Zoom. The world wide web is there in hypertext. Click on something and see it. Chat with your friends and co-workers. Video, audio, and shared media. You can see the basis for communities like Facebook, Twitter, Mastodon, and TikTok in the demo. You can see the beginning of Napster and file sharing. The source for Spotify and streaming. And to top it off the demo was done remotely. Engelbart had the terminal with him, but the computer was back at the office.

About the only two things the demo didn’t foreshadow were touchscreens and the size of today’s smartphones. While they weren’t in the demo, they were signs of what was coming. In 1965 the first touchscreen systems were developed and that’s the same year Gordon Moore first mentioned his eponymous Moore’s Law, so if you were aware of them you could see the writing on the wall.

So thanks Doug for the demo and the world you wished in to being with your demo. You should watch it when you get a chance.

Properties

Properties are more than squares on a Monopoly™ board. According to dictionary.com, a property is

1. that which a person owns; the possession or possessions of a particular owner:
2. goods, land, etc., considered as possessions:
3. a piece of land or real estate:
4. ownership; right of possession, enjoyment, or disposal of anything, especially of something tangible:
5. something at the disposal of a person, a group of persons, or the community or public:
6. an essential or distinctive attribute or quality of a thing:

Monopoly uses the first 5 definitions, but I’ve been thinking about the sixth version. Properties as attributes and how they relate to software in general and testing specifically. I’m a believer in unit testing. I like the confidence it gives me that what I’ve written does what I think it does. I like that it tells me that the public surface of the thing I’m writing makes sense. That it doesn’t surprise anyone or make them work too hard. And I try to test before as much as I can, but sometimes I end up testing after.

Regardless of when I test though, my first inclination is to think of edge cases and write specific cases for them. That works, but it leaves you vulnerable to the Enterprise Developer From Hell (EDFH). The EDFH is the kind of person who will write code designed to pass your tests, but not actually be correct. The kind of person that would write code that looks at the inputs by running your tests and set the output based on pattern matching. At some point you might convince the EDFH to actually write the code you want, but if you only have a handful of inputs then it’s easier and quicker for them to just pattern match.

One way around this is to use property-based testing. Take something simple like classifying triangles given the length of the three sides. One way set of classifications of triangles is this set of 4 types:

• equilateral: All three sides are the same length
• isosceles: Two sides are the same length
• right: The sum of the square of the length of 2 sides is equal to the square of the length of the third
• regular: Any other triangle

A conscientious developer would do some analysis of the requirements, right some if statements that compare the sides, and then return the correct value. The EDFH, on the other hand would run your tests, see that your inputs/expected results are

• 2, 4, 6 -> regular
• 1, 1, 2 -> isosceles
• 3, 4, 5 -> right
• 7, 7, 7 -> equilateral

And right a function who’s psuedo-code looks a lot like

switch (inputs) {
  case 1, 1, 2 : return "isosceles"
  case 3, 4, 5 : return "right"
  case 7, 7, 7 : return "equilateral"
}
return "regular"

Sure, the tests will pass, but it’s not right. The question is, is there a way to get around the EDFH? One way is to use property based testing.

You know the properties of the different types of triangles. You also know a couple of other properties of triangles in general you can use to your advantage. You know the order of the sides doesn’t matter. You know that the sum of the lengths of any two sides must be greater than the length of the third side. So, instead of hard-coding the lengths, pick two of them randomly and then set the 3rd based on the known properties of triangles. Then try all the different permutations of the order of the lengths you’re validating. They should all give the same correct results.

When you use the properties to validate the function you make things much harder for the EDFH. They can’t do specific pattern matching. They’ll have to fall back to writing code that actually uses the properties, which is much more likely to result in code that actually does what you want, not just passes all the tests.

Property based testing has value even if you’re not dealing with the EDFH. It can find gaps in your logic. There’s one case we haven’t talked about yet. What happens when your input breaks the rules. Sure, you said “Tell me what kind of triangle this is”, and defined what 3 of the types were rigorously, but the 4th type was an incorrect catch-all. Tthe input is pretty open ended. You can use the properties to write a test that breaks the side length rule. Inputs that aren’t even a triangle. If you pass in 3, 4, and 10 the developer should come back to you and ask what you want done if the sides don’t represent a triangle.

So when you write tests, even if you know the developer isn’t the EDFH, because you’re not just the test writer, but the developer as well, think about the properties you’re really testing.

The More Things Change

November seems to have slipped away, but I’m back now. This peice has been brewing for a while and it’s time to share.

Things sure have changed for developers. The first piece of software I got paid to develop was written in Fortran77 on an Apollo DN4000. The development environment was anything but integrated. I used vim to edit the code, f77 to compile the individual source files into object files, then f77 again, this time with all of the object files, to generate an actual executable file. All manually. Or at least that’s what I was supposed to do.

At the best of times it took a while to do all those steps. Throw in debug by print and things were really slow. Write some code. Compile it. Link it. Run it. Look at the output. The cycle time could be hours or even days depending on how long it took me to get the syntax and logic to both be correct at the same time.

To make matters worse, I would regularly forget one of those steps. Usually, the linking step. Unsurprisingly, if you compile your code, but don’t link the object files into a new executable nothing changes. And instead of remembering what I forgot to do, I’d edit and compile again, and still nothing would happen. Eventually we figured out how to use Make, which solved most of the problems, but not all of them. The thing that Make didn’t solve was remembering to actually save the file. Especially when I learned how to use multiple windows so I could keep the editor open while I compiled, linked, and ran things.

Today modern IDEs make this much easier and less error prone. They know what files you’ve changed. They understand the languages. They understand and can, in many cases, figure out the internal dependencies of your code and build only the things that need to be built. And for those situations that the IDE itself can’t handle there are external tools like bazel to handle the dependencies and manage the build.

At least they usually do. I was helping my daughter with programing homework the other day and we were doing some front-end development. Doing the usual console.log debugging we weren’t seeing the output we expected. It turns out that she didn’t have autosave enabled on her IDE, so there we were, 30+ years later, still having problems keeping our development velocity up because we forgot to save a file.

Of course, that’s an outlier. The typical compile/test cycle is seconds to minutes. Some systems even complain if your tests take too long to run. Which means in that for a typical developer you can get (nearly) instantaneous and continuous feedback on whether your changes are having a negative impact on your unit tests. You can feel confident that you haven’t broken anything, and if you’re a person who writes tests first instead rather than test after, you know when you’re done.

Some folks are still dealing with that long, drawn-out, error-prone process. Two such groups that I deal with regularly are Data Scientists/ML Researcher. To do something the often need to not just make a change to their analysis code, but make one or more upstream changes to the data itself, which involves a completely different code change in a different code path. And then they must regenerate the data. And point their analysis/training code at the new data. All of those steps take time. They’re hard to track. They’re easy to miss. There’s not great tooling around it. Even worse, for some things, like the data regeneration, you can’t do it locally. There’s too much data or there’s access controls around it or you’re in the wrong country and the data needs to stay where it is, so you need to submit a request to have some process run to regenerate the data. And that can fail for lots of reasons too.

Put together, doing data science/ml work often feels like development used to. Which isn’t surprising. The more things change, the more they stay the same. It’s just the frame of reference/scale that changes.

Omission vs Commission

Some bugs are bugs of commission. In my experience, it’s most of them. You get an equation wrong. You compare against the wrong, but similarly named, field. Or maybe as simple as flipping a sign in a test. The nice thing about bugs like that is that they’re usually easy to spot. Ideally with your test suite, but even if not, the response is generally wrong, so you notice, and then you fix it.

Other bugs are bugs of omission. The code you’ve written is 100% correct. It has the right logic in the right place. The algorithms and calculations you’re doing are correct. The code usually, almost always, works. Those are the ones that are hard to find.

They’re the ones that aren’t obvious. They only happen when conditions are just right (or maybe, just wrong). They happen because of assumptions and biases used when the code was being written. And they creep in at various points in the development process.

At the lowest level it happens when you get caught by your assumptions of how a languageworks. Back in the old days of C/C++ memory management was critical. Not just being sure to free everything you allocated, but avoiding buffer overruns and using uninitialized memory. Both of those less likely (or nominally impossible) with more recently developed languages, but there are similar things. Even if there’s garbage collection to clean up after you, you can forget to allocate something or allocate it twice. In go the differences between a slice and an array are subtle. Unless you are careful when passing/returning them you’ll end up with copies of copies. Your changes will be visible in some places and not in others.

The next level is dealing with more abstract things. Consider a search method. There are lots of things you could omit when using a collection. What if the thing you’re looking for isn’t there. What if there are more than one of them? What if the collection is empty (or worse, not defined)? How does your code handle that? Of course, it’s obvious when you use a method called Find() that you’re doing a search. But sometimes what you’re doing is logically a search operation, but you don’t necessarily recognize it. Create something. Store it. Update it. Do some other stuff, then update it again. What happens if things change while doing that other stuff? Like someone else deleted the thing you created. What do you do then?

Or in a distributed system, what if the remote call you’re making works, but the success response is lost. Did you handle that case? Do you just try to create it again, or do you do a search and then try to create it again? Or do you just ignore the response in the first place since “it always worked before”? That’s a big sin of omission.

Then there’s the meta level of omission. You’ve got an API that can do multiple things. Some of those things are permanent. Consider the lowly rm command. Back in the day it was possible to run rm -rf /. The command, being the good command that it was, assumed you knew what you were doing and began deleting everything. After all, computers are good at doing what you tell them, not what you want. In case you’re wondering, even with good backups, it’s really hard to recover from that. Since that’s not something you would want to do (there are better ways to actually erase everything on a disk) there was no code to prevent you from running a valid, but probably unintended, command. Now, on linux at least, there’s the --no-preserve-root option and a check to make sure you don’t make that mistake. But it’s indicative of a whole class of errors. The “Why would anyone do that?” error. And if you don’t check for it then at some point, it will happen.

The trick is to test for them. But how. Part of it is a mindset. Think of ways that things could fail, and test for those, not just the happy path. Think about ways your system could be misused, or at least misunderstood. Make sure you’ve guarded against those things too.

This is also where another set of eyes is good. Someone without your biases and assumptions. That person will think about things in a way you’re not. After all, if you thought about what would happen if someone tried to delete the entire filesystem you would have added the flag in the first place.

Strange Loop

Strange Loop Strange is, according to its own website, a multi-disciplinary conference that brings together the developers and thinkers building tomorrow’s technology in fields such as emerging languages, alternative databases, concurrency, distributed systems, security, and the web. I’ve had it on my list of things to do one of these years for a while. I’ve heard a rumor that next year is going to be the last one, so I guess I know what I’ll be doing in late September next year.

That said, this year’s conference is over, and according to the posts on twitter it has lived up to expectations yet again. The thing that has always interested me about Strange Loop is that it seems much more practical yet also big picture. Like any conference there are plenty of companies that use it as a chance to market their products, or at least toot their own horns. And that’s fine. There’s value in advertising for companies and there’s value for attendees and viewers in those talks.

If you look at the conference schedule there are plenty of talks that describe how a person/company solved a gnarly problem they faced. There are plenty of learnings in there, as long as you keep the context in mind. Talks about handling concurrency, distributed systems problems, observability, security, and testing. Things that I deal with every day at work and that are important to me. Learning from other’s mistakes and lessons, even if they’re in a very different field.

But it’s more than that. There are also talks like It Will Never Work In Theory. Talks about the difference between theory and practice and how to bridge the gap. Or a keynote on Expert Software Developers’ Approach to Error, looking into how bugs happen and how to not just fix them, which is critical, but also how to not just avoid them, but prevent them, which is the best way to fix a bug.

Then there’s what’s fascinating to me. How this stuff gets used, and how it ends up impacting people’s day to day lives. How it makes their lives easier, better, or just more interesting. How we use it to understand and experience the world around us and its history. How Live Music Is Evolving In A Post Pandemic World, and The Vera C. Rubin Observatory Legacy Survey of Space And Time. Sure, those will touch on computers and technology, but that’s not what they’re about.

Eventually the talks will be available online. There are a bunch I want to watch. And hopefully next year I can go in person, before it’s too late.

Tensegrity

It’s been a while since I posted anything. Sometimes life intrudes on posting, but I’m back to posting again. Today’s question is, can you push something with a rope? Sure, you can do it if the rope is short enough that it effectively can’t bend, but what about a longer rope? You can’t. But you can make it look like you’re pushing with a rope. You do it with tensegrity. Tensigrity is not new. The first examples were shown in the 1920s, and Buckminster Fuller (of geodesic dome fame) coined the term in the early 1960s. It’s the idea that by properly pre-loading the elements of a structure, some in tension, some in compression, you can build a stable structure. From an architectural standpoint it lets you right-size different components based on the load each component will actually be carrying. It also lets you build apparently impossible figures where chains push things up.

That’s really cool, and I need to build one of them, but what in the wide, wide world of sports has that got to do with software?

Simply this. When building complex systems, you have multiple competing forces (requirements) that you need to handle. You can do it in multiple ways. The easiest (from a design standpoint) is to make sure that each individual component can carry the maximum load of the entire system. Now building each component that way might not be so easy, or might be so expensive as to be impractical.

That’s where tensegrity comes into play. The forces are in balance. Putting one piece into compression puts other pieces into tension. That pulls things back to equilibrium. It’s a feedback system. It’s systems thinking. This applies to software as much as it applies to architecture.

It’s not that you can maintain everything the same in the face of increasing stress (load), but that you spread the load so that the system doesn’t fail catastrophically. The load on one system influences a different system, which in turn impacts the load on the first system. Consider a simple retry system. If something fails, instead of just retrying, add an increasing backoff time. That reduces the load on the downstream by temporally spreading the load. Good for the downstream system, but it does increase the load on the sender since now it needs to keep track of retries.

It’s by no means a magic solution to all your problems. As you can see in the GIF above, as long as the system is in balanced tensegrity, it stands. Add too much load in the standard direction and something will break. Add too much load in an unexpected direction and the system can’t compensate. Instead, it fails in a new and exciting way.

All the parts are fine, but the system isn’t holding. Like all other solutions, the answer to the question of whether it’s the right solution for a given situation or not is “It Depends”. And like every other solution, it’s one to keep in your toolbox for when the time is right.

Plus, it’s an excuse to show a LEGO build video.

More Broken Windows

I’ve talked about the Broken Windows theory before. The idea that visible signs of problems in a community, like persistent broken windows, is a leading indicator of further problems in a community. And I said that in software development problems, like frequent pages, technical debt, and failing to code for the maintainter, set precedents and lead to more of the same.

Since then, I’ve seen multiple cases where, anecdotally, that has proved true. It starts with one little thing. Then another. And another. Pretty soon, intermittent failures are ignored. Places to incorrect comments left in code. Times when, instead of simplifying a complicated conditional, more specific conditions are added.

Unfortunately, as important as individual cases and anecdotes are, as important to insights and understanding anecdotes are, they are stories. And while a story might be a singular datum, the plural of anecdote is not data. The thing is, just because there are stories and anecdotes, it doesn’t mean there’s no data. I just didn’t know of any. Until now.

According to a recently published paper, The Broken Windows Theory Applies to Technical Debt, there is now proof.

We now have data. Not just data, but also, rigorous analysis of the data. Sure, it was a small study. 29 participants and 51 submissions. All from a small geographic region. It’s not incontrovertible, but it’s a start.

According to the paper,

The analysis revealed significant effects of TD level on the subjects’ tendency to re-implement (rather than reuse) functionality, choose non-descriptive variable names, and introduce other code smells identified by the software tool SonarQube, all with at least 95% credible intervals. Additionally, the developers appeared to be, at least partially, aware of when they had introduced TD.

Not only did the level of technical debt go up, but the participants at least partially recognized that they were adding technical debt to the code.

The study didn’t say it, but if existing tech debt is strongly correlated with more debt being added, but it stands to reason that reducing tech debt will cause a reduction in the amount of new tech debt added.

Something to think about. And act on.

Readability vs Idioms

We’re all here trying to add value. Trying to solve the problems we’ve seen. The problems that keep us up at night. Coming up with solutions that make a difference to something that’s important to us.

The question though, is not how to add the most value in the next instant, but how to add the most value over time. That’s and important distinction. Because what adds the most value right now might be different from what will add the most cumulative value tomorrow, next week, or next year.

You can see that in many different places. Helping someone else solve their problem instead of focusing on your own all the time. Taking many more much smaller steps in the right general direction as you figure out the correct path. Instead of just fixing a failure/outage/mistake, using it as an opportunity to make sure that a whole class of problems doesn’t happen again, or at least is much easier to isolate and recover from. It happens when you write new code and when you modify existing code.

Readability is a big part of that. You can solve almost any programming problem in almost any language. And you can make it look the same in all those languages. But should you? Is it really the most readable, most understandable, most maintainable way if you do it that way?

I’d argue that it’s not. Consider Go’s structs vs Java’s classes vs Python’s classes. At a high level they’re all about encapsulation. Keeping things in a domain together and separating domains. But they’re also very different. Java loves to inherit, but can compose. Go loves to compose and doesn’t have inheritance. Python has inheritance and composition, but doesn’t care. Java has a strict type hierarchy, which must be obeyed. Go and Python are duck typed. If you write the same code in all three languages it’s going to look odd in at least one (and probably all) of those languages to people familiar with that language.

That’s not good. No matter how readable code is in one language, if you transfer how you write in one those languages to another language it will NOT be readable. It might be decipherable, but we’re not looking for decipherable, we’re looking for readable, easy to understand code.

Of course, readability is one of those ilities. The non-functional requirements that creep into your requirements and guidelines. And like most of the ilities, there’s not a good, clear, objective, metric for how readable your code is. Given that, you’re never sure how readable your code is. So how can you know if it’s readable enough?

You can’t. so maybe there’s a better thing to focus on. Like guardrails. Guardrails that make it easy to do the right thing. Guardrails that encourage working with the system. Guardrails that make it obvious what is happening, and what isn’t.

And that’s where libraries and idiomatic code come into play. An example in Java is using Try/Finally. In Go it’s checking the returned error value. In Python it’s truthiness and falseness and comprehension. If you write with those idioms (and others) other people familiar with the language will see what you’re trying to do. They’ll know what you mean, what you’re trying to do, and the benefits and limitations of the idioms, libraries, and language features you’re using. By working with your languages and libraries you help yourself and anyone else who works on the code do the right thing.

Saying you’re doing something to make it more readable is arbitrary and subjective. Making it easy to do the right thing and hard to do the wrong thing is clear and objective. Don’t just ask people to be careful, make their lives easier.

The Developer's Journey

I’ve been a developer for a long time now. I’ve seen others start down the path. I’ve seen people make mistakes and picked them up and helped them continue on their journey. I’ve seen their trials and tribulations. I’ve seen their highs and lows. And everyone’s journey is different. But there are some pretty common things I’ve seen along the way, especially amongst the most successfull.

Starting Out

In which the person sees an opportunity. It could be as simple as wanting to change the color of a texture or an operating parameter of an item in a game. It might be more complex, like automating some calculation in a spreadsheet, or it might even be some “advanced” thing like parsing some data out of a set of text files and then doing something with the data. But the developer rejects the opportunity and decides it’s easier to do it by hand or just live with things the way they are. After living with the problem for a while it turns out that someone the person knows has some experience and somehow, with a few deft keystrokes and function calls, solves the problem, almost magically. Armed with that new knowledge the person tries to do the same thing themselves.

Learning and Growing

Of course, doing it themselves is hard. At first it’s simply struggling with the language itself. What are the keywords? What kinds of loops and collections are there and which ones should be used when? Slowly it starts to make sense and they’re thinking less about how the language works and more about the problem they’re trying to solve. This kind of thinking expresses itself as levels of abstraction where the boundaries are between things. At first everything is ints, floats, and strings in a single giant method. As they progress in their ability and understanding they start to coalesce into types, libraries, and executables.

Systems, data, and their interactions start to be the driving factor instead of what a method or library does. They start talking about the problems in problem domain, not the code domain. Who took what? What’s the most efficient way to get that person from home to a hotel in a different city? How can we ensure that all of the food we need is made, everything we make is used, and leftovers are used to help those in need?

Sharing

At this point in their career, the developer is deep in the code. They understand it. They understand it’s place in the world. They know how a change in one place will have some unusual impact to something else. Not because of a problem, but because they understand the underlying connect between those two things. They’re usually very happy in that place. Often, they never leave.

But sometimes they do. Either they recognize themselves, or someone (a mentor or manager) points out to them that they could do more, for themselves and others, if they want to. They can teach others. They can help connect people and things. They can expand their scope and purview to include more and different things. They not only solve problems themselves; they help others solve the problems they have.

They recognize that being a developer is not just about code. It’s also (mostly?) about people, businesses, problems, and how they are all interrelated. Connection, influence, and passing on the learnings about how to avoid problems become the primary goals.

Or, as I’ve put it before, a developer’s career progression is about scope. Are you just looking at yourself, your team, your company, or your world. The broader your scope, the further you are in your journey.

Of course, I’m not the only one to look at the developer’s journey. According to Joseph Campbell, the developer’s journey goes through 17 distinct phases, often broken down into 3 acts. To wit:

1. Departure
2. Initiation
3. Return

Wait a minute. That’s not the developer’s journey, that’s the Hero’s Journey.

They’re two different things, aren’t they? Consider about Bilbo Baggins. He didn’t ask to go There and Back Again. He didn’t even know that such a thing was possible when the story started. But he went. He learned. He understood. He had a wizard and good friends to help him along the way. Then he came home. And made the shire a better place. For himself and everyone else.

Maybe the developer’s journey and the hero’s journey aren’t that different after all. Something to think about.

One Thing At A Time

Doing one thing and doing it well is The Unix Way. As I said in that article, tools should do one thing and do it well. If you need to do two things with two tools then connect them with a data pipe. It’s a great tenet for tools and applies to any number of systems that I’ve worked on. From text extraction to image processing to 3D model generation to an entire micro-service network.

It’s a great tenet in other areas as well. It lies at the heart of an agile development processes. Do one thing and finish it. See how it works. Get some feedback. Figure out what the next thing to do is. Do that. Lather, Rinse, Repeat. Take many more much smaller steps. You know where you are. You know where you want to be. The exact path between those two (and probably the exact destination) will change as you learn along the way. Uncovering better ways of developing software by doing it.

Another place it applies is change management. How you structure your checkins, code review (CR), pull request (PR), or whatever you call them. Every commit or CR should have one logical purpose. Sometimes that means that any given CR doesn’t add customer value. Sometimes, before you make a change that adds value you need to make a change (or multiple changes) that [makes the change you need to make easier)(/posts/2020/10/27/). And that’s OK. Because just like you should code for the maintainer, you should make CRs for the reviewer.

The question is, why is this important? After all, isn’t it faster to have one change, one big optimal step, that just works? In theory that might be the case. In practice it isn’t for, multiple reasons. Most importantly, ensuring that one big step ends up in the right place is probably impossible. We don’t know where that place is exactly, so there’s no way we can be sure we’ll hit it.

There’s another reason. A reason that has to do with combinatorics. Let’s say you have two changes you’re making. Let’s make it really simple by saying that the change either works or it doesn’t. Determining if it does or doesn’t is trivial. In this situation there are 4 possible outcomes. They both work, they both fail, or one works and the other fails. In this sitation. 75% of the possible outcomes are failures. Then. after you determine if the experiment is a failure you need to figure out which of the three possible failures it was. Then you need to fix it. The more things you try at once, the worse it gets. With 3 changes 88% of the outcomes are failures. Only the top path, with all results Heads (success) is a successful attempt.

With 4 changes 94% of the possible outcomes is a failure case. Any savings you get by taking that big step are going to be eaten up by dealing with all of the possible failures. You might get lucky once or twice, but over the long term you’re much better of makig one change at a time.

It doesn’t matter if the changes are in a single CR to be reviewed, a data processing pipeline, a microservice network, or the architecture of something you haven’t built yet. The more you change at once, the harder it is to know if the changes made things better or worse. So take the time to decompose your changes into atomic, testable, reversable steps and then make the changes, doing one thing at a time. You’ll be happier. Your customers will be happier.

And surprisingly, you’ll move faster as well.