Recent Posts (page 41 / 70)

Branch vs

Maybe I'm late to the party, but recently I learned about git worktrees. I'm familiar with branches, and they're great. Great for isolation. Great for bug fixes. Great for incremental changes. But they're not so great when you have large structural changes.

Or at least they're not great with structural changes in the presence of compiled languages and caches (both bazel and your IDE at least). From a correctness standpoint they work. You can switch between branches and your source code tree will faithfully represent the state of the branch every time. But unless your branches include the intermediate artifacts and state (which they shouldn't) then every time you switch branches your IDE will go nuts as it rescans everything and updates itself and `bazel` will invalidate itself and rebuild huge swaths of code. That's the correct behavior and it produces the correct artifacts.

But it comes at a cost. Depending on the scope of changes switching branches can cause as much work as a fresh `git clone` on a new machine. If you only do this once per branch that's not a big deal. But if you have to switch back and forth multiple times then swapping branches can become a major time sink.

My traditional approach to this has been to just `git clone` a new instance of the repo whenever I need to do this, and then remove it when I'm done. It works, but it burns 6+ GB of disk space for the `.git` folder every time, I need to remember to keep all of the master branches on my computer up to date (or at least in sync) across them, and syncing across them is hard if I need to since the common ancestor is up on github.

Worktrees solve this problem. The git repo isn't duplicated. The same one is shared for all trees, so I don't burn the disk space. There's only one master branch to keep up to date, so it can't be out of sync with itself. And the nearest common ancestor is right here on my machine, so sharing/cherry-picking is as straightforward as it can be.

And because it's a different directory tree on disk both bazel and my IDE don't lose their saved state when I switch trees. So I don't have to wait an extra 20 minutes for everything to download/build because some base version changed. I don't need to watch VSCode take over all my CPUs while it re-indexes the entire structure. I can name my tree (and the folder it's in) however I want to so that I know where I am.

So far it's working out well. It's much easier on my computer than just switching branches, and has less overhead for me than managing and syncing multiple `git clone`s. One thing I've been doing is having the new worktrees outside the base repo folder to make it easier on my IDE and make it easier for me to know what worktree I'm in. And it's simple to use. one 3 basic commands to remember. One to set up a tree, one to remove a tree, and one to list trees. After that I can switch trees just by changing folders, just like I would with a `git clone` based workflow. So far I haven't had any issues, but it's still early in my use of the pattern. Anyone already using `git worktree`? What's your experience been?

Easter Eggs

Extra hidden code that does something fun. Seems like a good idea, and generally speaking I think they're a simple, mostly harmless bit of entertainment. I've enjoyed them. I've built them. Falcon 4's shared memory output started out as a development tool then became an easter egg, and eventually a feature.

They're an old tradition. The first one I ran into was the "Identify 9" easter egg in the old Wizardry series. Try to identify the 9th item in an 8 item list and you got 100M exp points. Then there's the Konami code, which works in all sorts of strange places. And you can still find them in recent tools. The `opus` tool had a couple, `starwars` and `wizard`.  You can find them in hardware too. From chip designers baking their initials into circuit boards to sneaking a Commodore64 program onto a rock album.

But more importantly, while they might be surprises to users, they shouldn't be surprises to the development team. Especially when you're working on mission critical or safety of flight sorts of things. And that's because they need to be tested. And not the standard developer testing of the happy path, but fully tested, including fuzz testing on any inputs and ensuring that they handle them gracefully.

For example.early versions of Microsoft Excel had a hidden flight simulator. It needed to be tested. Not to ensure that the flight model was high quality or that it obeyed physics, but that it wouldn't corrupt data. That it wouldn't suddenly crash and cause data loss. It's bad enough that people don't know the limits and loose data because they add too many rows to a spreadsheet. Image what would have happened if the data loss was caused by a bug in secret code. That wouldn't have gone over very well.

So don't be afraid to have some fun and to spark joy in your customers, but think it through, and don't do things in secret. That never ends well.

Silos vs. Ownership

What's better, a very narrow, well defined scope or ownership of a problem? The answer to that, like so many comparative questions, is it depends. But first, what do those terms mean?

The term silo goes way back to when agriculture shifted from just in time/sustenance farming to commercial farming. It was a place to store all of a certain thing, and a place for everyone else to get it from. If you wanted grain, you went to the grain silo and bought/traded for some. You didn't need to care how it got there, and the people who filled the silo didn't care what you did with it. A very clean separation of responsibilities and both sides could focus on what they did best. The farmers handled filling the silo and the bakers turned the grain into bread.

In this context ownership is about the responsibility for something. It means you know who's responsible for getting something done. It shouldn't mean the person/team is doing everything.

Given that, what is better? Mostly it depends on how you define better. If you define better as "How can my team move the fastest?" then you want a really strong silo. You know what your inputs and outputs are, and you just build the best system for converting the former into the latter, You don't need to worry about anything or anyone else. It's up to those other folks to get you what you need and to take what you produce and do something with it.

On the other hand, if you define better as "How can I deliver my favorite feature to the customer the fastest?" you want to control everything. You want to control how the inputs are made (and everything that goes into them) you want to control how the feature is delivered. And you probably want to control the selection of features as well.

But there are problems with both. Silos, in the extreme, lead to building the perfect instance of whatever it is your building. That's good. As long as you're building the right thing. But if you aren't aware of what your customer is doing, you're probably building the wrong thing. And if all you do is demand a set of inputs, what happens when they're unavailable, or not available in the right ratio? You end up limiting what you can do because you're starved for inputs.

The other things silos lead to is a loss of responsibility. Everyone is responsible for their part and no-one is responsible for the whole thing. When you try to find out who is responsible for a change you want/need you don't get a straight answer. You get a lot of redirection and finger pointing that often turns into a circular dependency.

On the other hand, ownership has the opposite problem. It concentrates responsibility into one place, and that place becomes a bottleneck. Regardless of how much we tell ourselves we can multitask, there are only so many hours in a day, and there is a limit to how much can be done by that bottleneck. Context switching becomes hard, Things get lost in the context switch. Work gets serialized to make each part more efficient. At any reasonable product scale one person can't be responsible for everything and still be able to respond quickly.

So what's a developer or org to do? As I said at the beginning, it depends. You have resources and constraints and you want to maximize the output (customer value). Once you know what maximize means then it's just an engineering problem. And that's what we're really here for. To solve the engineering problems and maximize our customer value.

We do that by balancing silos (reducing blockers and cognitive load) with ownership (scope) so things keep moving AND continuing to re-evaluate the silos and ownership so that they work in service to the goals, not in opposition.

Agile, 3X, And 4X

Almost 20 years ago a bunch of folks, including Kent Beck released the Agile Manifesto. Waterfall planning stopped being the right way to do things. It took a while, but it, and the internet, changed the way a lot of software is developed. Agile, Scrum, Extreme, and Mob programming are all based on the precepts in the manifesto.

15 years later, Kent had a thought

What if those waterfall folks aren’t wrong, what if they are solving a different problem than I’m solving? What problem is that?

This led to his 3X model, eXplore/eXpand/eBtract. 

1. Explore: Try everything and figure out what you should be doing
2. Expand: You know what you want to do. Things are growing almost faster than you can handle. You fix one bottleneck just in time to deal with the next one
3. Extract: You've got growth under control. Time to extract value from the system. Economies of scale and efficiency dominate.

At different phases you value different things. When exploring, speed of experimentation and noticing results is key. Lots of things get tried and potentially abandoned if they don't work out. Short sprints and rapid iteration with plenty of feedback.

During expand you follow the hot spots. Quick fixes to buy time to give you a chance to do the work required for the next 6-12 months. Don't bother planning beyond that because things are still too fluid.

Extract, on the other hand. is when longer term planning is critical. Where do you need to be 3-5 years out to extract maximum value? Maybe not traditional waterfall with everything carved in stone at the beginning, but not breaking things along the way is crucial.

Uber as a whole is somewhere between 2 and 3, but different groups/teams are at different stages. Ride share (pre-Covid) has rapid, but stable/predictable growth, and we're extracting value. Eats is expanding. The general idea is understood, and enabling growth is key. ATG and Freight are mostly explore, with some expansion thrown in. But even within those large areas, different teams are at different stages.

That's Agile and 3X. What about 4X? Well, ever play any of the Civilization or Command & Conquer games or Settlers of Catan? One of my favorites back then was Stars!. Those are 4X games. eXplore/eXpand/eXploit(eXtract)/eXterminate. And the mental model is basically the same as Kent's 3X model, with a final phase of eliminating your opponents.

You already know how to think of things in the 3X model. The key is recognizing what phase you're in and adjusting your goals/methodology accordingly. So what stage is your team/group/org at, and are you using the correct approach for it?

Context

Words have meanings. We have whole books that do nothing but list words and their meanings. Of course they use other words to define words, so it’s slightly circular, but somehow we’ve managed to make things work.

Now some of those words can have multiple meanings. Sometimes it’s the same spelling but different pronunciation, like lead, You can have lead the element as a noun, or lead the verb as in “The captain will lead us to victory”. Or it can be the same pronunciation, but different spelling, like their , there , and they're . It’s not too bad when writing, but in spoken English it can be a problem.

Which make communication hard, because it’s not just the words and their meanings, it’s the words around them. And the time. And the location. And the discussion itself. Because context matters.

In the ST;TNG episode Darmok that gets taken to the next level. The words are just labels for an entire story. And if you don’t know the story, how can you communicate?

So next time you need to communicate with someone, remember that its more than just the words, its the entire context. Otherwise you’ll end up in a conversation like this tiktok

Now You Have Two Problems

Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems. -- Jamie Zawinski

Of course that's not strictly true. Even if you're not writing Perl you probably use a regex or two every day. After all, grep and ag (the silver searcher) take your input as a regex and use it to scan your file system.

And used correctly a regex is great for detecting and potentially extracting/transforming bits of a string. Need to know if a pattern appears in your strings? Regex is great for that. Need to know if a string is a phone number? There's a good regex for those things.

The thing to remember is that while a regex can scan text and has limited forward/backward looking ability, it's not a parser, and doesn't understand context. Arbitrary nesting of open/close pairs (I'm talking about you HTML) will drive the parser (and the developer) nuts, so just don't do it. If you're actually parsing HTML then use a DOM.

Even when a regex is the right answer, be careful. When your regex starts to become write-only code you've probably gone too far. Don't be afraid to split your regex into parts, capturing a bigger block of text and then using more specific regex's on it. It might not be quite a performant, but it will be more understandable and maintainable, and that's what we're really going for. When the next unsuspecting programmer (you?) looks at it in 6 months will they be able to understand what's happening?

And finally, a regex is a great place for an edge condition to hide. So when possible don't write your own. There are well known regex for parsing common things, email addresses, phone numbers, SSNs, credit card numbers, etc. If the thing you're looking for has a Backus–Naur form, there's probably a regex for it already, so use it. If not, the BNF form will help you generate the regex.

Pass By Value

Golang is pass by value. There is no pass by reference. It will allocate a new variable to hold every parameter passed to a function. Golang does however have pointers. And you can pass pointers to a function. In practice it works pretty much the same as pass by reference in C++. Inside the called function you have a pointer to a value declared/existing outside the scope of the function and can modify it.

This is a good thing. For slices, maps, and other large objects you don't have to copy so much data. And if you want to change the external value of a parameter you need to be explicit about it. It helps folks feel safer when calling functions.

Golang is also strongly typed. Every variable has a type, and there is no default conversion. You can't assign  float64 to an int and expect it to compile. You need to make an explicit conversion. Again, this is a good thing. You might need to type a little more code, but you're never going to be surprised by an unintended truncation of float to int.

Additionally, as strongly typed as it is, it also has structural typing (almost the same as duck typing, but happens at compile time). It's not inheritance, but it lets you specify the interface you need a function parameter to have and then you can pass in any struct that has supplies that interface. It's a big part of idiomatic Golang and lets you do all sorts of cool composition.

However, there is a bit of a loophole in there. Because of the way that Method Sets are defined, a pointer to T (*T) implements all of the methods that T does. So according to structural typing T and *T implement the same method set. So far, so good.

But it bit me the other day. Say you have a function that is going to unmarshal a json blob to a struct. You have to tell it what the struct is, so you pass in a pointer to the struct and then use reflection to do the unmarshalling. Since you don't want to have a new function for every type you use a bare `interface{}` as the parameter type. Everything works fine.

func MergeEntity(entityStructPtr interface{}, patch interface{}) interface{} {

...

if err := json.Unmarshal(entityBytes, entityStructPtr); err != nil {

    panic(err)

}

...

}

Unless you call the function without a pointer. The compiler can't tell the difference between T and *T in this case because they both implement the bare `interface{}` method set so both

MergeEntity(entity1, entity2)

and

MergeEntity(&entity1, entity2)

compile, but the first one throws a panic from json.Unmarshal because entityStructPtr isn't really a pointer.

And our unit testing didn't cover the combination of options that led to that codepath. So we got a runtime crash. And worse, from the message/trace, it wasn't obvious what the problem was. So, I fixed it by adding the missing & AND I added a bit of guard code to panic with a better error message.

paramType := reflect.ValueOf(entityStructPtr)

if paramType.Kind() != reflect.Ptr {

    panic("lib.MergeEntity: entityStructPtr must be a pointer not a value")

}

So the next person who runs into the problem will at least know what the problem is without having to do deep introspection of the code.

It's All Around Scope

We've all heard about different architectural patterns, from layered monoliths to microservices. And they all have their place. The thing that gets me is that many people look at enterprise architecture and think it's fundamentally different than other types of software architecture.

It's not exactly the same, but then, no two architectures are the same. If they were you'd end up with the same results, maybe with a different color scheme. Kind of like the way you can use different themes and make a presentation different. You're not changing the content at all, but it looks different. For some folks that might be enough, but I don't know about you, but I'm always solving new problems, so the old solutions won't work. They may be similar, and there's lots to learn from the past, but it's not just a copy. The same can be said for the role of enterprise architect vs writing a design doc for a function or library. But those are topics for later.

Today's topic is the difference between enterprise architecture in particular and software architecture in general. I maintain that it's really just a matter of scope. Both should be SOLID. Both should be DRY. Both should remember YAGNI.

Done right each of the parts of your microservice will follow the same principles as the overall system, just at a different scale. A set of concurrent go routines and a channel switch is an event based architecture. Calling different interfaces inside a single executable is (logically) the same as using calling the API of another part of your microservice architecture.

In either case you need to look at constraints. Constraints like storage, bandwidth, and execution time. One might be dominated by network bandwidth and latencies, while another might be dominated by bus bandwidth and L1/L2 cache hits, but the idea is the same. It's just a difference of scope.

In either case you need to understand requirements. And not just the explicit requirements, but the non-functional requirements (NFRs) as well. Some of the requirements might be different, but you still need to meet them. The "-ilities" are a thing for both. And it might feel like there are more requirements from the business/marketing/sales/finance departments in enterprise architecture, but I promise you, if you build a library that's too slow/takes too much memory/doesn't have the right feature set someone is going to let you know.

What do you think? Is enterprise architecture fundamentally different from any other software architecture? What did I miss? Let's discuss in the comments.

Git Bisect and Fruit

What do source code management and splitting things in half have in common? git bisect. Remember the murder mystery @andrew.gemmel wrote about? That was the story about how a simple change to a unit test ended up periodically killing the ssh-agent. What he didn't talk about was how we narrowed things down to that commit so we could find the problem.

To do that we used git bisect. If you're ever trying to figure out exactly which commit changed the behavior of your repo from A to B then git bisect is your friend. It's actually pretty simple. You tell git you want to do a bisect with git bisect start, you mark the current revision as good or bad with git bisect good|bad, then you identify the other known case with git bisect bad <sha>. From that point on, git is driving. It will find the middle revision between the two extremes and check it out for you. You then decide if things are good or bad at that revision and let git know with git bisect good|bad as appropriate. git will then bisect the appropriate side of the current revision, check out the middle, and wait for you to tell it the status. The process continues until you've arrived at the revision that caused the repo to go from the good state to bad. Once you know where the problem is you stop the bisect with git bisect reset. And if you can write a simple script to decide if things are good or bad you can even replace the iterative part with git bisect run <script> <args>. Pretty simple and pretty powerful. There are lots of other options, so check them out.

But what has this got to do with fruit you ask? Well, one type of fruit is squash, and squash can help or hinder your bisect. When looking at changes on the main branch, squashing commits (which we do) means that every PR that lands is a single commit, regardless of how many commits it took on the branch to get there. This makes the git history more manageable, ideally means each commit is a working functional change/addition, and bisect has many fewer commits to deal with.

However, there's a tradeoff. Because all of those commits are squashed into a single commit, the code changes involved can be pretty complicated. And in that case when you finally bisect down to the commit that caused the change you may find there are lots of files/chunks that you need to look at to figure out what happened. In that case it might have helped if you had all those individual commits.

If you find yourself in that situation then once you have identified the commit you care about, see if the branch that created it is still around. If it is then you can go to that branch and further bisect since it will have all the individual commits. If not, you're left to wade through all the code and use your deductive reasoning to figure out what happened.

And that's how git, bisect, and fruit are related.

Interviewing Up

You've probably heard the term managing up, and you've probably even done it, but have you ever been interviewing up? When I started at Microsoft hiring me was my boss's last checkbox before he moved on to bigger and better things. So, about 2 weeks after I got through New Employee Orientation and it was clear that I wasn't going to be thrown out, my boss put in his two week notice. 2 weeks later I was added to an interview loop for my new boss. We went through 3 or 4 loops before we settled on someone. Since then I've done that a couple of other times. I've also done similar interviews when making lateral changes inside MS talking to the potential hiring manager.

It's a very different experience from interviewing a teammate or even on IC on another team. For one thing the skill set you're looking for is different. Yes, you want your manager to have the technical chops to understand what you're saying, help you recognize potential issues, and provide pointers, but more and more, Engineering Managers have much more to do than to also be the tech lead for the team.

The role of an EM is to help the team succeed, not do the work. That's even more true for managers of managers and above, so what you really need to understand is how they'll support the team when no-one is around. And those are the things you want to look for. How have they supported their current team and direct reports? How do they approach conflict within the team and between teams? Are they focused on team health, individual growth, or team results? And do they have examples of how they do this, or are they just saying what you want to hear?

How can you find the answers to these questions? Getting them to sort a list of integers or optimizing conference room schedules won't tell you. You need a different approach here.

One of the best questions is how they define success. Is it just meeting goals, achieving business goals, or about team and individual growth? Knowing how they see success tells you what they value.

How do they stay in touch with their directs (and skip levels if that's a thing)? Do they see them as people or work units? How have they recognized and worked with those folks to make things better?

When was the last time you disagreed with/changed your boss's mind. This is a good one for any interview, but particularly for managers. It helps you understand who's side they're on. They team's, management, or the project itself? Are they looking for solutions to point problems, systemic issues, just reduced conflict?

Finally, what's their plan to make themselves replaceable, or at least, how do they think about succession and growth planning? How do they help the individuals on the team grow, and do they make it possible for them to explore new areas and expand their scope?

All of these things can give you insight into what kind of manager you're going to be getting. That's important because they say that people don't leave jobs, they leave managers. So whether you're looking at a new team or a new manager for your current team, you want to set yourself up for success.