Recent Posts (page 37 / 65)

by Leon Rosenshein

Pass By Value

Golang is pass by value. There is no pass by reference. It will allocate a new variable to hold every parameter passed to a function. Golang does however have pointers. And you can pass pointers to a function. In practice it works pretty much the same as pass by reference in C++. Inside the called function you have a pointer to a value declared/existing outside the scope of the function and can modify it.

This is a good thing. For slices, maps, and other large objects you don't have to copy so much data. And if you want to change the external value of a parameter you need to be explicit about it. It helps folks feel safer when calling functions.

Golang is also strongly typed. Every variable has a type, and there is no default conversion. You can't assign  float64 to an int and expect it to compile. You need to make an explicit conversion. Again, this is a good thing. You might need to type a little more code, but you're never going to be surprised by an unintended truncation of float to int.

Additionally, as strongly typed as it is, it also has structural typing (almost the same as duck typing, but happens at compile time). It's not inheritance, but it lets you specify the interface you need a function parameter to have and then you can pass in any struct that has supplies that interface. It's a big part of idiomatic Golang and lets you do all sorts of cool composition.

However, there is a bit of a loophole in there. Because of the way that Method Sets are defined, a pointer to T (*T) implements all of the methods that T does. So according to structural typing T and *T implement the same method set. So far, so good.

But it bit me the other day. Say you have a function that is going to unmarshal a json blob to a struct. You have to tell it what the struct is, so you pass in a pointer to the struct and then use reflection to do the unmarshalling. Since you don't want to have a new function for every type you use a bare `interface{}` as the parameter type. Everything works fine.


func MergeEntity(entityStructPtr interface{}, patch interface{}) interface{} {

...

if err := json.Unmarshal(entityBytes, entityStructPtr); err != nil {

    panic(err)

}

...

}


Unless you call the function without a pointer. The compiler can't tell the difference between T and *T in this case because they both implement the bare `interface{}` method set so both

MergeEntity(entity1, entity2)

and

MergeEntity(&entity1, entity2)

compile, but the first one throws a panic from json.Unmarshal because entityStructPtr isn't really a pointer.

And our unit testing didn't cover the combination of options that led to that codepath. So we got a runtime crash. And worse, from the message/trace, it wasn't obvious what the problem was. So, I fixed it by adding the missing & AND I added a bit of guard code to panic with a better error message.

paramType := reflect.ValueOf(entityStructPtr)

if paramType.Kind() != reflect.Ptr {

    panic("lib.MergeEntity: entityStructPtr must be a pointer not a value")

}

So the next person who runs into the problem will at least know what the problem is without having to do deep introspection of the code.

by Leon Rosenshein

It's All Around Scope

We've all heard about different architectural patterns, from layered monoliths to microservices. And they all have their place. The thing that gets me is that many people look at enterprise architecture and think it's fundamentally different than other types of software architecture.

It's not exactly the same, but then, no two architectures are the same. If they were you'd end up with the same results, maybe with a different color scheme. Kind of like the way you can use different themes and make a presentation different. You're not changing the content at all, but it looks different. For some folks that might be enough, but I don't know about you, but I'm always solving new problems, so the old solutions won't work. They may be similar, and there's lots to learn from the past, but it's not just a copy. The same can be said for the role of enterprise architect vs writing a design doc for a function or library. But those are topics for later.

Today's topic is the difference between enterprise architecture in particular and software architecture in general. I maintain that it's really just a matter of scope. Both should be SOLID. Both should be DRY. Both should remember YAGNI.

Done right each of the parts of your microservice will follow the same principles as the overall system, just at a different scale. A set of concurrent go routines and a channel switch is an event based architecture. Calling different interfaces inside a single executable is (logically) the same as using calling the API of another part of your microservice architecture.

In either case you need to look at constraints. Constraints like storage, bandwidth, and execution time. One might be dominated by network bandwidth and latencies, while another might be dominated by bus bandwidth and L1/L2 cache hits, but the idea is the same. It's just a difference of scope.

In either case you need to understand requirements. And not just the explicit requirements, but the non-functional requirements (NFRs) as well. Some of the requirements might be different, but you still need to meet them. The "-ilities" are a thing for both. And it might feel like there are more requirements from the business/marketing/sales/finance departments in enterprise architecture, but I promise you, if you build a library that's too slow/takes too much memory/doesn't have the right feature set someone is going to let you know.

What do you think? Is enterprise architecture fundamentally different from any other software architecture? What did I miss? Let's discuss in the comments.


by Leon Rosenshein

Git Bisect and Fruit

What do source code management and splitting things in half have in common? git bisect. Remember the murder mystery @andrew.gemmel wrote about? That was the story about how a simple change to a unit test ended up periodically killing the ssh-agent. What he didn't talk about was how we narrowed things down to that commit so we could find the problem.

To do that we used git bisect. If you're ever trying to figure out exactly which commit changed the behavior of your repo from A to B then git bisect is your friend. It's actually pretty simple. You tell git you want to do a bisect with git bisect start, you mark the current revision as good or bad with git bisect good|bad, then you identify the other known case with git bisect bad <sha>. From that point on, git is driving. It will find the middle revision between the two extremes and check it out for you. You then decide if things are good or bad at that revision and let git know with git bisect good|bad as appropriate. git will then bisect the appropriate side of the current revision, check out the middle, and wait for you to tell it the status. The process continues until you've arrived at the revision that caused the repo to go from the good state to bad. Once you know where the problem is you stop the bisect with git bisect reset. And if you can write a simple script to decide if things are good or bad you can even replace the iterative part with git bisect run <script> <args>. Pretty simple and pretty powerful. There are lots of other options, so check them out.

But what has this got to do with fruit you ask? Well, one type of fruit is squash, and squash can help or hinder your bisect. When looking at changes on the main branch, squashing commits (which we do) means that every PR that lands is a single commit, regardless of how many commits it took on the branch to get there. This makes the git history more manageable, ideally means each commit is a working functional change/addition, and bisect has many fewer commits to deal with.

However, there's a tradeoff. Because all of those commits are squashed into a single commit, the code changes involved can be pretty complicated. And in that case when you finally bisect down to the commit that caused the change you may find there are lots of files/chunks that you need to look at to figure out what happened. In that case it might have helped if you had all those individual commits.

If you find yourself in that situation then once you have identified the commit you care about, see if the branch that created it is still around. If it is then you can go to that branch and further bisect since it will have all the individual commits. If not, you're left to wade through all the code and use your deductive reasoning to figure out what happened.

And that's how git, bisect, and fruit are related.

by Leon Rosenshein

Interviewing Up

You've probably heard the term managing up, and you've probably even done it, but have you ever been interviewing up? When I started at Microsoft hiring me was my boss's last checkbox before he moved on to bigger and better things. So, about 2 weeks after I got through New Employee Orientation and it was clear that I wasn't going to be thrown out, my boss put in his two week notice. 2 weeks later I was added to an interview loop for my new boss. We went through 3 or 4 loops before we settled on someone. Since then I've done that a couple of other times. I've also done similar interviews when making lateral changes inside MS talking to the potential hiring manager.

It's a very different experience from interviewing a teammate or even on IC on another team. For one thing the skill set you're looking for is different. Yes, you want your manager to have the technical chops to understand what you're saying, help you recognize potential issues, and provide pointers, but more and more, Engineering Managers have much more to do than to also be the tech lead for the team.

The role of an EM is to help the team succeed, not do the work. That's even more true for managers of managers and above, so what you really need to understand is how they'll support the team when no-one is around. And those are the things you want to look for. How have they supported their current team and direct reports? How do they approach conflict within the team and between teams? Are they focused on team health, individual growth, or team results? And do they have examples of how they do this, or are they just saying what you want to hear?

How can you find the answers to these questions? Getting them to sort a list of integers or optimizing conference room schedules won't tell you. You need a different approach here.

One of the best questions is how they define success. Is it just meeting goals, achieving business goals, or about team and individual growth? Knowing how they see success tells you what they value.

How do they stay in touch with their directs (and skip levels if that's a thing)? Do they see them as people or work units? How have they recognized and worked with those folks to make things better?

When was the last time you disagreed with/changed your boss's mind. This is a good one for any interview, but particularly for managers. It helps you understand who's side they're on. They team's, management, or the project itself? Are they looking for solutions to point problems, systemic issues, just reduced conflict?

Finally, what's their plan to make themselves replaceable, or at least, how do they think about succession and growth planning? How do they help the individuals on the team grow, and do they make it possible for them to explore new areas and expand their scope?

All of these things can give you insight into what kind of manager you're going to be getting. That's important because they say that people don't leave jobs, they leave managers. So whether you're looking at a new team or a new manager for your current team, you want to set yourself up for success.


by Leon Rosenshein

Not Even Wrong

Sometimes I run across articles that make me want to reach through the screen, find the author, and say "Stop it. You missed the point entirely and you're just making things worse." The article The essential difference between microservices and APIs is one of those cases.

To use an analogy, consider an article titled The essential difference between Ethernet and communication theory. And then you read the article and it talks about how Ethernet is better than the old networks because you can get better overall throughput if you have relatively sparse transmitters. It doesn't name the older network technology, just talks about how much better Ethernet is. Yes, Ethernet can do better than a TokenRing network, but that's not what you said you're going to be talking about.

Basically, the title compares an implementation of one concept to a different (but related) concept itself, then goes on to compare two implementations. Which does a disservice to both implementations and the concept.

Microservices and monoliths are architectural design patterns, and the both have their pros and cons. For example, microservices let you deploy small pieces and scale them up/out individually, but are hard to test end 2 end. Monoliths are easy to test and debug end 2 end, but hard to scale up/out. There are lots of other differences and other options besides those two, but that's a whole different topic. And choosing the correct implementation is important if you expect your system to be flexible enough to last. (If you want to dive deeper let me know and maybe we can do a Kata event with your team)

APIs, on the other hand are all about high level architecture and design decisions. Your API (and the way this article is written we're talking about the public API) defines what you can do from outside the system and how you interact with it. It defines the interface but explicitly does not define the implementation. You could have a single monolith, you could have an API gateway over microservices, or something completely different behind the gateway. The whole point of an API is that the consumer of the API doesn't need to know (or care) what the implementation is. And defining the correct API for a project is critical to making it easy for others to use, easy to expand, and easy to adapt to future requirements.

So, the next time you need to compare two things, make sure you know what you're comparing and make sure the things you're comparing are, in fact, comparable. Otherwise you might find yourself in this situation. Or, as Wolfgang Pauli is supposed to have said,

That's not right. It's not even wrong

by Leon Rosenshein

Happy Birthday

Way back in the depths of time (fall of 1988) I was a new grad working for a 3rd tier aerospace company called Eidetics and I got the task of connecting an Apollo DN10000 to an SGI 70GTX to create my first flight simulator. It turned out that we had a cockpit mockup that was used for instrumentation development and a batch simulation system that was running faster than real-time since the latest CPU upgrade. _Unfortunately_ the Apollo network we had used a proprietary token ring network, and the SGI was sitting in the corner by itself. So what's a new college grad to do in a time like that? I became a network engineer/administrator.

Of course first I had to figure out what a network was, and what could be used to connect two different systems, one System V Unix and one that claimed to be "Just like real Unix" but wasn't. Luckily there was a relatively recent addition to the networking world designed for rapid communication between disparate systems, Ethernet. So I got out a catalog, bought some thick-net backbone, a few transceivers and drop cables and hooked it up. Then I learned about terminating resistors. And kernel drivers. And TCP/IP.  And single source of identity in a cluster. And IRC. And email management. It was a great learning experience, and it taught me the importance of stepping up to a task I might not have known how to do when I started, but was able to figure out along the way.

Of course Ethernet has spread and grown since then. Gigabit ethernet over Cat5 twisted pair wires is a thing, and no-one blinks at connecting 10s of thousands of computers in a datacenter as a single network/cluster. Now we have Ethernet clusters riding around in cars that have more devices and compute power than the biggest network Eidetics ever put together.

So Happy Birthday Ethernet, and thanks for everything you've done for us.

by Leon Rosenshein

OKRs and KPIs

Here we are in the middle of planning and people are talking about OKRs and KPIs, but what's the difference between them and when should you use one over the other?

Starting from the definition, OKRs are Objectives and Key Results, while KPIs are Key Performance Indicators. Sounds similar, no? Key Results and a Performance Indicators sure sound like the same thing. And your KPIs might be some of your Key Results, but they're absolutely not Objectives

And that's where the important difference is. Very much like the difference between a vision statement and a mission statement. A vision statement defines the goal, the end state, the objective. On the other hand, a mission statement defines what you do to enable the vision, and the KPIs/Key Results measure how well you're doing in that mission.

Or put another way, the KPIs are how you define done. Like everything else we do, the most important thing to do is decide what to do (the objective), and the second most important is to decide what it means to be done/succeed (the key results).

Objectives come in a few flavors, but the big ones are existential and aspirational. Existential ones are the ones that you need to do to keep going. Things like RTR and NVO. Down here in the engine room it's things like "Increase Capacity" and "Secure the Data". If we fail to do those things we hit a wall and everyone has problems.

Aspirational objectives, on the other hand, are longer term, and progress is important, but not reaching the end state at the specified time is not fatal. These are things like "Replacing personal car ownership" or, for Infra, "Workloads seamlessly and transparently transition between on-prem and cloud datacenters". We won't get there this year, let alone this quarter, but we'll make progress.

Similarly, KRs/KPIs often fall into two categories. actionable and vanity. Actionable metrics let you know what's going on quickly and accurately and can help you make decisions through uncertainty. Vanity metrics look and sound impressive, but don't really tell you much. As an example, consider a simple website, It's got a handful of static pages and a few images from a database. Vanity metrics are epitomized by that hit counter at the bottom of the page. It just keeps going up. Millions and millions of views. Or maybe you're a little more advanced and you're tracking P95 page load times. And it's always under 100ms. That's great, right?

Not necessarily. On the surface those metrics look wonderful, but they don't tell you anything. Actionable metrics would be unique daily user counts, or even better, unique daily user trends. PLTs are important, but since you've got a 50ms timeout on your database calls that p95 doesn't mean much. You should be looking at 2XX vs 4/5XX responses. And not just raw numbers, because 1 error out of 3 is really bad, but 1,000 errors out of 100,000,000 isn't bad at all (but still might not be acceptable), but could indicate a brewing problem you need to take action on.

So as you think about planning, not just for Q4, but longer term (1, 3, or 5 year plan) as well, think about how you choose your Objectives and Key Results and the Key Performance Indicators you're going to use to measure yourself. Are they really driving you where you want to go?

by Leon Rosenshein

OOP Isn't Dead

I've been hearing that object oriented programming is dead for a while now. Like 15 years or so. But I don't believe it. I don't believe it for a bunch of reasons, but the simplest are that it's still around, and it's still useful.

That's not to say that there aren't problems. There are problems with everything. With OOP the problems can get pretty tangled and it can be almost impossible to sort them out without refactoring everything. Usually the problem can be traced back to one of two things. Either an explicit decision that everything is an object, or if only some things are objects, a poor choice of naming/segregation of the hierarchy.

When everything is an object nothing can stand alone. You want your objects to interact, so they need to understand each other, at least partially, Or you create some other object that knows about the different objects so it can handle the interactions. Despite your best efforts you end up with a big ball of mud.

For the second problem, having to make decisions with imperfect knowledge is a given, especially if you're being honest with yourself. And invariably something will come up that makes your current hierarchy less than ideal. Eventually you get to a point where what you really want is multiple inheritance, but that never works out well.

On the other hand, if you go back to how OOP was originally defined there's no mention of inheritance or polymorphism. It was defined as a group of small things that communicated by message passing, or, in today's language, a distributed system of microservices.

And if you consider a function call and return as message passing (which it is), then there's still a lot of life left in OOP. What OOP is really about is encapsulation, decoupling, and late binding, which are all things that make your system more stable, easier to test, and more resilient to change.

So the next time someone tells you OOP is dead, tell them no, we're just now actually getting to use it they way it was intended.

by Leon Rosenshein

What is Scrum?

Do you do Scrum? Do you have an official ScrumMaster who isn't your EM/PM? Do you rotate the job of Scrum Master? And what's the job of your Scrum Master anyway? According to the official scrum guide,

The Scrum Master is responsible for promoting and supporting Scrum as defined in the Scrum Guide. Scrum Masters do this by helping everyone understand Scrum theory, practices, rules, and values.

The Scrum Master is a servant-leader for the Scrum Team. The Scrum Master helps those outside the Scrum Team understand which of their interactions with the Scrum Team are helpful and which aren’t. The Scrum Master helps everyone change these interactions to maximize the value created by the Scrum Team.

The Scrum master's responsibility is to make the Scrum better. Both inside the team and through team interactions with others. The Scrum Master role is more about kaizen than anything else. It's not sprint management. It's not translating business requirements to a User Story. It's not presenting the team's work at demo days or delivering a burndown chart.

I've spent a lot of time on teams that said they were doing Scrum, but really were just doing two week planning cycles and daily standups. That's not Scrum. It can be agile, and it can be effective, but it's not Scrum.

Personally, the thing I like best about Scrum, and something you can do with "doing Scrum" is the retrospective. After a cycle, however you define it, look back at what you've done. And more importantly, look back at how you did it. What were the process problems, not the technical ones. What were the gaps in User Stories that you didn't notice until you were busy implementing them? How could you have found those gaps sooner? What are you going to do next time to prevent those gaps?

Because Scrum, at its core, falls under the scope of the Agile Manifesto. As such, it's about people over process and continuous improvement. So blindly following the forms of Scrum, without following the principles, isn't Scrum at all.


by Leon Rosenshein

How Big Is That File Anyway?

There are lots of hard problems in computing, but you wouldn't think counting bytes is one of them. Counting bytes is easy, right? If you want to know how big a file is just count the number of bytes in it.

Or maybe not. It depends on what you're counting and what you're going to do with the number. If you want to know how many bytes of RAM it will take to hold the data in a file (assuming it's just a blob of data) then that might be correct. But what if it's a compressed file?

Or maybe you want to know how much disk space you'll get back if you delete it. In that case you need the number of bytes in the file, rounded up to the next whole block size. Because your disk allocates things by block. Different OS's and different devices have different block sizes, so the space used on one storage device could be different than that on another, even on the same computer.

You can't forget the overhead of actually remembering where you put that file and it's blocks. That data gets stored on the disk somewhere, usually with multiple copies.

If you're trying to figure out where your disk space went, things get even more complicated. How do you count a soft link? What about a hard link? What are you really measuring, disk space used by a directory, or how much data you would transfer if you copied the directory?

And what if you have file versioning enabled (at the OS level)? Windows Shadow Copy/ZFS/LVM Snapshots all take space. Is that included in the file size? Should it be deleted when you delete a file? Replicated file systems like HDFS make this particularly complicated by sometimes reporting the number of bytes in a file and sometimes reporting the total bytes used for all replicas.

Or, to paraphrase Clausewitz, Everything in computing is very simple. But the simplest thing is difficult.

https://devblogs.microsoft.com/oldnewthing/20041228-00/?p=36863