Recent Posts (page 40 / 65)

Ideas In The Shower

I don't know about you, but I've solved a bunch of problems in the shower. It's the perfect combination of no distractions, some manual activity, and no immediate cognitive tasks. It gives the brain a chance to wander, sift, and make connections.

Sometimes you need to step away from the problem you're trying to solve and not think about it so hard. Step away from the pressure of a blinking cursor or a blank whiteboard demanding to be filled. That freedom lets you change the scope of your thinking. It lets you focus on the big picture and not get bogged down in the details of syntax and language or where to put the box on the page to avoid crossing lines. occasionally you end up with a Eureka or A Ha moment, but more often, and at least as useful, the broad outline of the solution to your problem will come to you. That doesn't mean that the full solution with all the details is suddenly known. There usually lots of detail work left (and all the language and syntax stuff), but you'll have a framework to hang all that detail on.

Back when we were in an office it wasn't unusual to find me walking around the office quietly or stopping to play a few rounds of darts. It wasn't quite as good as the shower, but it came close. And during those times I was able to come up with approaches to multiple problems. Sometimes solutions that were bigger than what I was originally looking for. Like when I was working on that map visualizer. I was looking for a way to keep the views in sync for the map visualizer and realized that I wasn't talking about two points (look from/look at) and a rotation, but a point, a unit quaternion, and a direction (at/from). It was not only less data, but it also worked for syncing across and between 2D maps, 3D Maps, BirdsEye views and StreetSide image cubes. There was still lots of math to be done to make it work, but that framework gave us the context to build it in.

XY Problem

Every now and then a question/request will come up on Slack or in a conversation that seems odd, but in an attempt to be helpful I'll answer it and a follow-up question or two. And then the next question breaks my mental model of the situation and I do an internal double-take. In those situations my response is usually something along the lines of Wait, what is it you're really trying to do?

And that's an XY problem. The kind of problem where you each step requires a more esoteric and specific solution from a generic tool. When you find yourself in that situation it's usually a good idea to step back and make sure you're approaching things the right way. You'll usually find that if you work with the system instead of against it things will go much smoother.

It often happens when your go-to tool isn't quite the right tool for the job. sed is in fact a turing complete language, meaning you can do just about anything with it, but that doesn't mean you should use it for your problem. It's great for examining/modifying text one line at a time, and is OK for multi-line text, but beyond that use something else. xargs is where I often find myself getting too far down into the weeds.

The other day I was trying to clean up some kubernetes pods, and selecting the right ones was a little tricky. It seemed easy at first, but as the condition got more complicated the amount of escaping and nesting was getting out of hand. I fought with it for a while, consulted the bash grimoire and stack overflow, but eventually realized that the best solution was to stop trying to mash it all into one line and use an external script.

It never quite got to the point of writing a Golang program to handle it, I came really close, and I promised myself that the next time it needs to be extended I'm moving to Golang and taking the opportunity to merge 3 custom scripts into a more generic, extensible program.

Sometimes you just need to go back to first principles. Look at the requirements, look at the non-requirements, and start down a different track with the knowledge you've gained. The trick is to balance the determination to see a plan through as written against the possibly slightly longer time to get a more maintainable, long term solution

Good Design

What I'm NOT Doing

You know what's just as important as what you're working on but almost never gets talked about? What you're not doing and why you're not doing it. And that's a problem. There are a bunch of reasons why describing what you're not doing is just as important as describing what you are.

First on that list is that it helps you decide if a task fits or not. Let's say you're working on a function that takes in an image and calculates a histogram. Simple enough. You do your RFC, you pass around the API and everyone is happy and you begin to implement it. You're almost done, and someone says "We've got this other project that uses compressed images. Why aren't you able to handle that?" You know that wasn't part of the plan, but not everyone knows what's in your head. If your docs explicitly said "This will not handle compressed images then either requirements and schedule would have been adjusted or someone would have built the shared converter. At the very least you wouldn't have been blindsided by the request.

Along those lines, it helps to clearly define domains and separation of concerns. If you know and share what you're not doing, it's much easier to see the gaps between components and fill them if needed. Right now I'm working on a shared Authentication model for users and services that spans clusters. I'm not working on Authorization. It's a very closely related item, and the same group of people will probably be at the core of that too, but it's a very different problem. And our docs are explicit about that. Yes, we know it's coming and we know how it will fit in, but we're not working on it now.

Another way non-goals can help is when you're defining the actual task list. You have a set of goals, and you have an idea of what the end-state looks like, but something about it isn't quite right. And you keep adding new requirements to try and bend the result into what you're looking for. You end up with some really specific rules. Something like "schedule the meeting before 3:00 unless it's a Friday, then do it before Noon, but don't schedule it for early on Monday, or around lunch". That might be easier said as "Don't schedule the meeting when people are catching up on old work or will need to leave before we're done, even if it runs long" That's much simple and covers the real requirement.

So next time you're defining what you're doing, regardless of the scope, make sure you include the things you aren't doing and don't want to happen as well.

Planting A Tree

In some ways refactoring code like planting a tree. Not just the growing, branching, and pruning aspects (although that's something to talk about at another time), but also, like the old proverb about the best time to plant a tree being 20 years ago, the best time to refactor your code was back during the design phase. If you had made the change then you wouldn't be in the mess you're in now, and you'd be able to add all those new features with minimal friction.

The second best time is now. Sometimes it's actually the best time. You've come this far without needing the changes, so it wasn't too big of a mistake. You know a lot more about the situation, so you can make a better decision. You can set yourself up for the future.

Back when I was still at Microsoft we we built a data validation/QA tool to help people build up truth sets, easily view maps and fix/mark issues, and compare the 2D, 3D, and StreetSide (SS) views of the same area. And one of the things we knew from the beginning was that we needed to have a way to keep the viewpoints in sync. But there were issues with that. We could think of a bunch of different ways to keep 2D and 3D in sync, with different leaders/followers and considering if you were looking from a point or looking at a point and all of the different combinations of them. Some of those combinations were familiar to our staff, but some weren't, and. And none of them had any clue about how they would add a SS image cube to the mix, let alone be able to draw in one view and see it in the others, in real time.

So our solution was to handle the 2D/3D case and not worry about the SS cubes. We made the 2D/3D part clean, and applied pretzel logic to make the SS view work. That let us get the tool out to our production team so we could add value and find out exactly what they thought, where the pains were, and what more we needed to do.

Then, with that information in hand, we did the refactor and made SS really be just another view. And since we did it when we knew enough to make it simple to use but easy to extend, when someone came up with the idea to change our SS cubes into a voxel based world, that view just slipped right in and worked without a hitch.

If you want more information on exactly how to refactor, check out refactoring.com.

https://www.mindfueldaily.com/livewell/inspiring-chinese-proverb/

https://t.co/bF409g4pfl?amp=1

https://uberatg.slack.com/archives/CLVTB4W20/p1576251000000100

Situational Awareness

We all have biases. Some we know about, and some we don't. One of the benefits of being data driven is that, if you pick your metrics correctly, you can avoid those unconscious biases.

Going back to my aviation roots, that's why one of the leading causes of aircraft accidents is controlled flight into terrain. Basically it's the pilot not knowing there's a problem until it's too late to do anything about it. In aviation it's most likely to happen when flying on instruments. In those cases your eyes and ears will lie to you, telling you you're flying along straight and level, when actually you're spiraling into the ground. That's what we think happened to JFK Jr. off the coast of Martha's Vineyard. It's a mindset problem. It's a "what you know that just ain't so" problem. Kennedy was a relatively new pilot, and as experience shows, and the Dunning-Kruger effect explains, that's one of the most dangerous times for a pilot. He "knew" he was doing fine, so he ignored what his instruments were telling him.

But what does this have to do with software development? The same thing happens to us. We "know" things are true, so we don't take the time to make sure they really are. We make network requests, and we check the result to make sure things worked. But what about timing? Yes, the response is almost always very fast, but depending on the environment, it might be 30 seconds or more before the request times out and you get a failure message. Sure, you'll do the right thing, but it will be too late.

We ran into this with our healthchecks. We were doing user level testing of a distributed system, starting a job and making sure we could stop it. Everything was fine, usually. But every once in a while that process would take 10+ minutes. By itself that might or might not be OK, but because it was a blocking call in a set of serial tests it meant everything got delayed. Which threw off metrics, which caused other problems.

So we took it as a learning experience. We ended up fixing it in two ways. First, we parallelized the tests, which cut down the total time in the happy path. Second, we added a timeout on the test, so we'd know it took too long, mark it as a failure, and move on. We also did a quick review of other systems and added some other timeouts we hadn't realized we needed.

So what's your blind spot, and how can you close it?

Careers

Last week at the Core Platform All-Hands meeting I did a short presentation on Owning Your Career, but I was having some internet issues and there were some dropouts so I figured I'd take some space here to spread the info a little wider and make sure I got all my points out. As a refresher, I started with a review of what a level is and that it's really a reflection of your scope of influence.

The other two key things to remember about level is that it's recognition of consistently demonstrating the core competencies, not an aspirational motivator, and that while the arc of the competencies tends up and to the right, it's not always a steady increase. How you are measured against those competencies is impacted by not only experience, but also role changes, job changes, and external events. As I described in the slides, over my 32 year career thus far, my effective ability to demonstrate those competencies dipped quite a few times. Each time I was able to recover, but it does have an impact.

But all of that was really a lead-in to the core of the matter, which is that you are responsible for your career. Your manager and your mentor(s) can help. but it's up to you to know what you want and drive to get there. To that point, the key takeaways are:

1. It's Your Career - There are lots of people who want to help, but the key is help, you own your career
2. Be Valuable, Not Critical - You want to be adding value to the company. Do important things and make a difference, but if you're critical in one spot you can't move and grow.
3. Find Your Balance - Understand what you want and what makes you happy/fulfilled. Recognize that it might change over time and that's OK. I thought I wanted to eventually be a Director or similar, but as  I learned when I was a manager of managers, that's not me, so I don't do that anymore.
4. Personal Growth, Not Career Growth - This goes back to the original premise, that level = scope of influence and it's a recognition of what your scope is. If you spend your time chasing the level you'll miss the opportunities to learn and grow along the way. Grow your scope and the career will follow.
5. Understand the Big Picture - Finally, keep the bigger picture in mind. Don't get hung up on bumps in the road or detours. There's something to learn in those situations as well. The overall arc is up and to the right, but it's not constant.

And finally, the most important of the action items. Own your career. Figure out what you want your career to look like and talk to your manager about opportunities to make it happen. Understand the differences between how you see yourself, how others see you, and the engineering competencies by level.

A couple of years ago I put together the eng-stories framework, which I used to describe the arc of my career. I've found it to be a really good way to frame those discussions and identify the differences. Once you know the gaps you can work to close them. Without that knowledge it's impossible to know if you're working on the right things.

If you've got any questions about levels and scope, the arc of your career, using the eng-stories tool, or anything else related, happy to discuss in email or 1:1.

DDD and Microservices

Last week there was a post on Uber's engineering blog about domain oriented microservice architecture. The first thing that stands out to me is that Jaeger graph of everything talking to everything. When I started at Uber 5 years ago there were more services than engineers in the graph. While microservices, reducing span of control and isolation, are good things, like anything else, you can take them too far. As the article notes, having to wade through 50 services and 12 teams to debug a single issue with a rider pickup is a bit much.

On the other hand, that level of isolation also allows for rapid growth and experimentation. You can change one thing and not impact the others. With a good service mesh you can bleed off a very limited amount of traffic for a live A/B test and see how it goes. You can let measured, real-time results drive your decisions.

But to me the most interesting thing wasn't the design itself or how we got there, but the outside analysis. Specifically the parts where folks were saying "Domain Driven Design has been around for years and they're claiming to have invented it." I know a bunch of the folks involved and no, they're not claiming to have invented it. What they did do was figure out a way to take best practices and retroactively apply them at scale, and that's no trivial task.

So the question we should be asking ourselves now is "What industry/academic best practices and methods should we be aware of?" That doesn't mean we should go chasing off after then next shiny new hotness. Just that we need to be aware of what's happening in the industry at large and use that to help guide our long term plans.

Kicking It Off

Inertia is a thing. Not just for spherical chickens on a frictionless surface in a vacuum or 3 ton robots. It’s also a thing in organizations. And one of the places you’ll see this the most is when starting a new project that requires multiple groups to change direction. Some groups don’t think it applies and make no changes, some over-rotate, and the rest end up somewhere in between.

But there are ways to provide guardrails and help people stay on track. And they all pretty much come down to making sure that there is a shared vision of what the result is going to look like. So write it down and share it. That doc, your project kick-off doc, is going to frame the work and provide the context needed to allow everyone working on the project to make all of the small, daily decisions needed to make the project a success.

• Executive Summary
• Problem Statement
• Product Vision
• Minimal Viable Product (and follow-on phases)

Executive Summary

A paragraph (two at most) with what you’re doing, why you’re doing it, what done looks like, and the major milestones along the way.

Problem Statement

An outside looking in view of why this needs to be done. Define the customer and what their problem/pain point is.

Product Vision

Your proposed solution. Not the technical details, although some might be needed, but again, outside looking in. Show how things are different for the customer. How their life is better/easier with this change.

Because the people that made the plan aren’t going to be part of every decision that gets made. They certainly won’t be in the room when the decision is made, so make sure everyone involved knows why the big decisions have been made so they can align their decisions with them.

Another way to frame things is Amazon’s Working Backwards approach. Instead of a kick-off doc, they start with the press release for the product. And when you get down to it, a good press release contains all of the same information as what I just described for a kick-off doc, An overview, the customer problem, how this new thing solves customer problems, and teasers for the next great thing.

So before you start on your next big thing, make sure you define what you’re doing, why your customers need it (even if they don’t know it), and how they’ll be happier when you’re done.

C Shell, Z Shell ...

I'm in a quandary. MacOS has been on Bash 3.2.X for a long time, while Ubuntu is on 4.4.X. And instead of upgrading Bash, the folks at Apple have decided to switch the default shell to `zsh`. And to make matters worse, they've made it very hard to upgrade bash and have the default version be something else.

So what's a developer to do? And I can hear a bunch of you saying "Get an Ubuntu machine". I've thought about it. I even asked for one, especially since my current MBP is, shall we say, sick, but all they offered was a little 13" MBP. I've also got an Ubuntu desktop in the office I ssh into all the time. If nothing else it's got better upload speed than going through the VPN from home.

So here's my dilemma. Sort of upgrade Bash on my current MBP and figure out how to beat it into submission, which has the benefit of matching what our official build systems and clusters run, but struggle against our Apple overlords every time they update the OS, or switch to zsh, which, admittedly is a nice experience, and will just work across OS updates, but means my local scripts either still need to use Bash 3.2.X or I have to either duplicate them for two shells, restrict them to the subset of commands that work with both, or struggle against Apple and upgrade the version of bash.

I'm thinking the best thing to do is just figure out (and document) all of the machinations to get my MBP running Bash 4.4.X as it's primary shell and just keep it that way. What do you all think, and if you've done it, got any documentation?