Recent Posts (page 38 / 65)

Excuses, Excuses

Sometimes it's so tempting to not test your changes. You're in a rush, it's just a config change, it's a small change, or maybe you know it has no side effects. And if it's late on Friday or you're about to go on vacation and who has time? What could possibly go wrong?

Everything, that's what. If you've been around for a while you might remember the entire SJC datacenter going dark over a 20 minute period. It was just a simple change. To an IPTABLES config. That got distributed far more widely than it should have. With no oversight. By mistake.

The bottom line is that you do need to test all changes. Or at least run the automated tests you already have (you do have them, right?). And get code reviews. Even on the simple things.

But just in case you need it, here's a quick set of common excuses for not testing things. What's your favorite excuse?

Code Golf

Code golf is fun and all, and sometimes it can be fun to see just how terse you can write something and still have it work but it's a good idea to

Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live. Code for readability.

The comma operator in C/C++ is a great example. You can save a handful of keystrokes, but it's almost never a good idea. 

Another place that people are often a little too terse is bash scripting. Especially when you're sharing scripts or saving them for future use. When you're writing a script it's very tempting to stop as soon as it's completed its task once. And if you delete it right after that it's probably OK. But if you're going to share it or save it for future use experience says that you probably want to be a little more verbose. And probably a little more rigorous. Maybe even a lot more.

Here's a few good ideas to use when building scripts that are going to be used multiple times. Especially if they're going to be used by others on random machines as part of a complex workflow.

Fail Fast: Use set -euo pipefail to make your script stop as soon as something goes wrong so you don't make things worse

trap: Use handlers to ensure cleanup in error conditions. You can handle signals, ERR, and EXIT something like:

on_error() {

  # Log message and cleanup

}

trap "on_error" ERR

shellcheck: is part of the `rNA` toolset. Use it and pay attention to the results. Consistency is a good thing.

Use default values for variables where possible: Bash variable expansion lets you get the value of a variable or a default if it's unset. You can use it to make positional command line variables optional with something like

FOO=${2:-default} # If command line option 2 is not set or null, use default.

Include common functions via shared files: The `rNA` repo has some common helpers to deal with MacOS vs Ubuntu. They might be helpful. You might have something similar for your team.

Clean up after yourself: Consider using a `/tmp` folder then deleting it when you're done. Cleanup would be a good use of a trap.

What are your favorite bash scripting tips?

Kata Time

Who wants to put together a team and enter O'Reilly's Architectural Kata competition? If you're not familiar with the idea of architectural katas check out the links below. Basically it's a way to respond to these questions.

How do we get great designers? Great designers design, of course.

    -- Fred Brooks

So how are we supposed to get great architects, if they only get the chance  to architect fewer than a half-dozen times in their career?

    -- Ted Neward

Basically you get a one page definition of the problem, access to the "customer" and a short time (typically 2 hours or so) to come up with an architecture and identify the key technologies/components you're going to need to solve the problem. It's a lot of circles, boxes, and arrows on a whiteboard and justification for the choices as well as the potential areas of problem. It's also about being able to present your architecture quickly and clearly to a group of people who haven't been studying the problem and don't have all the context you do.

They're a lot of fun. I've done them a bunch of times and led a few sessions at the various ATG sites and everyone had a great time and learned a lot along the way. If you're interested in doing this let me know in the comments. If your team(s) are interested in doing a session outside the O'Reilly event let me know and we can work something out. It's a good learning/team building exercise any way you slice it.

Visualizing Code

Code is complex. Understanding complex code is either complex squared or complex to the power of complex, and I lean towards the latter. So what's a developer to do? I've thought about this on and off for a while now, and one of the best descriptions of how I "see" big projects is described in this blog post.

Using those metaphors, I'm definitely a mapper, with some packer thrown in for the little nuggets of things that don't fit onto. I also like the idea of merging the spatial relationship of the code with the temporal. For me it helps with the mapping of the physical (code files) into the virtual space (mental model) and the domain space.

The visualizations in the blog speak to me, and I really want to try out the tool, but the tool isn't open source yet, and at this point I'm afraid it never will be :( The tool for visualizing concurrency was finally open sourced, so there is hope.

Meanwhile, how do you get a feel for big new codebases?

Multiple Oopsies

Sometimes things go wrong. If they go wrong in unrecoverable ways it can be good stop immediately and let someone know. In Go that's typically handled by returning an Error from your function. Straightforward enough.

Sometimes tough, multiple things go wrong, but the actions you're taking are independent enough that it makes sense to keep going. What do you do then? You could keep a list and do some formatting magic or, you could use Uber's shared multierr.

The errors returned by the multierr methods implements the Error interface, so you can just drop then in wherever you need it. The benefit comes at the end when you check for/display errors. We use this in the infra CLI in many places. The one you're probably most familiar with is in infra auth refresh. If you've got multiple AWS profiles defined the tool will try to refresh all of your AWS creds. It's possible the service is down, but the more common error case is that there's a problem with one or more of your profiles. In that case we just add the error to the multierror and keep going. That way as much useful work as possible is done and all errors are noted and returned to the user. Then when everything is done we just check for an error and print it out if needed.

It's a neat pattern. Very useful when you have tracking multiple, independent streams of work. Especially if they're happening in parallel.

Hiring

Part of the return to normalcy, whatever that means in 2020, is a return to hiring. Open positions are starting to show up, recruiting is ramping up, and interview requests are starting to show up in our calendars. Making good hires is one of the most important things we can do for the long-term health and success of our teams and the overall organization. Not making bad hires is even more important.

I've been at Uber over 5 years now and I've been a Bar Raiser for about half that. Back at Microsoft I was an "As Appropriate" interviewer, which is roughly the same thing. Between college campus, phone screens, and on-site interviews I've done over 2000 interviews in my career. All of which is to say I've got some experience doing it.

While I had done interviews before Microsoft, it was at Microsoft that I spent enough time interviewing and hiring that I learned and formalized the framework I still use today. One of the things I learned is that interview prep is just as important for the interviewer as it is for the person being interviewed. And it starts with the hiring manager (HM).

The HM needs to work with the recruiter to define the role and the requirements to make sure the right candidates are brought in, and often needs to help with the sourcing and filtering. After all, no one knows the requirements better than the HM. And once candidates have been identified the HM is responsible for interview loops. Not the actual scheduling, but making sure that they run smoothly, all of the areas are covered, and that the candidate has a positive experience, regardless of the outcome. The HM not only provides the scheduler with a pool of interviewers, but once the loop is set, makes sure that each person knows what area they're supposed to cover, both technically and culturally.

That's when the interviewers get involved. When you know what you're going to cover you can start your preparation. Good questions are critical, and making sure you don't duplicate things on a given loop is important. What makes a question good is depth. Not complexity, although that's sometimes a part, but layers. You want to be able to go one level deeper into the problem. You're looking to understand the candidate's reasoning. Asking why a choice was made, probing deeper into how choices are made, because that reasoning is at least as important as the choice. Maybe there's something you didn't think of or maybe something the candidate didn't think of. When you provide some new information/constraint, how does the candidate incorporate that into the decision? As an interviewer you need to have thought about this before the interview and be ready when the time comes.

After the interview is the scorecard and the debrief. Here the most important part is your decision. And I mean decision. Don't be on the fence. If it's not absolutely yes, then it's no. Make sure you write down the objective "why" behind your choice. At the debrief you're expected to say what your choice was and why. You don't need to do a play-by-play recap of your interview. Just what your area(s) was, how the candidate did, your reasons, and any potential red flags. Then listen to the other interviewers. If you have questions, ask them. Think about trends and consistency across interviews. It's ok to change your mind. The BR should ask at the end, but if not bring it up.

So as we (re)start the interview process, remember how important it is. As a BR, and previously as an HM, one of my goals was to make sure we didn't make the wrong hire. I'd much rather have an open req a couple of months longer than have the wrong person in the seat for years.

Finally, besides looking at what I. M. Wright has to say about interviewing, look around at the other Hard Code entries. I talked to Eric a few times when I was at Microsoft and almost worked for him once. He's got a lot of interesting things to say.

Flexibility

The first two principles of the agile manifesto are

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.

Or, in other words, you need to be flexible and adapt to changes in the world around you. And, as it also states, 

Continuous attention to technical excellence and good design enhances agility.

And a lot of people forget about that part. They think all you need is a daily scrum and the occasional retrospective, and you're agile. But that's not really how things work. If you start from scratch every two weeks, and you only do things that fit in a two week sprint you're just fluttering in the breeze.

Plans are good. Goals are good. Knowing what done looks like is critical. All of these things let a group of people work together to produce, as the manifesto says, working software as a measure of progress. And that's what's really important. The definition of done may (will) change over time, but at the end, you've got to add customer value.

Here's a real-world example for you. My daughter got married on Tuesday. It was wonderful. But it wasn't what we planned 9 months ago. Back then, in the before times, it was going to be the Wings Over The Rockies Museum, amidst the airplanes. And it was going to be in late October, with lots of family and guests coming from out of town. But then Covid. So we had to be flexible. Move the date up a couple of months. Scale things back. A small group in our backyard. We can do that. Most things are the same. Still need a wedding dress, caterer, and cake. Still need to send out invitations, (but with a different date). But add masks and face shields, a tent and some outdoor air conditioners because early September in Colorado can be in the upper 90s. We've got this.

Then, fires. Not just in California adding a hint of smoke to the air, big swaths of western Colorado, and the smoke pours through the I-70 corridor into the Denver area. Ok. We can deal. It's not too bad. Then a new fire starts 25 miles away and it's raining ash in the back yard. Let's get some big umbrellas. This will work.

Of course this is Colorado and the weather is odd. After a weekend of record highs we get a 60 degree temperature drop and 4 inches of rain/snow. Quick. Move the wedding inside. But we can deal with that. The furniture is the same, the decorations are the same, just packed a little tighter. The caterer can be here a little early in case there's problems with travel.

It wasn't quite agile/scrum, but all of the pieces were there. We had a clear goal. We worked with the customer (my daughter) and understood the real goal. The building blocks were sound and well designed, so we could adjust them, do some rescoping and rearranging, and put them back together. And like any good project we made it happen.

Like I said, it was wonderful. Not exactly what we planned, but wonderful nonetheless. And that's why it's important to have a good design and be flexible.

PS: That wasn't the end of the flexibility. The morning after, as they were about to head out for their honeymoon the hotel called. Reservation canceled due to one of the fires. But they were flexible and were able to transfer their reservation from Steamboat to a hotel in the same chain in Breckenridge. :)

What Would You Say... Ya Do Here?

Unless you work at Initech, chances are you're not got to be asked that question. You've probably got a Jira ticket or two assigned to you that you're working on and periodically you close one. Sometimes that involves landing a PR, and sometimes it's releasing a document. That's what you do, right?

But is that really all you're doing? Probably not, and it's important to make all the work you're doing visible, not just the work assigned to you. There are a bunch of good reasons for doing it. It helps your manager, your team, the organization overall, and not inconsequentially, it helps you.

Before I get into how it helps, let's think about a typical day. In the morning I look at my assigned tasks and think about where I left it the day before. I'll spend some time working on that. But I'll also spend time doing code reviews. Most days there are one or more quick design discussions where we'll talk about tradeoffs of something. It's a rare day where I'm not having a discussion with another team about something we're doing together. Then there are the weekly deep dives. I don't go to all of them, but they're really good to go to occasionally to understand what else is going on and make sure I see the big picture. And lately we've started to do interviews again. So there's interview and debrief themselves, but also prep and document time. Lots of other things that I do on any given day,

So how does making all of those little things visible help? It helps your manager by providing visibility into what's happening beyond the Jira board. Your manager has a different perspective on the big picture, so knowing about those other things might spark a connection and eliminate some duplicate work or give you a solution you didn't know about. If you need resources your manager might be able to help.

A lot of that applies to your team and the organization as well. Your work likely impacts and integrates with others, so making sure they know what you're doing and how it's going to both enable and constrain them is important. It prevents wasted effort on both sides and makes sure that there aren't any missed expectations. Additionally if, as part of implementing a feature, you're researching some new technology there's probably someone around who knows it and can help you out. Or at least they could if they knew you were looking into it.

And most importantly, it helps you, a lot. If you're proactively making sure your manager knows what you're doing you won't get asked. If there was some particularly gnarly problem you needed to work through then letting your manager know explains why the original estimate was wrong. The work you're doing is valuable so why not get credit for it? 

Github And VSCode

I keep finding more things to like about VSCode. There's LiveShare, which I talked about a few months ago for pair programming/demos/help debugging. There's integration with remote containers, which lets you develop remotely almost like you're there. And one thing I recently started using, the GitHub Pull Requests and Issues plugin.

With the GitHub plugin you can do your PR reviews right in VSCode. There are lots of benefits. It starts with not having to go open some other app/website. It's all right there for you. You can create a new PR, see someone else's, and add comments/approve/ask for changes right away.

The editor is the same, so however you've got it configured and trained your fingers just works. Diffs are displayed the way you want, and always displayed the same. You also have access to the entire codebase, with all the context. You can see how/where methods and classes are used. You can see the history of the code. You can see the annotations. So next time you need to create/review a PR, give the extension a shot.

What's your favorite VSCode extension?

To Rewrite Or To Not Rewrite

That is the question, and like so many other questions, the answer is, it depends.

Presented here for your perusal, 6 different stories with 6 different initial conditions, 6 approaches, and 6 different results.

I've been through a few of these cycles myself, and I've seen most of the different results. It's often easy to see what was going to happen in hindsight, but looking forward it's not so easy. And sometimes doing exactly what you planned and having it work the way you expect just isn't enough.

My first foray into entertainment software was Falcon 4.0. It was a critical success, and you can still get it on Steam. It was even a commercial success (sort of), but it also played a large part in the end of the company.

When I came on board the project was already a year late and the direction was unclear. We ended up scrapping the code from V3 and starting from scratch. That let us build a state of the art system, with the most detailed graphics, flight model, and AI available at the time. We made the choice to add 16 player multiplayer capabilities to it.

This was the late 90s, and LAN parties were all the rage. People would lovingly pack their computers in the back of their cars, gather at someone's house, hook them up to a brand new network created just for the day, say some magic incantations, and 7 times out of 10 the computers would connect and the games would begin. Our fanbase loved it. They'd spend thousands on their computers, then thousands more on custom seats, cockpits, sticks, throttles, and display systems to get an advantage.

But doing all that took time. By the time we released Falcon 4.0 the market had shifted. We were building study simulations, when the mass market wanted survey simulations. So in one sense our gamble paid off. Our target market loved what we did, and we got all sorts of awards. On the other hand, it stretched the company and there were games with better ROI (Star Trek and Magic the Gathering in this case) that our new owners decided to focus on. So in that sense our gamble didn't pay off, and the rest of the Falcon series never got made.

Should we have built Falcon 4.0 as a modification of Falcon 3.0? We'll never know for sure, but things certainly would have been different.