Recent Posts (page 44 / 71)

Hiring

Part of the return to normalcy, whatever that means in 2020, is a return to hiring. Open positions are starting to show up, recruiting is ramping up, and interview requests are starting to show up in our calendars. Making good hires is one of the most important things we can do for the long-term health and success of our teams and the overall organization. Not making bad hires is even more important.

I've been at Uber over 5 years now and I've been a Bar Raiser for about half that. Back at Microsoft I was an "As Appropriate" interviewer, which is roughly the same thing. Between college campus, phone screens, and on-site interviews I've done over 2000 interviews in my career. All of which is to say I've got some experience doing it.

While I had done interviews before Microsoft, it was at Microsoft that I spent enough time interviewing and hiring that I learned and formalized the framework I still use today. One of the things I learned is that interview prep is just as important for the interviewer as it is for the person being interviewed. And it starts with the hiring manager (HM).

The HM needs to work with the recruiter to define the role and the requirements to make sure the right candidates are brought in, and often needs to help with the sourcing and filtering. After all, no one knows the requirements better than the HM. And once candidates have been identified the HM is responsible for interview loops. Not the actual scheduling, but making sure that they run smoothly, all of the areas are covered, and that the candidate has a positive experience, regardless of the outcome. The HM not only provides the scheduler with a pool of interviewers, but once the loop is set, makes sure that each person knows what area they're supposed to cover, both technically and culturally.

That's when the interviewers get involved. When you know what you're going to cover you can start your preparation. Good questions are critical, and making sure you don't duplicate things on a given loop is important. What makes a question good is depth. Not complexity, although that's sometimes a part, but layers. You want to be able to go one level deeper into the problem. You're looking to understand the candidate's reasoning. Asking why a choice was made, probing deeper into how choices are made, because that reasoning is at least as important as the choice. Maybe there's something you didn't think of or maybe something the candidate didn't think of. When you provide some new information/constraint, how does the candidate incorporate that into the decision? As an interviewer you need to have thought about this before the interview and be ready when the time comes.

After the interview is the scorecard and the debrief. Here the most important part is your decision. And I mean decision. Don't be on the fence. If it's not absolutely yes, then it's no. Make sure you write down the objective "why" behind your choice. At the debrief you're expected to say what your choice was and why. You don't need to do a play-by-play recap of your interview. Just what your area(s) was, how the candidate did, your reasons, and any potential red flags. Then listen to the other interviewers. If you have questions, ask them. Think about trends and consistency across interviews. It's ok to change your mind. The BR should ask at the end, but if not bring it up.

So as we (re)start the interview process, remember how important it is. As a BR, and previously as an HM, one of my goals was to make sure we didn't make the wrong hire. I'd much rather have an open req a couple of months longer than have the wrong person in the seat for years.

Finally, besides looking at what I. M. Wright has to say about interviewing, look around at the other Hard Code entries. I talked to Eric a few times when I was at Microsoft and almost worked for him once. He's got a lot of interesting things to say.

Flexibility

The first two principles of the agile manifesto are

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.

Or, in other words, you need to be flexible and adapt to changes in the world around you. And, as it also states, 

Continuous attention to technical excellence and good design enhances agility.

And a lot of people forget about that part. They think all you need is a daily scrum and the occasional retrospective, and you're agile. But that's not really how things work. If you start from scratch every two weeks, and you only do things that fit in a two week sprint you're just fluttering in the breeze.

Plans are good. Goals are good. Knowing what done looks like is critical. All of these things let a group of people work together to produce, as the manifesto says, working software as a measure of progress. And that's what's really important. The definition of done may (will) change over time, but at the end, you've got to add customer value.

Here's a real-world example for you. My daughter got married on Tuesday. It was wonderful. But it wasn't what we planned 9 months ago. Back then, in the before times, it was going to be the Wings Over The Rockies Museum, amidst the airplanes. And it was going to be in late October, with lots of family and guests coming from out of town. But then Covid. So we had to be flexible. Move the date up a couple of months. Scale things back. A small group in our backyard. We can do that. Most things are the same. Still need a wedding dress, caterer, and cake. Still need to send out invitations, (but with a different date). But add masks and face shields, a tent and some outdoor air conditioners because early September in Colorado can be in the upper 90s. We've got this.

Then, fires. Not just in California adding a hint of smoke to the air, big swaths of western Colorado, and the smoke pours through the I-70 corridor into the Denver area. Ok. We can deal. It's not too bad. Then a new fire starts 25 miles away and it's raining ash in the back yard. Let's get some big umbrellas. This will work.

Of course this is Colorado and the weather is odd. After a weekend of record highs we get a 60 degree temperature drop and 4 inches of rain/snow. Quick. Move the wedding inside. But we can deal with that. The furniture is the same, the decorations are the same, just packed a little tighter. The caterer can be here a little early in case there's problems with travel.

It wasn't quite agile/scrum, but all of the pieces were there. We had a clear goal. We worked with the customer (my daughter) and understood the real goal. The building blocks were sound and well designed, so we could adjust them, do some rescoping and rearranging, and put them back together. And like any good project we made it happen.

Like I said, it was wonderful. Not exactly what we planned, but wonderful nonetheless. And that's why it's important to have a good design and be flexible.

PS: That wasn't the end of the flexibility. The morning after, as they were about to head out for their honeymoon the hotel called. Reservation canceled due to one of the fires. But they were flexible and were able to transfer their reservation from Steamboat to a hotel in the same chain in Breckenridge. :)

What Would You Say... Ya Do Here?

Unless you work at Initech, chances are you're not got to be asked that question. You've probably got a Jira ticket or two assigned to you that you're working on and periodically you close one. Sometimes that involves landing a PR, and sometimes it's releasing a document. That's what you do, right?

But is that really all you're doing? Probably not, and it's important to make all the work you're doing visible, not just the work assigned to you. There are a bunch of good reasons for doing it. It helps your manager, your team, the organization overall, and not inconsequentially, it helps you.

Before I get into how it helps, let's think about a typical day. In the morning I look at my assigned tasks and think about where I left it the day before. I'll spend some time working on that. But I'll also spend time doing code reviews. Most days there are one or more quick design discussions where we'll talk about tradeoffs of something. It's a rare day where I'm not having a discussion with another team about something we're doing together. Then there are the weekly deep dives. I don't go to all of them, but they're really good to go to occasionally to understand what else is going on and make sure I see the big picture. And lately we've started to do interviews again. So there's interview and debrief themselves, but also prep and document time. Lots of other things that I do on any given day,

So how does making all of those little things visible help? It helps your manager by providing visibility into what's happening beyond the Jira board. Your manager has a different perspective on the big picture, so knowing about those other things might spark a connection and eliminate some duplicate work or give you a solution you didn't know about. If you need resources your manager might be able to help.

A lot of that applies to your team and the organization as well. Your work likely impacts and integrates with others, so making sure they know what you're doing and how it's going to both enable and constrain them is important. It prevents wasted effort on both sides and makes sure that there aren't any missed expectations. Additionally if, as part of implementing a feature, you're researching some new technology there's probably someone around who knows it and can help you out. Or at least they could if they knew you were looking into it.

And most importantly, it helps you, a lot. If you're proactively making sure your manager knows what you're doing you won't get asked. If there was some particularly gnarly problem you needed to work through then letting your manager know explains why the original estimate was wrong. The work you're doing is valuable so why not get credit for it? 

Github And VSCode

I keep finding more things to like about VSCode. There's LiveShare, which I talked about a few months ago for pair programming/demos/help debugging. There's integration with remote containers, which lets you develop remotely almost like you're there. And one thing I recently started using, the GitHub Pull Requests and Issues plugin.

With the GitHub plugin you can do your PR reviews right in VSCode. There are lots of benefits. It starts with not having to go open some other app/website. It's all right there for you. You can create a new PR, see someone else's, and add comments/approve/ask for changes right away.

The editor is the same, so however you've got it configured and trained your fingers just works. Diffs are displayed the way you want, and always displayed the same. You also have access to the entire codebase, with all the context. You can see how/where methods and classes are used. You can see the history of the code. You can see the annotations. So next time you need to create/review a PR, give the extension a shot.

What's your favorite VSCode extension?

To Rewrite Or To Not Rewrite

That is the question, and like so many other questions, the answer is, it depends.

Presented here for your perusal, 6 different stories with 6 different initial conditions, 6 approaches, and 6 different results.

I've been through a few of these cycles myself, and I've seen most of the different results. It's often easy to see what was going to happen in hindsight, but looking forward it's not so easy. And sometimes doing exactly what you planned and having it work the way you expect just isn't enough.

My first foray into entertainment software was Falcon 4.0. It was a critical success, and you can still get it on Steam. It was even a commercial success (sort of), but it also played a large part in the end of the company.

When I came on board the project was already a year late and the direction was unclear. We ended up scrapping the code from V3 and starting from scratch. That let us build a state of the art system, with the most detailed graphics, flight model, and AI available at the time. We made the choice to add 16 player multiplayer capabilities to it.

This was the late 90s, and LAN parties were all the rage. People would lovingly pack their computers in the back of their cars, gather at someone's house, hook them up to a brand new network created just for the day, say some magic incantations, and 7 times out of 10 the computers would connect and the games would begin. Our fanbase loved it. They'd spend thousands on their computers, then thousands more on custom seats, cockpits, sticks, throttles, and display systems to get an advantage.

But doing all that took time. By the time we released Falcon 4.0 the market had shifted. We were building study simulations, when the mass market wanted survey simulations. So in one sense our gamble paid off. Our target market loved what we did, and we got all sorts of awards. On the other hand, it stretched the company and there were games with better ROI (Star Trek and Magic the Gathering in this case) that our new owners decided to focus on. So in that sense our gamble didn't pay off, and the rest of the Falcon series never got made.

Should we have built Falcon 4.0 as a modification of Falcon 3.0? We'll never know for sure, but things certainly would have been different.

Semantics

Debugging customer problems can be a challenge, especially over zoom or email where you can't see the exact input/output. To make it easier I have a set of questions I usually ask at the beginning. One of them is "What version are you running?" That's really important info since there can be many versions out in the field.

For the infra CLI it's pretty easy to get the version. Just run infra version and it will tell you. This is really important in *NIX based systems, because unlike Windows DLLs/executables, posix ELF files don't have a standard way to store/retrieve the version info of a file. Instead, for shared objects you use file name and a system of symlinks with prescribed names to manage version. Using that is a topic for another time.

The topic for today though is what to put in the version. The simplest is just a continuously incrementing number. Every time your build process produces a public version the version number gets incremented by one. This covers the most basic requirements of versioning. It ensures that all public versions can be uniquely identified and you can tell which of two versions is newer. That's a good start. But it leaves you with a problem. You can tell that one version is older than the other, but you can't tell how much older, and you don't know from the version number if it was yesterday to 10 years ago.

Another versioning scheme is to use the build date/time as the version. Something like YYYY.MM.DD.hh.mm.ss.<buildnumber>. Year/Month/Day/Hour/Minute/Second. That's almost certainly unique, but if you have a parallel build system that can start multiple builds at the same time then you can add the build number for disambiguation. It's important to use that order because that ensures that you can sort by version. The date scheme gives you the ordering of a simple number, and lets you know when the releases were done. That's a lot more helpful, and users really like it because it's obvious to them if something is old

As a developer though, that doesn't tell me something I really want to know. Should I expect that version to have what I'm looking for? And that's where semantic versions (SemVer) come into play. Semantic versions are of the form <major>.<minor>.<patch>. The theory is that all versions for a given <major> version are backward compatible, and increasing <minor> versions only add functionality, Removing functionality would be a breaking change and require a new major version. For a given <major>.<minor> everything is the same, and new versions fix emergent bugs/issues that are discovered after release. Pretty simple. SemVer trades the absolute time information for functionality/compatibility info. In my experience that's a trade worth making.

As with any system, you'll find that sometimes you need to extend it. You might have a on-off release, a shared fork, or simply need to test a version locally, but still be able to keep it distinct from all of your current and future released versions. A common way to do that is with an extension. That's where user name and date could be very helpful. Let's say I want to give you a test version I built based on the code in version `0.17.1`. In that case I might give it version `0.17.1-leonr-test_myfunc`. That way everyone who sees it would know the base version, who built it, and why.

And of course, you'll want this tied in to your version control system so that it's easy to get the code that went into a specified version and your build system, so you don't have to worry about getting it wrong.

Rockall

Back in my early Microsoft days on the FlightSim team we were always looking for better perf. If nothing else we needed to give as much time to the rendered as possible because our customers always wanted to run the graphics with everything set to 11. And since this was somewhere between C and C++ (home made cross module vtable and dynamic linking, but that's another story) memory management was a thing. And Intel's vTune told us that allocating memory was taking a measurable amount of time.

Our solution was to build free lists for each type of object we wanted to make faster. The initial implementation was mostly replacing free/delete with putting the object into a linked list and having malloc/new check the list then allocate it if needed. This worked, but we needed to be careful with re-use and re-initialization, and didn't give us much protection from stale pointers to existing objects. Slowly we added more safety and protection. But it was always work and made things more cumbersome.

One of the nice things about working for Microsoft was access to Microsoft Research (MSR) a whole division of folks whose job was to think about big problems in computer science and figure out solutions. And they were always looking for new problems and ways to get their solutions into shipping products. We got all sorts of interesting tools and ideas from them. One of them was a memory management system called Rockall.

Rockall was based on individual heaps, basically a heap per type of object, and gave us a couple of advantages over simple new/malloc. The biggest was that it was a drop in replacement. We didn't need to do anything but set it up at the beginning, and then it did it's magic. Don't ask me how that part worked. I just used it :) We would pre-allocate the space for however many of each of the different types we wanted, then getting a new one was blazingly fast since the memory was already allocated and waiting. Initialization magically happened. It was also contiguous, so we got better memory packing. Just that level of usage gave us a lot of benefits.

But it also had a bunch of debug features which helped out. If you had enough memory you could tell it to put each object at the end of a page and mark the next page as unreadable. Any memory overruns are quickly pointed out in that case. You could set it to mark the page as unreadable on delete, and again, any use after delete quickly became apparent.

It was also faster on cleanup. Transitioning out of flying and into the UI just required a few calls to delete each heap instead of making sure each and every object was individually deleted, in the right order, so we didn't leak anything. At debug time it could also tell us which things hadn't been deallocated manually if we wanted, but since we used it on the way out we didn't care.

That kind of thing can also help out in GC based code as well. Memory that's alway in use is never subject to GC, you don't end up with things moving around in memory to compact, and your allocation times go down. That was really important when we were doing 3D rendering in the browser for VirtualEarth. We didn't have Rockall, so we had to use a manual free_list, but we still got most of the benefits.

Something to think about.

In Defense Of Agile

And just to be clear, it's not that I think everything was better 30 years ago and all you kids these days should get off my lawn. Like everything else, there are cycles and sometimes things swing too far, or more often, people pick up the terminology / trappings of a new idea and then ignore the details.

Agile can be like that. Pair/mob programming. Just in time design. Customer feedback. Short cycles. On the surface, or blindly applied it makes no sense. Two people working on the same line of code is half the efficiency. Just in time design turns into no design. We can't ask customers what they want. They just want a better buggy whip. And how can you get anything done in two weeks? It takes two weeks for us to agree on who's doing something, let alone get it done.

But really, these new ideas aren't about how. They're about why, and the how is the means to an end. Pair programming isn't about efficiency. It's about a broader perspective, making sure things are clear and understandable, knowledge sharing, and up-leveling a team. Just in time design isn't "Make a choice then correct it later by fixing the mistakes" it's "Make the informed decisions you need when you need them and know enough".

Customer feedback is really customer involvement. You need to understand what they're doing, speak their language, and understand their pain so you can collaborate on a solution. And short cycles is part of that. It's ongoing collaboration to iterate on a good solution instead of providing the wrong solution later. It's about providing value sooner so in the ling term everyone is better off.

And that's the big take-away/conflict. The focus on the short term requires a long-term view. You need to make the right choice at the moment that balances what you're doing today with what you're going to be doing the next day/week/year.

Pick 2

We've all heard that when it comes to quality, scope, and the time you can only optimize 2 of them at a time. And for any given, localized, immediate decision that might be true. But what if I told you that over the long term you're better off with all three?

Following up on yesterday's topic, another one of my issues with two week sprints is the emphasis it places on time and showing value quickly. If every decision is based on what's going to produce the most value in the current sprint then you end up pushing a bow wave of cruft ahead of you. Eventually the resistance from all that cruft brings your forward progress to approximately zero. That's the classic description of technical debt.

Or to use a car analogy, consider Harbor Freight tools vs Snap-on. Harbor Freight sells project tools. Buy them, use them for one project, two if you're lucky, then move on. Snap-on sells tools for life. If you've ever had the opportunity to A/B compare the two you know what I mean. If you need a specialized tool you can run down to Harbor Freight, pick up the tool, and be using it for less money long before you can get the Snap-on truck to stop by. The second time you need the tool it will probably still work, but the third time you'll need to go buy another one, wasting time and money. And if you need it regularly pretty soon you realize you would have been much better off, getting things done faster and with higher quality, if you had just spent the time and money to get the good tools the first time.

Software development is the same thing. There are times when you need to be fast or cost conscious. Back in the day when I was working on games we needed to have the game on the shelf the day after Thanksgiving. If we didn't we'd lose a large percentage of initial sales, and never make them back up, so making the trade-off made sense. If we didn't get the game out and get the revenue the company might have gone out of business. On the other hand, on Falcon 4.0 we missed the next product cycle because we were still digging ourselves out of the hole we dug. It was the right choice, but only because we literally couldn't afford to spend the money up front.

Over the long haul you're better off doing it right. You'll find it's cheaper, faster, and provides more customer value.

Cycles

There's a lot of value in short sprints and frequent planning cycles. At any given time there's so much we don't know about the future needs and requirements of whatever piece of software we're working on that to write a detailed 6 month plan, let alone a 2 year plan, doesn't make a lot of sense. Moving away from waterfall development has codified that reality and helped to normalize the level of effort across the development cycle. Instead of the 3 month long crunch times (12+ hour days, 6-7 days/week) at the end of a 2 year development cycle we have a busy day or two every two weeks.

While short sprints might be better for the project, and definitely smooth out the level of effort, I'm not so sure that's good for the developers. We've done away with the crunch period at the end, but we've also done away with the low effort time at the beginning. Back in the day the first milestone was often a planning milestone. The output was a set of documents. Those documents themselves didn't have a lot of value and were often out of date before the virtual ink was dry, but writing them had a lot of value. Ideas were floated, adjusted, readjusted and prototyped. Some things were discarded and some things were discovered. Blind alleys were explored and opened or closed depending on what was found.

And for the developer there was very little pressure. Expectations were low. Time was available for random exploration. I think we've lost some of that now. In a sprint there's not a lot of time, so it all needs to be accounted for and there's pressure to make sure it directly accrues to some higher goal. And I'm not saying that's a bad thing. In fact, it's good to make sure that work done has some connection to the bigger goals. But it makes it harder to experiment. And it makes it harder to do things that take more than two weeks or don't have explicit outcomes. Yes, a Spike is a thing, but officially it shouldn't be longer than a Sprint. And it should have defined acceptance criteria. But it's not the same thing as "Here's a big hairy problem. You two folks go off and noodle on it for a month. Try things out. See what works and what doesn't. Come back with a design."

That's what we've lost. The ability for developers to have the scheduled downtime to decompress from the daily grind, then come up with something new without a ticking clock. Supersprints are good, but they're not providing the kind of freedom I'm talking about, and that's not what they're for. Whether we call it M0, a Planning Period, or something else entirely, I think we're worse off without it.

Don't get me wrong. I'm not suggesting we go back to waterfall. I much prefer the current model. I just think we need to figure out a way to make our cycle a little more dynamic. I don't want a 3 month crunch, but I do want to figure out how to get the periodic unstructured time back.