Innovative Data Transfer

While we're on the subject of moving things over the interwebs, it's not just copper wires or fiber that can be the backbone of the internet. Have you ever heard of IPoAC? There's IETF RFC 1149, A Standard for the Transmission of IP Datagrams on Avian Carriers. And if that's not good enough there's also IETF RFC 2549, IP over Avian Carriers with Quality of Service, so you can ensure you get enough bandwidth for your VOIP data. And these aren't just jokes. A group in Norway was able to issue a ping command via carrier pigeon. This was a little more complex as it involved some OCR, and it was pretty slow (64 bytes/42 minutes), so probably not something Nexflix will be using.

Of course, avian data transfer isn't always slower than traditional copper. Many moons ago I did a campus interview with a candidate who had worked at South Africa's Telekom the previous summer and learned about Winston, an 11 month old pigeon that was able to transfer a 4GB memory stick 60 miles in just over 2 hours (25x faster than the local ISP), including local upload time.

And remember, there's an XKCD for that.

GeoStatisticians vs Default Values

Statisticians love numbers. They love patterns. They find patterns everywhere. Add in a healthy understanding of GIS and/or an ArcGIS license and you never know what they'll find.

Consider the case of the 500 mile email. Imagine you're the part-time admin of the email system for a university and you get a call from a department head letting you know that they haven't been able to send email more than 500 miles away for a few days. What do you do? First, you ask if they're joking. then you ask how they came up with that range, then why it took so long to complain Once you've convinced yourself there's really a problem, you debug and fix it.

It turns out that in SendMail 5 the default connection timeout is 0ms. Not 50 or 10000, but 0. So if you have a bad config file the default is used. In most multi-user, shared systems the timeout is a minimum value, not an exact value. In this case, what with context switching, kernel transitions and blocking I/O calls, 0ms turned out to be about 6ms.

Do you know how fast a signal travels over fiber? Pretty close to C. And a good NIC can send an SYN/ACK back pretty darn quick (microseconds). So, the transport delay allowed for a connection is about 3ms * C, or 3 millilightseconds. and 3 millilightseconds is right about 500 miles. So with a bad config file any time the recipient was over 500 miles away the connection would time out, and the send would fail.

The moral of the story? ALWAYS have sensible defaults for your config variables OR fail on startup if they're not set.

Owning Your Comments

<soapbox>

I've been providing feedback on documents (RFCs, RFPs, Incident Reports, Proposals, Code Reviews/PRs, etc) for a long time. I've been doing it so long that I remember doing it with notes in the margin and sticky notes attached to the relevant pages. For the last 10+ years, of course, the comments have been in MS Word/Google Docs and Codeflow/Phab/GitHub. Generally, there's no question who owns the document being reviewed. Who owns those comments however, is less clear.

The way I see it, a comment starts a new virtual collaborative document. Like any other document, the person who created it "owns" it. That person owns the content and is responsible for driving the resolution any issues raised. And like with any other document, that doesn't mean the owner does the work. It could be the person who produced the work being commented on needs to do something, It could be a mentioned third party. It could be that the comment just brings awareness and nothing needs to be done. Regardless of what ends up happening, the author owns the "document". And that means only the author can close the document.

Clear ownership is important to our success. If everyone is responsible then no one is responsible. At Uber the idea of "being an owner" is a thing, and for a long time ownership meant grabbing things and holding on to them. And when there are problems languishing without owners that's a good thing. But, like anything else, it can be taken too far. And that's something I want to push back on, particularly in this area.

Recently I've seen an uptick in comments getting closed by someone other than the person opening it. That's bad for a bunch of reasons. First, it's not very respectful. When you answer someone's question it's not up to you to decide if the question has been answered, and closing a comment abruptly ends the conversation. Second, the discussion helps others understand the situation. They could learn from it, recognize similar situations, or take it one step further and solve bigger issues. Finally, You prevent others from seeing the comment/response. If you leave it out there for others to see you might save yourself from having to answer the same question multiple times.

So, unless the "owner" has left the company, or you've reached out and asked them and gotten no response, don't close someone else's comments.

</soapbox>

Managing Your Inbox

Remember Inbox Zero? The idea that to keep your sanity and maximize productivity you should keep your inbox empty. Basically, treat your email inbox as a task list, and make sure that by the end of every day you've dealt with all of the items. Now this didn't mean everything was done, just that you'd dispositioned it. There were 5 basic options, delegate, delete, defer, do, or respond. Regardless of which one you chose, you didn't need to worry about that email (at least for a while). And for many people it worked.

Of course, like anything, the law of unintended consequences kicked in. Inbox zero was supposed to free people from the tyranny of their inbox. Instead, for many there was lots of action and anxiety. It got a lot of people checking their email nights and weekends to keep their inbox empty. And that was almost 15 years ago when life (and social media) was much simpler.

Which of course led to a recent update of the concept. Which was really an acknowledgement of how the world had changed, and a return to the basics, the idea of getting things done. Starting with the idea that we don't just have one inbox anymore. There's work, personal, professional (but not work), and some number of social media inputs. And after acknowledging that there's multiple input streams, changing the goals. Instead of an empty inbox being the goal, understanding and focusing on the right priorities at the right time is the key. Rather than just taking control of your email, take control of your time. Figure out what your priorities are, not just at work, but in general. Differentiate between the important and the urgent. Then spend yout time appropriately.

And that's not just a good way to manage your inbox. It applies to your your sprint goals, your OKRs, and your life in general.

From BeyondCorp to BeyondProd

Like what they do and stand for or not, there's little question that Google has a very large presence and impact on the technology world. From defining a language (Golang), a build system (Bazel), a container management system (Kubernetes), and a database (Bigtable) to an entire service mesh to tie them together (Istio), a lot of what is used is based on what Google has done and is doing. And it's not just the technology itself, it's how those technology pieces get put together. Google's recent white paper BeyondProd is all about how to securely connect all those microservices. And not just securely connect, but know that the thing you're running is the thing you expect it to be. That it was built securely, from a secure, reviewed source repository, with approved libraries and dependencies. 

And it all starts with Identity. In the security world it's called Authentication (AuthN). Everything command/message/request has to come with an identity that is verifiable. And until you have AuthN you don't need to worry about Authorization (AuthZ). Without AuthN, AuthZ is a false sense of security.

And that's what Zero-Trust is all about. Ensure that everything you hear is from who you think sent it and that identity is allowed to do whatever it's trying to do. That's "Zero Trust", and where we're headed. Without it, you're never really sure. For a long time in Core Business there were a set of APIs that were only accessible to the rtapi service. And those APIs knew that a caller was rtapi because there was a header, `uber-caller` that was set to `rtapi`. As long as the caller was on the PROD network and had that header the services accepted it and did their thing.

As you might imagine, pretty soon every service added that header. A few different things happened. First was that there was no traceability. Then the services fell over because they were scaled for the expected traffic. Then the abstraction leaked. Then some data got to other services. Eventually it turned into a big bowl of spaghetti that we're still trying to unravel.

So let's not do that to ourselves. We might not be Zero Trust now, but we will be, so keep that in mind as you design your services/libraries/systems.

Technical Debt And Inflation

I've talked before about technical debt, and as I've noted, I think it's a really good way to think about and explain to others the fallacy of a simple, linear tradeoff inside the quality, feature set, and ship date triangle. You can usually trade between feature set and ship date and think of it as an opportunity cost. At any given moment you have a certain amount of "developer capital" (capacity) available. Every bit of that capital you apply to one (feature set or ship date) is capital you can't apply to the other. Quality doesn't quite work that way. You can trade quality for either features to development time, but that's not opportunity cost, that's borrowing against the future, and you will have to pay it back, with interest. And that's something that's pretty easy to understand and explain to others. And if you don't at least make the "vig" then eventually you get so far behind that you can't do anything.

What I'd like to add to that is the concept of inflation. That's the idea that if you do nothing, you fall further behind. If you take $100 and put it under your mattress for safekeeping and then go to spend it 5 years later you'll find it won't buy as much. If you don't make investments then the value goes down.

For software it's not the value of a dollar, but the environment. Think about Python. If you don't put in the effort to keep it up to date you get behind, lose out on security updates, and eventually all support. If you have any COBOL laying around think about what you need to go through to maintain that. Not just that, but requirements and expectations change. If Netflix was still delivering movies at 1024x768 would it be worth as much? Probably not.

All of this is not to say that you never take on new debt, only that you should do it knowingly and with a plan to repay it. Like I mentioned a few days ago, when games shipped in cardboard boxes you needed to have them on the shelves for Black Friday. If you missed that date you lost an entire development cycle. In cases like that taking on debt to ship made sense. But those situations are rare. With the advent of online sales and distribution, Black Friday is still important, but not nearly as critical.

So how are you investing your capital today?

Panzer Is NOT German For Tank

Or at least it wasn't originally. When I was working on Combat Flight Sim 2 we released 5 fully localized versions. Not just English, but FIGS (French, Italian, German, and Spanish). Since it was Microsoft, the process of localization was well defined. We did the easy things like using the correct locale for dates and numbers, and all of the UI and in-game text were resource based. We had 3rd party vendors to provide the resources and people trained in testing such things.

Of course, since word order can be different, not just that, but our strings were tokenized as well. We didn't just localize the words, we localized the format strings so we could but do word order and gender matching as needed. We'd just pass in enough strings (which were localized resources themselves) to the formatter and it would pick and choose which ones to use. And we did the same thing for our sound UI. Radio communications were done as voice instead of text, so we had to do the same kind of audio buildup. From a programming standpoint that wasn't much different from the text, but we did put a big load on our sound team. They had to make sure all of the snippets were done in a way that they could be strung together in some random order and sound right. That was a huge combination of strings, and my hat's off to them. Somehow they were able to do it.

All of that was relatively straightforward. The place we got bit was more cultural. As is typical, we did our UI wireframes in English. And there were cases where our UI got dense. Orders of battle, post action reports, pilot statistics. And our designers made it look good. There were several places where we had "Tank(s)" in the UI. And knowing that some languages use longer words than English, there was room for some extra characters. Then the localizers got involved. We got back the new resources, plugged them in, and tried it out. It worked, but the visuals, not so much. Turns out that instead of "Panzer", which we expected, we got back "Panzerkampfwagen" (armored combat vehicle). That's correct, but it didn't fit. Germans also don't like acronyms. They tend to make up new words out of groups of words, so we had to deal with that as well. In some cases we had to re-work the UI. In others we used modern words (like Panzer) instead of the more temporally appropriate terms.

So the moral of the story? Be flexible, Be adaptable. And remember, not everyone does things the way you do.

Quality Is Job #1

Ford may have stopped using that tag-line over 20 years ago, but it’s still relevant. Quality is more than building something that usually works or ‘meets requirements’. It’s more than that. It’s something that not only works when things are perfect, but degrades gracefully and in a controlled manner when things aren’t. It’s not just done according to spec, it’s something the user thinks is done. It’s something that provides more value than cost. It’s something that delights. And it applies at every level. From the RFC describing what you’re building, to the entire SDVP ecosystem, quality stands out. You can rely on it. Does it cost more? Very often it does, but it provides even more value.

Quality is stable and reliable. As of last week, grep is 45 years old. Yet we still use it every day and you never wonder if it missed something. You might wonder if you got your regex correct, but you know grep did its job correctly. Yes, it’s been updated and new features have been added, but someone who used it 45 years ago would still be comfortable using it today. That’s quality. Similarly, the Douglas DC-3/C-47/Gooney Bird. First flown in 1936, with production ending in 1945, there are still over 1000 flying today. If you bought one you certainly got your money’s worth.

So how does that relate to us? Simple, and it goes back to Ford’s tag-line. Quality isn’t something you bolt on later. It’s something you build in. From the use cases to requirements to implementation to operationalizing and support. Regardless of your function, role, title, or part of the product life cycle you work on, quality is job

1.

Mini-Maxing Your Java

"It is by caffeine alone I set my mind in motion.
  It is by the beans of Java that thoughts acquire speed,
  the hands acquire shakes, the shakes become a warning.

  It is by caffeine alone I set my mind in motion."

    - The Programmer's Mantra

I first ran across that line in the mid 90s, and to the best of my sleuthing, it was pretty new then. Now coffee has never been my preferred source of caffeine, but I've used caffeine to great effect. Back in the day software was shipped in boxes, and you had to reserve time in a production facility to print/box/ship things. That was expensive, and there might not be another spot in the schedule for a while, so meeting dates was pretty important. Factor in that I was working for a gaming company, and if you didn't have you boxes on the shelves in time for Black Friday you might as well wait for next year. That also meant that production houses were booked solid and the next available slot was after the deadline so any slip meant you would miss the holiday. Because of that, I was part of more than one 3 month long, caffeine-fueled, tech debt generating, crunch mode.

And you know what, they mostly worked. We shipped, and the company didn't go bankrupt. They weren't without cost though. A big part of the team would quit, we'd spend a few months patching things to fix problems, and then have to figure out how to move forward with a new version or the next great thing. but those were issues for another day.

Nothing beats getting enough sleep at ensuring you're at peak effectiveness, especially over extended periods of time. Chronic sleep deprivation will really mess you up, both short term and long term. the negative impacts range from forgetfulness and lowered alertness to high blood pressure and strokes.

And while I absolutely don't recommend it, especially for extended periods, sometimes it is justified and you do need to spend the extra time and just get things done. In that case, caffeine can be a big help. But like anything else, you want to maximize the impact. So how do you know just how much coffee to drink, and when? Well, the Army has thought about it, and while they're not quite ready to release the app, there's a beta website you can use to get an approximation. And the tech is licensable. Wonder if we should add it to the Driver/Operator console.

Shift-Left Is Silly

Ok, the idea isn't silly, but…

When I started in this business it was developers, developers, developers and the ubiquitous Ballmer Monkey Dance (I was there for that). Write some code, throw it over to the testers, ship it over their objections, and then release updates. Maybe there was a PM (Program Manager) in there to mention the customer. Then came agile. We renamed our software test engineers (STE) to software development engineers in test (SDET), asked them to do some automation, called the PM a scrum master, then shipped the code. Then came devops, and the operations folks joined the team. The team charter got broader and included observability and alerting. Now they're talking about the wisdom of DecSecOps, saying we need to get security involved earlier. And again, they're not wrong, but…

I'm a Mechanical Engineer by training. Mechanical engineering is _old_. It's as old as the screw, stick (lever), rope, ramp, and wheel. And over those thousands of years mechanical engineering has learned a few things. Not simply, and not without pain, and not without big mistakes (ever seen the video of the Tacoma Narrows Bridge wiggling?), but like the Farmer's guy said, "We know a thing or two because we've seen a thing or two." And one of the most important isn't engineering per-se, it's everything that happens *around* the engineering. From the building the pyramids to the Manhattan project to Apollo, it's not just the results that are impressive, the system required to build them is impressive too. Coordinating 10s of thousands of people in hundreds of locations and when you get done the result works. Learning how to do that is at least as big of an accomplishment as the thing being accomplished. We don't always get it right, but we try.

So what has that got to do with Shift-Left? Well, all Shift-Left is saying is that in software development you can break down a system into components and implement them separately, as long as you think about the entirety at the beginning. You need requirements, designers, engineers (software and hardware), program/project managers, testers, operations, security, supply chain management, and anyone else that might be involved going from an idea to a product.

And that's another way to describe Systems Engineering.