Recent Posts (page 51 / 65)

by Leon Rosenshein

Incompatible Markdown

Quick question. Are these two html blocks equivalent?

<table>
  <tr>
    <th>Product</td>
    <th>What it's good for</td>
    <th>When to avoid/limitations</td>
  </tr>
  <tr>
    <td>Spinnaker<br>[Docs](http://docs/for/spinnaker)</td>
    <td>Services on K8s</td>
    <td></td>
  </tr>
  <tr>
    <td>Batch API<br>[Docs](https://docs/for/batch.html)</td>
    <td>Batch Jobs</td>
    <td>Not SparkSQL</td>
  </tr>
</table>

and

<table>
  <tr>
    <th>Product</td>
    <th>What it's good for</td>
    <th>When to avoid/limitations</td>
  </tr>
  <tr>
    <td>
Spinnaker<br>[Docs](http://docs/for/spinnaker)</td>
    <td>Services on K8s</td>
    <td></td>
  </tr>
  <tr>
    <td>
Batch API<br>[Docs](https://docs/for/batch.html)<br></td>
    <td>Batch Jobs</td>
    <td>Not SparkSQL</td>
  </tr>
</table>

According to the browsers I’ve tried, they are. But put them in a markdown file and things get interesting. Throw doxygen into the mix and it gets even weirder.

According to the W3C, whitespace inside tags is generally collapsed. LF and CRLF are mostly ignored, and browsers all (pretty much) do the same thing. That’s because it’s a standard. Yes, Microsoft did embrace, extend, extinguish, and IE6 became a standard unto itself, and now chromium is the standard, but at least you know what to expect.

In the case of Markdown, not so much. First, it’s not a standard, in any of the standard senses. There’s no governing body, there are no validation tests, and there are lots of flavors. There was (is?) an attempt to create a standard, but I think it’s a little too late for that.

With markdown, # is the big heading. Then you can have some number of them (depending on the implementation) of smaller headings. Surrounding your text with backticks ` generally gets you inline code, _ gets italics, ** gets bold, and sometimes, ~~ gets you strikethrough. There’s a way to do links. You can use ° to make bullet lists, and putting three backticks ``` before and after lines of code is a code block. And there are usually tables. Generally defined by an arcane series of -s and |’s, with some :s thrown in to make justification suggestions.

Beyond that it gets weird. Sometimes you can tell the code block what language you’re writing and it will do some syntax highlighting. Some of the format specifiers can be nested (italic strikethrough), but others can’t. Then there are flavors of Markdown. Phab adds icons, images, and internal references. Github has its version, as does Doxygen.

One thing I’ve never been able to figure out is lists in tables cells in pure Markdown. I’ve been able to do it using embedded HTML in markdown, which both doxygen and GitHub support. And that’s what led to the problem at the start of this. They both work in doxygen and give the result I expected, But in Github Markdown the first didn’t have named hyperlinks. It wasn’t until I added the spaces and blank lines between the td and the text of the cell. that the links worked. It took some experimenting to get it to work in both cases. At least I was able to in both, so we’ve got that going for us.

by Leon Rosenshein

WIP It Good

As I talked about in DevOps Book Club, reducing work in progress (WIP) is generally a good thing. It lets you focus, makes things more predictable, reduces time lost to context switches, decreases time to value, and in my case, makes my boss happy. The place where I have some disagreement with the idea of minimizing WIP is around bigger things and things that mostly come under the category of Internal Projects. Particularly in a larger organization. Things that require organization of lots of teams across an org or inherently have large blocks of "waiting for things to settle". If that's all you're doing you've squandering your time.

Yes, you could break things down into tiny little pieces like "Send an email to team A", "Respond to email from Team A", and "Share results with Team B", but really, the task is "Coordinate with Teams A & B". And that task inherently has a bunch of built in delays and down-time. So you can have a single piece of WIP, or you can have a couple of those coordination tasks, which become basically interrupts to the main task you're working on.

You do need to be careful to not have too many of those interrupt tasks. The whole point of minimizing WIP is to focus and keep things from becoming interrupt driven, with nothing getting done. If you're not careful that's what happens.

Back when I was doing near-real-time game coding on Windows we used to use the Windows Multimedia Timer to get things to happen at something approaching a steady rate and QueryPerformanceCounter to figure out how much time had actually elapsed between calls so we'd know how much time to simulate. And it worked really well for a while. Then we started to work on some networking and remote coordination. We'd send a message to a remote node, but we couldn't wait for the response because it was longer than our frame time. And we couldn't schedule lots of little tasks because they weren't periodic or something we could plan for. Our solution was to create a comms state machine that generally did nothing, but when there were messages to handle it handled them. In between those times it existed, but didn't take up usable time and cycles. By scheduling event handlers we were able to get lots of useful work done, stay within our time budget,  and minimize overhead.

At the more macro level of project tasks and work items the same thing can be done. Most things are individually scheduled, but you carve out some time for things with lots of wall-clock time, but only a little processing time, and let it coordinate itself. And that's why I have more WIP than my boss likes. Because I'm coordinating a lot of "remote" work that needs to get done, and the coordination takes some time, but the time isn't regular or predictable on a short scale. The total work is, but not the work in the next few hours.

Net-Net, minimize your WIP. Let it increase only as much as it takes to keep making forward progress on something, but only as much as it takes to do that without negatively impacting the high priority items, thereby reducing "Time to Value".

by Leon Rosenshein

Scope Of Influence

Perf season is over (for now). Time to think about your career. And not just think about it, but write it down and share it with your manager. What do you want from your career? That's a personal decision and there are lots of things to think about. Manager vs. IC? Technology Driven? Product or infrastructure focus? Customer facing? Data science vs PM vs Engineering (or some combination of the 3)? Only you can make the call on what you want to do, and it's ok to be unsure or change your mind along the way as you learn more.

But even with all those choices, there is one thing that stays pretty consistent as you advance through your career. And that's scope of influence. Whether you're a manager or an IC, as your level goes up, so does your scope of influence.

Generally speaking, an L3 engineer, right out of school, has a small scope of influence. They are expected to be able to manage themselves. Given a framework (either in code or documentation), turn a small set of requirements into business value. They should know to ask questions if they get stuck, and think a little about what they're doing means to the future.

At L4 your scope of influence increases. You might be a mentor, you might own a feature and have to work with other people on your team to get something implemented. You're expected to be able to estimate your work and think about how it will impact people who have to use it.

As a senior engineer (L5A) or EM1 your decisions impact the entire team. You're talking about what gets implemented, not just how. And the how part changes too. You're thinking about how your designs and changes impact the future, not just over the current release cycle, but for the next year or more. And you're not just thinking about technology, you're thinking about process and efficiency and communications.

From there scope just gets bigger. It could be the technology roadmap or the deep owner of a specific technology. It could be as a director thinking about which of the competing business goals get prioritized over the other. All three of those things can impact not just what, but how, hundreds of people are doing their jobs.

And of course, none of this is to say that everyone can't or shouldn't be thinking about these kinds of things. We all should. When it comes to your career it's the consistent demonstration of your scope of influence that drives things.

by Leon Rosenshein

Breaking New Ground

Ever have to work on a new technology? Something new to you? Maybe building a gRPC client/server and sending the first gRPC message to it. Or deploying your first service on Spinnaker. Takes a long time, doesn't it? Then you look back and wonder what took so long. You've been writing code for a while. You've sent messages before. You've deployed code. This should be old hat, right? Wrong. 

Think back to high school physics and consider the lowly ice cube. Ignoring altitude, impurities and other non-ideal situations, it takes ~2.1 Joules of energy to raise a gram of ice 1° C. That's not too bad. Take an ice cube out of the freezer and put it on the counter. The water temperature rises steadily, from ~-20°C (depending on your freezer) to 0°C. Then the temperature stops going up. The room is still the same temperature, so you're pouring energy into the ice at the same rate, but the temperature is steady. That's because a state change takes a lot more energy than just heating the water. It takes almost 80x the amount of energy (333.6 J/g) to turn a gram of ice into liquid. The water is still at 0°C. No temperature change, just a state change. So lots of energy was transferred, but as far as the thermometer is concerned, nothing  happened. After that temperature goes up steadily again (~4.2 J/g) until you hit 100°C, at which point it takes a whooping 2256 J/g to turn water into steam. (Side Note: That's why steam burns are so bad. All of that energy is transferred to your skin as soon as the steam condenses)

Doing things the first time is a lot like that. It is *NOT* incremental work. It's a step change. It's a state change. From "never did" to "have done". So you need to put in a lot of "work" to make the change. After that you'll go back to your more normal pace of advancement.

So when you hit that "state change" wall, don't get discouraged. Try some things. Read some books/articles. Ask for help. Once you get over the state change things will be better.

by Leon Rosenshein

Improve The Daily Work

Following up on last week's entry on DevOps, one of the key tenets in The Phoenix Project is  that "Improving daily work is more important than doing daily work". Or maybe you've heard it said as "Work smarter, not harder".

Whether you call it friction, cycle time, down time, idle time, or wasted time, anything that isn't time spent thinking about, codifying, and implementing the solution to the business problem you're trying to solve is time time not adding value. To maximize efficiency and productivity you want to minimize that time.

We spend lots of time trying to make computers do our bidding. Parts of it are very creative, figuring out ways to do something no one has ever done or figuring out a more efficient way to do something we already know how to do. In between those parts is a lot of doing the same thing over and over again. Everything from writing boilerplate code to running tests to deploying/validating products. A lot of that is rote repetition. It takes time. Sometimes a lot of time. And a little bit of our attention. Not a lot, but it breaks the train of thought and kicks us out of the flow. Then you spend 15 minutes getting back into the flow, then it happens again. And you think to yourself, "There's got to be a better way, but I don't have time for that right now,", and you might be right the first time that happens, or even the second, but at some point, often a lot sooner than you think, you find that you would have been better off taking the time to fix your workflow instead of brute forcing it.

Case in point. The other day I was working on building some new machine images to stand up a Kubernetes cluster for our new datacenter. Turns out the easy way to test is to write the code, push it to a branch, trigger a remote build, deploy it, then see what happens. Lots of steps, but each one is easy, there were no prerequisites, and I got to mark lots of steps as done. It felt like I was really doing lots of work. But in reality it was slow. I had to start each step when the previous one finished. I had to wait for steps to finish. then test. Then do it all over again. It was a 25 minute cycle. I made some quick progress at first, but then it got slow, and annoying. I'd make silly mistakes and waste a cycle. Or I'd have to back up. As Macbeth said, it was a tale, told by an idiot, full of sound and fury, signifying nothing.

A couple of hours in I stepped back, built some tools, wrote a makefile, figured out how to do things without needing to restart a remote service. I spend about 4 hours and turned a 25 minute cycle into a 5 minute cycle. It took 8 cycles to make up that 4 hour setup time. 8 cycles, with a break, took an hour. So in less than a day I was way ahead of where I would have been. A couple of days later I was about a week ahead. And that doesn't count the time saved by teammates who didn't have to go through my struggle. If you're wondering where the breakeven point is, there's even an XKCD for that.

Whether it's a Makefile with some common commands and arguments, a fix-it week project to make something smoother, or, even better, a culture that says "If you have a choice between improving developer productivity or developing a new feature, choose developer productivity", one of the best ways to be more productive is to make it easier and smoother to do your work.

by Leon Rosenshein

Dev Ops Book Club

The Phoenix Project, Accelerate The Dev-Ops Handbook, and recently, The Unicorn Project. Two novels and a two how-to books backed by research. Full disclosure, the novels, as novels aren't that good. Character development is shallow, the plot moves at the speed of the message, and there are more stereotypes than you can shake a stick at.

However, if you can get past that there's some really good information in all of those books. At the top of the list are the 4 kinds of work (Phoenix) and 4 key metrics (Accelerate). They're at the top, not for themselves, but for what they drive, which is Value or impact. If what you're doing doesn't have value it doesn't matter how well you're doing it. You're wasting your own and other's time.

The 4 kinds of work are:

  • Business Projects - The things your customers ask for
  • Internal Projects - The things you do to make your life easier
  • Operational Changes - The work you do between finishing something and seeing value, deploying things
  • Unplanned Work  - Things you need to do right now. The kind of work the on-call is doing. Dealing with failures, bugs, and sudden changes in plans.


The 4 metrics are:

  • Lead Time for changes - The time between a request coming in and customers seeing the change
  • Deployment Frequency - How many times you can deploy on any given day
  • Mean Time to Recovery - Average time to recover from a failure
  • Change fail percentage - The percentage of changes that fail to work as designed


There are also other operational things mentioned and explained that help achieve those goals while delivering value. Things like customer focus, developers finding joy from getting into the flow, improvement of daily work, and minimizing work in progress (WIP). The first one is key to making sure you're not wasting your time and truly adding value.

That last one is also interesting and something my boss and I appear to have divergent views on, but really, that's not the case. He likes to minimize WIP. Do one thing, finish it, then do the next. And generally speaking, I agree with him. Sometimes there are good reasons to not do that. But that's a discussion for another day.

by Leon Rosenshein

Good Fences Make Good Neighbors

Speaking of bounded contexts, I love contexts, or more precisely, I love the idea of bounded contexts. Boundaries are great tools for simplification. They help you maintain separation of concerns and isolation both inside a module and between modules. And that's domain driven design. One of the things you'll find when you dig a little deeper into microservice architecture is that everyone talks about Bounded Contexts and Domain Driven Design. The important thing to remember though is that regardless of the "architecture" you're building, good architecture is good architecture. Microservices might give you more scale-out, but you pay for it with more cognitive load.

Most of the things we do with computers these days is a simulation of some real-world thing, whether it's passing notes in class, driving a car, or high energy physics. Some are mostly in our heads, some are very dynamic, and some have more interaction with the physical world than others. They have different levels of detail, but they're all models or abstractions that we simulate. And like any other model, they have boundaries. Things that are part of the model and things that aren't. And the clearer the boundaries, the easier it is to know what's in and what's out. And that's a Bounded Context. And once you know the Bounded Context for each model it's much easier to put them together into a model of the entire system. 

Of course, scope matters. Bounded Contexts isn't something that only applies to architects or how microservices fit together. You should use Bounded Contexts at every scale. Functions in a class should have clear boundaries as well. It makes no more sense to have the function that sets the input path print out the final report than it does to have your sensor fusion service control emergency braking.

So know what your boundaries are. And if they're not clear, find the person on the other side and work to firm them up. You'll both be happier.

by Leon Rosenshein

The More Things Change

Last week I wrote about the difference between an engineer and a technician. Another way to think about it is where in the technology stack you operate. 30+ years ago the tech stack was a CPU, some ROM, some RAM, and maybe a persistent storage device. And that was enough to go to the moon. Today you can find that amount of processing and storage in the wall charger for your phone. Now we've got languages that manage all of your memory for you, IDEs that point out your mistakes, refactor your code, and suggest what you want before you type it, and serverless clouds to run you code in. Or, as others have said, we're "Standing on the shoulders of giants"

Best practices and the environment we're operating in might have changed, but the core problems really haven't. Functional may have replaced Object Oriented as the new hotness, but if you go back to first principles, you find they should live together. It's just a matter of understanding/expanding the scope. 

Cognitive load of the developer is still one of the biggest limiting factors of system scale. We just have new ways (microservices, Domain Driven Design, smarter IDEs, etc) to help manage the load. Execution time is still important. Now we have more/faster processors, so we can spend time making it easier (if less efficient in isolated single case) to parallelize, distribute, and scale things out without increasing cognitive load. But we're still limited by cognitive load

Perceived wait time is still important to users. But now we have enough local horsepower/bandwidth to do a bunch of local validation instead of waiting for a round trip. We have spinning circles of hope on the screen during the roundtrip. We can send bigger/higher resolution images instead of reducing everything to some standard 128 colors. But wait time, bandwidth, and resolution are still things we need to worry about.

by Leon Rosenshein

Rightsizing

How big is too big? How small is too small? How do you decide? What criteria goes into the decision? A good place to start thinking about this is the Unix Philosophy. Small and simple. Composable. Do one thing and do it well.

And you can apply it at just about any level you want. Functions should be small and single purpose. It would be odd to see a function called `HandleFile(...)` that parsed it's inputs and sometimes printed a file, sometimes executed it, and sometimes deleted it. Similarly, you don't expect your graphics library to render images to a canvas and control a robot. Going even broader in scope, Microsoft Office and the Google Suite are collections of (mostly) single purpose tools (documents, spreadsheets, drawing, presentation) that are composable and look/feel like they belong together. But you don't want or expect to use them to control a 6 axis CNC mill.

You can go even larger than that. Think about a distributed microservice architecture. The microservices act as functions/libraries in a larger system. Separation of concerns and bounded contexts help you keep things "simple". Whether you're talking about a function, library, tool, application, or service, knowing where the boundaries are lets you maintain context.

So keep things small. Good advice. but like any other piece of advice, you can take it too far. You can make things too small. If your boilerplate comment block is larger than most of your functions you've probably gone too small. If, instead of ls and and option to be recursive (ls -R) or show attributes (ls -l) you have new commands for each option (lsr, lsl, lsa, etc) you've gone too far.


by Leon Rosenshein

FI/RI, Flag, Merge, Rebase, Trunk

How do you develop a feature? Do you go off into your own little world (feature branch), develop for a few weeks/months, then spend another week dealing with merge issues and releasing your shiny new feature on the world fully formed and integrated? Or maybe after weeks of work you just rebase, handling the text conflicts relatively easily, but then dealing with hidden logic changes for a week. Or you might have tried to just live in master (trunk) and make small, innocuous changes all along until somehow there's enough there there for users to notice and ever so slowly becomes a full feature? Or maybe you've got 1000's developers working on "features" that are roughly the size of most applications and you ~want~need to share code and release them in a big-bang event?

Lots of different options. And of course, when there are lots of options there's no one right way that always works. It's far more fluid and personal. It depends on the level of coupling between the changes you're making and the changes others are making. It depends on how long you expect to be different than everyone else. It depends on the overall velocity of the codebase. It depends on the size of the codebase and the size of the teams. It depends on your personal style and workflow, and the team/org's style and workflow.

So what's a poor software engineer to do? Compromise of course. Find the best solution for your current set of constraints. The worst thing you can do is blindly do things the same way all the time. So next time you have an option, think about it. Think about where you want the branch/PR to go in the future. Discuss it with someone. And do it mindfully, not out of habit.

So what's your preference and why? Put it in the thread.