Recent Posts (page 44 / 70)

Cycles

There's a lot of value in short sprints and frequent planning cycles. At any given time there's so much we don't know about the future needs and requirements of whatever piece of software we're working on that to write a detailed 6 month plan, let alone a 2 year plan, doesn't make a lot of sense. Moving away from waterfall development has codified that reality and helped to normalize the level of effort across the development cycle. Instead of the 3 month long crunch times (12+ hour days, 6-7 days/week) at the end of a 2 year development cycle we have a busy day or two every two weeks.

While short sprints might be better for the project, and definitely smooth out the level of effort, I'm not so sure that's good for the developers. We've done away with the crunch period at the end, but we've also done away with the low effort time at the beginning. Back in the day the first milestone was often a planning milestone. The output was a set of documents. Those documents themselves didn't have a lot of value and were often out of date before the virtual ink was dry, but writing them had a lot of value. Ideas were floated, adjusted, readjusted and prototyped. Some things were discarded and some things were discovered. Blind alleys were explored and opened or closed depending on what was found.

And for the developer there was very little pressure. Expectations were low. Time was available for random exploration. I think we've lost some of that now. In a sprint there's not a lot of time, so it all needs to be accounted for and there's pressure to make sure it directly accrues to some higher goal. And I'm not saying that's a bad thing. In fact, it's good to make sure that work done has some connection to the bigger goals. But it makes it harder to experiment. And it makes it harder to do things that take more than two weeks or don't have explicit outcomes. Yes, a Spike is a thing, but officially it shouldn't be longer than a Sprint. And it should have defined acceptance criteria. But it's not the same thing as "Here's a big hairy problem. You two folks go off and noodle on it for a month. Try things out. See what works and what doesn't. Come back with a design."

That's what we've lost. The ability for developers to have the scheduled downtime to decompress from the daily grind, then come up with something new without a ticking clock. Supersprints are good, but they're not providing the kind of freedom I'm talking about, and that's not what they're for. Whether we call it M0, a Planning Period, or something else entirely, I think we're worse off without it.

Don't get me wrong. I'm not suggesting we go back to waterfall. I much prefer the current model. I just think we need to figure out a way to make our cycle a little more dynamic. I don't want a 3 month crunch, but I do want to figure out how to get the periodic unstructured time back.

Circle Of Life

Did you know that Microsoft released a 3D world map with the ability to look at any part of the globe from any angle? Roads were mostly accurate, as were rivers, lakes, and islands. There were thousands of highly detailed 3D models of significant buildings and landmarks, and over 20K highly detailed areas. Of course, since it was Flight Simulator 2000  those 20K areas were airports and your house almost certainly wasn't one of them. I was on the team then and it was a pretty significant accomplishment at the time. We used digitized satellite data, digitized topographical maps, survey data, government data, and some 3rd party content to make the world look real.

A few years after that Google Earth was released and Microsoft, in true fast moving tail lights fashion, needed to have one. So a small group of us left the FlightSim team and went over to the newly formed VirtualEarth team to build a 3D world in the browser. We did that too. C# management in an ActiveX component over Managed DirectX. We used FlightSim data for roads, rivers, airports and all of the landmark 3D models. It was cool and all, but not many people cared. They didn't have the interest, need, graphics card, or internet bandwidth to really use it. And the world was pretty flat. So we bought a company that made aerial cameras and turned the imagery into a 3D model and helped them scale up. Oh, and we put a very simple 6DOF flight model in the rendered so you could fly around. No cockpit/instrument panel/navaids, but it felt "airplane-like". So Bing Maps is a direct descendant of FlightSim.

 By the time MS sold our part of Bing Maps to Uber we were building 1 meter or better resolution 3D models of the world from 5 - 30 cm resolution imagery. We were generating our own road networks. We incorporated data we derived from street level imagery to get road names, turn restrictions, speed limits, and business names/addresses. Because we needed a routable map for our users.

And here we are in 2020. Microsoft just released a new version of Flight Simulator. This time outsourced to a 3rd party developer. And guess where they got their data? Bing Maps. So a new game, with a highly accurate 3D world, built from satellite data, aerial data, government data, and user generated content is now available on-line (or as a 10 DVD box set).

So we've come full circle. FlightSim started from existing maps then turned into a map with VirtualEarth/Bing Maps, and now Bing Maps has turned back into FlightSim

Linkapolooza

I use StackOverflow as much as anyone when I am looking for the answer to a specific question/problem, but when I'm looking for a deeper description of a topic or approach there are usually better places to go. And sometimes I'm just looking to see how other folks do things and why. Not just to solve today's problem, but to add tools to my toolbox. You can never have too many tools, you just need to know when to use them.

The first few are very active, while the latter are more static, but still have a lot of useful info and current comments. Enjoy these and add your favorites in the thread.

https://www.developertoarchitect.com/

https://martinfowler.com/

https://medium.com/@jamesagwa/38-company-engineering-blogs-to-follow-as-a-software-engineer-c369e4be9afe

https://www.joelonsoftware.com/

http://wiki.c2.com/

No-Code

Lately I've been reading a lot about no-code platforms and the "end of programming", but I don't buy it. Not just because I'm a software engineer and want to keep my job, but because it's just another bubble. We've seen this happen before, in all sorts of fields.

First, as I wrote last year, no-code is just another domain specific language (DSL), albeit a visual one. And done well they're incredibly useful. LabView and Mindstorms are great examples of them. Any visual form builder is another. If you're building something visual then doing it visually is the way to go. For well understood situations where customization is limited then it makes a lot of sense.

Second, to use an automotive analogy, when cars were new you needed to be your own mechanic. You had to fix/fabricate things as you needed them, and only people who could (or could afford to have full-time people to do it for them) had cars. This peaked in the 40's and 50's when the dream was to get an old Ford and make it better than it ever was. Today, cars are (mostly) commodities. The majority of people in the US can operate one, but fixing/modifying them is left to the professionals.

Third, someone needs to build those visual DSLs. Doing one of them well is non-trivial. They take time, effort, and a deep understanding of the problem space. And even then, it's best to have an escape hatch that lets people actually modify what's happening.

Back in the Bing Maps days we had a tool called IPF, the Image Processing Framework. It was designed and built for embarrassingly parallelizable tasks, such as image processing, color balancing, image stitching, and mesh building. You could define steps and tasks within them, and the dependencies between steps. It was the second gen framework, and one of the things we added was a visual pipeline building. Since it was Microsoft the tool for this was Visio, different shapes represented different things, and nesting them inside each other and drawing lines between them managed containment and dependencies. And you could add any custom or required values to any shape in the form of a note, and they would get translated into runtime parameters. Things like required cores/memory, timeouts, retries, and what to do on failure.

And it was great when things were simple. When blocking out a pipeline it was great. Put in the big boxes. Draw some lines. Set the overarching parameters. Move things around as desired to see what happened and if it could be made more efficient. We did a lot of that. And for pipelines of < 10 steps we made quick progress.

But as things got more complex and the pipelines got bigger it got pretty cumbersome. For many things we ended up falling back to the XML the visual tool generated for details and just used the visual tool to move the big blocks/dependencies.

There's no question no-code and visual tools have their place and we should use them in those places. You could get all the info presented in RobotStudio from tables, graphs, and images, but playing time series data back visually is just more efficient and makes more sense. Add in the ability to swap out a node in an AXL graph and run the simulation again and see what happens can be a magical experience. That's no-code at work and we should definitely do it. But to get to that point requires a lot of code, so I don't see the end of coding any time soon.

SESE

Single Entry, Single Exit. Is it a Draconian or outdated rule? I'd argue that tools and languages have changed enough that for what most people do, it's more of a guideline that should be part of separation of concerns.

When Dijkstra first wrote his paper on structured programming things were different. In assembly GOTO was one of your only branching mechanisms and you could go anywhere. Jump into the middle of a routine. Jump out of a loop into some other block of code and never come back. And when he talked about single exit it wasn't so much having one return statement, but always going back to where you came from, not going to some other place and continuing execution. So at that time ALL structure was the responsibility of the developer.

In today's languages those things aren't really possible. Pretty much everything is part of a function/method, and it's approximately impossible to jump into the middle of a function. Some languages have some form of exception handling for not returning to whence you came, but generally you go back up the call stack until something handles the exception. So how does SESE apply today?

It still applies because the underlying desire is to help the reader understand the code. If there's one way in and you always go back to where you came from that's a big reduction in cognitive load. And we still want/need that. We always want that.

But that doesn't mean there should only be one return per method. In some cases having multiple returns can make the code easier to understand. Guard clauses, or input validation is a great example. Validating the input up front, with individual checks for each input and case, gives you a logical SESE for each check. It makes it easy to see what's been checked and lets you return (or throw) specific info back to the caller. Then, when you get to the meat of your method you know that the inputs are at least mostly correct and you don't need to worry about them, A good check for this is if you find yourself indented halfway across the page when your doing the actual work, you should probably have some guard clauses and early returns.

Similarly, there's the finally or catch clause at the end of the method. Sometimes there are things that need to happen on the way out of a method, regardless of whether or not there has been an error. This is very common with languages like C# or Java where you are dealing w/ native resources and need to manage them yourself, but also other things that don't have that kind of interop. One of the most common causes of deadlocks in multithreaded code is having a return path in a method that doesn't release a lock. A single return path makes that much harder to get wrong.

So while the notion of SESE, as written 50 years ago don't strictly apply, the reason for it is still important and there's a lot to be gained by following the principles instead of being Draconian or ignoring it completely.

Dependency Injection - NOT

I first heard the term dependency injection (DI) a couple of years ago when some folks were talking about UberFX and how DI was the hot new thing and would save us all. And I was very skeptical, because why would I want someone injecting random code into my process? Code injection is an attack vector, not a design principle. Why would anyone build a system around that?

But the thing is, Dependency Injection isn't actually injection. And it's certainly not the injection of random code without the developer's consent. I wasn't in charge of figuring out the name for DI, but if you were to ask me, I'd say they got it almost completely backwards. It's not "injection", its "demand" and "consumption". Instead of finding/creating the things you need, you just tell the "system" what you need and it hands it to you.

There are lots of advantages to this method. First, you're programing to bare interfaces, not specific implementations, so separation of concerns is baked right in. That's a big reduction in cognitive load right there.

Second, whatever system is providing you with your dependencies can update them and the next time you start you get the update. That's great for bug fixes, vulnerability batching, feature updates, and infrastructure changes. As long as the interface doesn't change the calling system can make whatever changes it wants.

Third, automated is much easier. When the system under test is created, the test framework can just hand it special implementations of the interfaces that have additional testability hooks. For instance, the infra CLI accesses both the network and the local file system. When testing the network and FS interfaces passed in provide all kinds of hooks to let the tests control the results of calls, but to the infra CLI itself, absolutely nothing has changed.

UberFX (and JavaFX I'm told, but haven't used) goes one step further. It includes a bunch of standard interfaces (network, logging, metrics, etc), and you can add your own interfaces, each with its own dependencies. Then, when you try to start the program it analyzes those dependencies and builds an ordered graph of the order to call them and then passes the results on to the next layer as needed.

So despite its terrible name, DI is actually a useful system that enforces separation of concern and control inversion. The next question is of course, "If DI is so good, shouldn't we use it everywhere and stop linking against libraries?", but that's a question for another time.

Ideas In The Shower

I don't know about you, but I've solved a bunch of problems in the shower. It's the perfect combination of no distractions, some manual activity, and no immediate cognitive tasks. It gives the brain a chance to wander, sift, and make connections.

Sometimes you need to step away from the problem you're trying to solve and not think about it so hard. Step away from the pressure of a blinking cursor or a blank whiteboard demanding to be filled. That freedom lets you change the scope of your thinking. It lets you focus on the big picture and not get bogged down in the details of syntax and language or where to put the box on the page to avoid crossing lines. occasionally you end up with a Eureka or A Ha moment, but more often, and at least as useful, the broad outline of the solution to your problem will come to you. That doesn't mean that the full solution with all the details is suddenly known. There usually lots of detail work left (and all the language and syntax stuff), but you'll have a framework to hang all that detail on.

Back when we were in an office it wasn't unusual to find me walking around the office quietly or stopping to play a few rounds of darts. It wasn't quite as good as the shower, but it came close. And during those times I was able to come up with approaches to multiple problems. Sometimes solutions that were bigger than what I was originally looking for. Like when I was working on that map visualizer. I was looking for a way to keep the views in sync for the map visualizer and realized that I wasn't talking about two points (look from/look at) and a rotation, but a point, a unit quaternion, and a direction (at/from). It was not only less data, but it also worked for syncing across and between 2D maps, 3D Maps, BirdsEye views and StreetSide image cubes. There was still lots of math to be done to make it work, but that framework gave us the context to build it in.

XY Problem

Every now and then a question/request will come up on Slack or in a conversation that seems odd, but in an attempt to be helpful I'll answer it and a follow-up question or two. And then the next question breaks my mental model of the situation and I do an internal double-take. In those situations my response is usually something along the lines of Wait, what is it you're really trying to do?

And that's an XY problem. The kind of problem where you each step requires a more esoteric and specific solution from a generic tool. When you find yourself in that situation it's usually a good idea to step back and make sure you're approaching things the right way. You'll usually find that if you work with the system instead of against it things will go much smoother.

It often happens when your go-to tool isn't quite the right tool for the job. sed is in fact a turing complete language, meaning you can do just about anything with it, but that doesn't mean you should use it for your problem. It's great for examining/modifying text one line at a time, and is OK for multi-line text, but beyond that use something else. xargs is where I often find myself getting too far down into the weeds.

The other day I was trying to clean up some kubernetes pods, and selecting the right ones was a little tricky. It seemed easy at first, but as the condition got more complicated the amount of escaping and nesting was getting out of hand. I fought with it for a while, consulted the bash grimoire and stack overflow, but eventually realized that the best solution was to stop trying to mash it all into one line and use an external script.

It never quite got to the point of writing a Golang program to handle it, I came really close, and I promised myself that the next time it needs to be extended I'm moving to Golang and taking the opportunity to merge 3 custom scripts into a more generic, extensible program.

Sometimes you just need to go back to first principles. Look at the requirements, look at the non-requirements, and start down a different track with the knowledge you've gained. The trick is to balance the determination to see a plan through as written against the possibly slightly longer time to get a more maintainable, long term solution

Good Design

What I'm NOT Doing

You know what's just as important as what you're working on but almost never gets talked about? What you're not doing and why you're not doing it. And that's a problem. There are a bunch of reasons why describing what you're not doing is just as important as describing what you are.

First on that list is that it helps you decide if a task fits or not. Let's say you're working on a function that takes in an image and calculates a histogram. Simple enough. You do your RFC, you pass around the API and everyone is happy and you begin to implement it. You're almost done, and someone says "We've got this other project that uses compressed images. Why aren't you able to handle that?" You know that wasn't part of the plan, but not everyone knows what's in your head. If your docs explicitly said "This will not handle compressed images then either requirements and schedule would have been adjusted or someone would have built the shared converter. At the very least you wouldn't have been blindsided by the request.

Along those lines, it helps to clearly define domains and separation of concerns. If you know and share what you're not doing, it's much easier to see the gaps between components and fill them if needed. Right now I'm working on a shared Authentication model for users and services that spans clusters. I'm not working on Authorization. It's a very closely related item, and the same group of people will probably be at the core of that too, but it's a very different problem. And our docs are explicit about that. Yes, we know it's coming and we know how it will fit in, but we're not working on it now.

Another way non-goals can help is when you're defining the actual task list. You have a set of goals, and you have an idea of what the end-state looks like, but something about it isn't quite right. And you keep adding new requirements to try and bend the result into what you're looking for. You end up with some really specific rules. Something like "schedule the meeting before 3:00 unless it's a Friday, then do it before Noon, but don't schedule it for early on Monday, or around lunch". That might be easier said as "Don't schedule the meeting when people are catching up on old work or will need to leave before we're done, even if it runs long" That's much simple and covers the real requirement.

So next time you're defining what you're doing, regardless of the scope, make sure you include the things you aren't doing and don't want to happen as well.