Recent Posts (page 62 / 69)

Doc Culture

Doc Culture I've touched on this a couple of times before, but never focused on it. ATG is a large organization. There are lots of people to talk to and lots of channels to talk through. We also have lots of meetings. So many that teams are starting to institute "No Meeting xxxday" so people can focus for extended blocks of time. Uninterrupted time for focus is great. We should have more of it. Clustering meetings definitely helps, but it doesn't actually reduce the meeting load. What if we could do something to make meetings more effective and reduce the total meeting load? That would be a real advantage. A place to start is to think about why you're having the meeting and decide if it's necessary. The second is to to decide who needs to be there, which depends greatly on the answer to the first question.

There are lots of reasons to have meetings. For the sake of this discussion let's talk about the 4 big ones. Information Sharing, Brainstorming, Collaborating, and Decision Making. For information sharing meetings, do you really need it, or can you just send out an email/doc with the information? Sometimes realtime feedback or the topic is important enough that it makes sense, but think about it. Brainstorming/jamming requires people to be together (at least virtually) and have real-time interaction with each other to build on ideas, but who needs to be there? It's important to get diverse perspectives, but i a person is unengaged then they're not helping anyone by being there. Let people vote with their feet. Similarly, with deep collaboration, a small handful of people, focusing on a topic can be very effective. Bystanders, kibbitzers, and lookie-lou's just slow things down, so think about who's there.

Decision making. There's something we can really work on. How many meetings have you been to where the person who called the meeting asked "Has everyone read the doc?" and most folks looked down at the table? Or you get halfway through the meeting and realize that the person who should be making the decision isn't there? Or the walls are lined with folks need to know the answer, but don't really have any input? Think of the person-hours we could save if we made that part better.

Another way to make decision making meetings more efficient is the Amazon approach. Make sure you have the right folks in the room, both decision makers and people who can provide both overview and detailed information. Start the meeting by presenting a written narrative discussion of the problem and proposed solution and have everyone take 15-30 minutes and read it. Then, with an agenda and that context, discuss the points, make a decision and leave. Of course if it were that easy everyone would be doing it. It takes work, It's much harder to write a doc, in full sentences, that convinces people then to put together a set of bullet points and steamroll to a pre-ordained decision, but doing it the hard way gets a better decision, wastes less people's time, and when you're done you have a record not only of the decision, but why it was made. And, it puts the burden of communication and making sure the right message is received on the person sending the message, which, according to communication theory, is the right place.

SLOs, SLIs, SLAs

Objectives, Indicators, and Agreements. What’s the difference, who do they apply to, and why should you care?

The Objective is the vision you share (with your customers). It’s what you want to do. It’s how things would look in a (realistic) perfect world.

The Indicator is how you know what’s really going on. How you define your indicators is critical, because if the don’t measure and report on your objective your setting yourself (and your customers) up for failure.

The Agreement is the contract you have with your customers. It’s the promise you make about how close to your SLO you’re guaranteeing, and what you’re going to do if you don’t get there.

And before we get to far along, I want to respond to that mumbled comment in the back row about this not applying because “I don’t have any customers”. Yes, you do have customers. Unless you’re doing some side project that impacts no-one and you never tell anyone anything about, you have customers. From the obvious things like a service guaranteeing to respond within X ms, to a single number in a weekly verbal status report, someone is looking at the information and functionality you’re providing. That person is your customer. The person could be on your team, could be on a elated team, could be a data scientist pulling data out of the data lake, or even, one day soon, a person on the side of the rode that pushed a button and expects a care to show up. We ALL have customers.

Now that that’s out of the way, back to the SL’s. You care because they impact everything you do. You design for your Objectives. You monitor your Indicators, and you operate to your Agreements. And you don’t just care when you miss your SLAs. Your SLA’s can help you way before that.

Remember when you started to design that system and you had to cut something? How did you know what to cut? How did you know what could wait until the next release? You used priorities. And where did those priorities come from? They didn’t appear fully developed and delivered by a PM as holy writ. They started with the Objectives and Agreements. What do you need to do, what do you want to do, what can you not do, and when do they need to be done by? That’s why you care about SLOs and SLAs you write the first line of code.

As for the SLIs, you get what you measure. SLIs don’t just wake you up in the middle of the night. If they’re done right they let you sleep in peace because you know things are going to be OK. They let you know when things get close to a red line before they get there so you can do something about it (and sleep peacefully). They do the same for your customers. They know that they can rely on you and can focus on their value add. They can tell when a problem is theirs and not bother you.

Open Source

We all benefit from it. I know I do. The number of open source plugins/add-ons/tools I have had to install to make OSX useable as a developer system is way higher than I think it should be, but there it is. I've done a little bit to help out open source projects, from HDFS to Docker to Peloton. Not PRs or commits, but bug reports, beta testing, code contributions that someone else added, but the benefits I've getting fro OS tools far outweighs what I've put in. And that's true for just about everyone I've talked to or heard about, but not everyone. So how do those folks get compensated? What do we expect from them going forward? I think everyone should get to make that choice for themselves, but it can be a slippery slope. I don't think it's as bad as this article says, but it does raise some good points. How do we make sure we're not valuing "code" over "people"?

One On One Meetings

What are they? That's easy. They're (usually) regularly scheduled 2 person meetings with a manager and direct report. What are they good for? More fundamentally, who are they for? Once you know and internalize the last question answers to any other questions become much clearer, so let's tackle that part first and be very clear.

1:1 meetings, whether between a manager and direct report, a skip level, or a meeting with the CEO, belong to, and are for, the subordinate. Not the manager. Not the Director. Not the CEO. That's not to say those people don't get value, but the 1:1 meeting belongs to, and is for, the subordinate.

What are your 1:1 meetings good for? As much as I don't like design by counter-example (there's a topic for the future), 1:1 meetings are NOT status meetings. Status may come up in the course of a discussion, but that's not why you're there. That said, they can be about anything you want, because it's your meeting. Things that they're commonly used for are giving and getting feedback, understanding what your manager really values, identifying blockers and enlisting help, finding and taking advantage of opportunities, bringing up new ideas and issues, and career development. A corollary to owning the 1:1 meeting is that you also own the growth of your career. Your manager is there to help you and help you take advantage opportunities, but the responsibility is yours.

So take ownership of your 1:1 meetings. Make sure they're scheduled. I find weekly is best because if you miss one it's only a week. If you have them scheduled every other week and miss one suddenly they're monthly meetings and that's probably not frequent enough. Be flexible, but don't let you manager slide every time. Be willing to move an instance, but have it. Have an agenda, and share it with your manager early so they're prepared. Keep a documented record of the meetings, including notes and action items.

Old Passwords

Nothing serious today, just Ken Thompson's old password. Hopefully he's changed it by now.

Writing It Down

RFCs are critical. None of us know everything about every system, deeply understand all of the customer use cases, or can anticipate what some other group is working on. An RFC gets your ideas and plans down in a shareable and discussable format. It's about documenting a specific solution to a problem and gives others insight into what you're doing, why you're doing it, and how it might impact them. It lets others use their experience and point of view to help avoid problems in the future. This is critical to what we do and we should not only do more of, we should do more to at least be aware of everyone else's RFCs, commenting and questioning where appropriate. You can think of an RFC as a mission statement for a team/project.

But today's entry isn't about RFCs. It's about what comes before the RFC. The problem with RFCs is that they're largely static and represent the thinking at the beginning of the implementation cycle. Why are you writing that RFC anyway (hint: it's not because you need to check a box for next year's promo packet)? Whether you call them Filememos, Architectural Decision Records, RFPs, Vision Documents, or something else entirely, you write them to answer that question.

Writing the vision doc means understanding the problem you're solving, getting agreement on the required use cases, and documenting the goals and why those are the goals. It's about providing context for the next person (and sometimes that person is future you) who needs to care so they are working to the same goals and can make a choice compatible with those goals.

Development takes time. Existing use cases get changed and new ones get added. New people join the team. Multiple decisions get made every day. part of making those decisions is how things fit into the detailed design as put forth in the RFC. Equally important to those decisions is the business problem we're trying to solve. Vision docs describe the end state we want so those decisions, regardless of who makes them, are all in service of the same end state.

It doesn't matter if you're architecting a whole new self-driving vehicle, building an entire ML pipeline, or adding an endpoint to an existing service, you won't be working alone. For something the size of self-driving vehicle you need to understand the problem you're solving and make sure you, your co-developers, and especially your customers understand the problem you're trying to solve. Features are great. You solve problems with features, but your customers don't buy features. They buy solutions to problems (use cases). A vision doc keeps those solutions in the forefront at all times.

Pattern Matching

We are pattern matching machines. We love them and we'll find and match patterns even if they're not there. From raindrops on a sidewalk to the stock market to assassinations of US Presidents, we project patterns into statistically knowable events and then make predictions. If your take the long view you'll do OK, but Hari Seldon's psychohistory? Maybe we'll get there one day, but today is not that day. The same applies to large, interdependent systems. We know they're going to have problems, we might even be able to predict how many will happen over a sufficiently long period, but we can't accurately predict when the next one is going to happen.

A Black Swan is a seemingly random event that, in retrospect, we can see that it was inevitable. Just like the black swan, we don't know exactly when one of our systems is going to have a problem, but we know it will. Knowing that it will happen, we can prepare. We can think about a taxonomy of problems and put addition systems and processes in place to minimize their impact. Here's one such taxonomy and how we can learn from it.

Lego

In today's Lego related ideas, and in honor of tomorrow's team build of the Imperial Star Destroyer, Systems Thinking at Work and Play. Systems are all around us, and there are lots of educational shows that show them. From "Dirty Jobs" to "The Magic School Bus", understanding how things are built out of parts and work together is the system mindset. One of my favorite systems, and it has been lo these many years, is the Lego System. Yes, when you dump 4784 bricks on a conference room table it's not a system, it's just a pile of plastic parts, but when you put it together it's more than the sum of its parts, it's a system of reusable components. Add in some Technic parts and/or a Mindstorm kit and now you're really talking. But what is it about systems that is so interesting and important? To me, it's the fact that you can take things that individually have no, or very simple, responses and then combine them, and through their relationships create complex responses. Systems are also a way to manage complexity. Boundaries and relationships help us understand our world (or our SDV) and let us focus on individual components as needed.

In The Beginning Was The Command Line

Back in the old days the competition for users came down to Operating Systems. OS's were proprietary walled gardens and everything was done in service of selling the OS. Sun and Solaris owned the Internet (the network is the computer), Microsoft and Windows were going after the desktop (a computer on every desk), and Apple had the creative niche (think differently), and BeOS and NextStep were out there selling themselves as better than everything else, but no-one was buying. Except for that oddball OS in the corner, Linux. It's available online, you need to know what you're doing (or have access to someone who does), doesn't support all of your hardware, and it's free, but outside of those constraints, it just works. Oh, and it lives by the command line.

20 years ago Wednesday, Neal Stephenson released his take on the battle. It has car analogies, fan-bois, cultural relativism, and oral histories. A lot of it is either dated or no longer relevant, but the themes certainly still apply, and Linux is still Linux. And the command line is still there, and we spend oh so much time using it.

Understanding the past is key to the future, whether it's user interface, architecture, cultural directions, market forces, or AV platforms.

Statistics

Big data is all about statistics, and lots of what we do comes down to statistics. Precision, Recall, MPBE, Code Coverage, SLAs, expected battery life, Texas Hold-em, and more. We look at a large enough sample, do some math (or let a computer do so math that we may or may not be able to follow), pick a threshold, and declare that meeting it is enough. No-one has found a better way. Of course none of the statistics guarantee how an individual instance will go. We can know that 99.9999999% of calls to a web service will take less than 150ms, yet it's still possible for 3 calls in a row to take 200ms. So what's a poor developer/engineer/data scientist/PM to do? It's the old boy scout motto, "Be Prepared". It's defense in depth. It's adding watchdogs and timeouts. It's fallback positions. And it's being honest with ourselves that statistics and probabilities are not certainties, but that's the way to bet. Here's a view of how it can impact results and decisions in the medical field.