Over the course of many years working as a embedded software engineer, I’ve seen the same things over and over again. At different companies, I’ve seen the same ideas about design, the same approaches to project management, the same group dynamics, the same mistakes, and, sometimes, the same successes. Consequently, I thought it might be worthwhile to share some of these observations. While I’ve codified these observations into a set of rules, they are really just things you would hear me say if you were around me at work. And I say these things a lot. Maybe because sometimes it just helps to say them out loud.

John’s Rules of Development

Some of these rules are original, and others have been shamelessly paraphrased or borrowed from coworkers. As with all generalized rules, there are always exceptions, except for rule #1.

1. Never trust the software guys.

This rule covers a multitude of sins such as

• Software developers always have bugs in their code.
• Software developers are terrible at estimating effort.
• Software developers’ priorities do not always align with program managers’ priorities.
• Software developers are either too conscientious (e.g., this must be a 100% solution) or not conscientious enough (e.g., “we don’t need no stinking testing”).

In all seriousness, I use rule #1 with managers, project managers, program managers, peers, and my spouse, to remind everyone that software guys are human, and, as such, our work has imperfections.

2. Software is not soft, it’s hard.

Software is complex, challenging, and time consuming and requires skilled professionals—no matter how much management, marketing people, and customers would like it to be otherwise. Unfortunately, in the embedded space, software development is a tertiary activity. Many stakeholders simply do not grok the software development process. So, in addition to developing software, you also get to spend time and energy educating folks about why things are taking longer than their emotional expectations.

3. “Similar but different” is still different.

Early on, software developers are introduced to the magic of subroutines. Then they learn about generic libraries, classes, and frameworks whose miraculous functionality reduces a lot of the grunt work and heavy lifting of coding. However, once we have a hammer, it tends to become a golden hammer. This leads to a tendency to hit similar things with the same hammer, even though they are different. The classic example of this would be treating squares and rectangles the same since a square is a rectangle. However, a rectangle is not a square, and you will encounter errors if you create a single interface for handling rectangles and squares. In short, do not force things to be the same. And always follow the Liskov Substitution Principle (LSP).

4. The hardware is always late.

Hardware is hard, lead times are a bitch, and PCBAs cost money. Enough said.

5. There are always new requirements.

The only constant in life is change. This is especially true with software requirements. You can rage against change, or you can embrace it. I discovered in my career that embracing change and finding ways to construct designs to be “change tolerant” was much easier on my blood pressure. In addition, while change is constant, you can never fully anticipate what will change. This leads to the corollary principle: 80% of your code never changes, but the 20% that changes can come as a surprise. The simplest way, then, to address not knowing what code will need to be changed in the future is to start with the assumption that all of the code will change.

6. Don’t ask open-ended questions.

One of my first software projects as an intern was to automate a test station for testing a protection circuit inside a compressor to prevent motor damage. There were no formal, documented requirements, and I was told to work with one of the mechanical engineers on what the test station needed to do. I naively asked the mechanical engineer, “what does the test station need to do?” His response was a verbal list of requirements, which I went off and implemented. After I was finished, I demonstrated the test station. The response was, “that’s great, but how about we add A, and do X instead of Y?” This cycle of getting new verbal requirements, reimplementation, demonstration, followed by a change of verbal requirements continued over the course of the entire summer.

The lesson I took from this is that customers often don’t know what they want, and it is futile to expect that at some point they will. Consequently, if you ever want to be done with a project, only ask if they want option A or option B. Never present an option C.

7. Testers are your friends.

There is always some tension between the test team and the development team. This is understandable because it’s a tester’s job to find and report all of the mistakes the developer has made. That said, a friendly tester will make your life better, if for no other reason than they can give you a “heads up” on problems and give you a chance to fix or identify a root cause before the bug becomes “public.” And having to explain a bug to a stakeholder when the root cause is unknown is an uncomfortable experience. To put it another way, an alienated tester will cause you heartache and pain.

8. Just move the cursor to the right.

Most software developers struggle when it comes to writing documentation. (I include myself in this group; it’s why I had a coauthor for my books.) Documentation tasks can seem overwhelming. Sometimes, the best way to start is to just start typing, not worrying about format, good grammar, logical progression, and so on—just to set down ideas in some form. After some content is “on paper,” as it were, it is a much simpler task to then organize, format, and wordsmith it into a finished document. To say it another way, writing and editing are two separate mental processes; don’t try to do them both at the same time. Just start by moving the cursor to the right … and keep it moving …

9. They only remember the number, never the preconditions.

My first introduction to this management principle was when I gave a PowerPoint presentation to the directors and VP of engineering about the development for a new controller. On the first slide, in big bold red lettering was the caveat that the proposed timeline of six months did not start until tasks A, B, and C were completed. However, somehow the directors and VP left the meeting with the idea that the new controller project would be done in six months from today. I tried later to remind them that six months was dependent on A, B, C being completed first, but their response was “yes, but you said six months.” Know that even if your manager asks you for a nonbinding swag for when a piece of work will be finished, your answer will be binding.

10. Money is always an issue.

I have been involved in too many planning sessions where the solution to saving a project schedule was to throw money at the problem. That is, we could hire new developers, invest in hardware and tools, and co-locate everyone to a common site. As stakeholders were leaving the meeting, we were filled with a sense of accomplishment. However, after the reality of spending the money—or, more precisely, not spending the money—set in, the schedule was doomed. Unfortunately, the stakeholders’ expectations for the schedule did not change even though the money was not spent (see rule 9). Consequently, never base your plans on the assumption that additional money beyond the current budget allocation will be available.

11. Never believe never.

Never is a trap. It’s similar to having a pet cat roll over on its back and present its nice soft fluffy stomach to be petted. Inviting as it is, as soon as you touch the cat’s fur, claws and fangs will dig into your hand. The same thing happens when a stakeholder, marketing genius, customer, system engineer, and so on state that “the product will never have to do X, and you shouldn’t spend any time or effort accommodating it in your design.” Over and over, I’ve been told “never,” and then all of sudden, X becomes a must have feature that I get to spend my nights and weekends working on. Here are some examples:

• “This is just a prototype and doesn’t need to be production quality code.” But after the demo, management now needs it to ship tomorrow.
• “The contract manufacturer will develop the end-of-line testing software.” This usually lasts until someone actually gets the quote from the CM for the work.
• “There will only be one version of the hardware and one version of software that has to be supported at any given time.” This usually goes away as soon as you release a version of hardware to customers and start on the next board spin.

12. All developers are not created equal.

This is about setting expectations. Whether you are an individual contributor, technical lead, frontline manager, or director, be prepared for the human element. It is also a bit of a personal rant against the corporate culture—the actual culture, not the pretty posters on the wall—that treats developers as commodities that are all interchangeable. Developers are individuals. As individuals, they all differ in their experiences, interests, abilities, and skills (both technical and personal). Don’t assume that your coworkers are just like you.

13. If you can’t write it down, you can’t code it.

This is a paraphrase from the Albert Einstein quote, “If you can’t explain it simply, you don’t understand it well enough.” If you can’t describe your design in natural language, with some supporting diagrams, how can you realistically expect that you can code it correctly? And how can you expect someone else to understand your design or implementation?

Here is a very simple sanity check you can make when designing a feature. If you are struggling to create the design document, or struggling to write comments for a class or interface, you probably need to think through the problem more thoroughly. Do not spend time writing code that will almost certainly have to be reworked or thrown away. My personal habit is to write the comments first, before starting on the implementation. Incorrect assumptions and bad design choices quickly show up when you write out the comments for a piece of software. Another good way to catch bad design choices is simply by talking out loud with another developer about it.

A corollary to this rule is, if it doesn’t make sense when you say it out loud, start over.

Wayne’s Rules for Development

Wayne is the co-author of my books. He helped me codify my rules – and he has a set of rules himself.

Some of these rules are overstated for effect. That is, while there may be exceptions and contexts where a rule might not entirely apply, adding the extra language made them seem less pithy.

1. People are messy.

Code works or it doesn’t. Compilers and linkers work the same way every time you use them, and when something unexpected happens, it’s usually because the internal logic or semantics of the tools weren’t understood well enough. Unfortunately, software gets developed by people who use these tools, and people are messy. They get married, they get divorced, they get their appendix out, they get stuck in the Florida Keys, they don’t like to wear socks but do like to take their shoes off in a shared office space. All the unexpected and frustrating things people do will, to one degree or another, affect the project. And there’s really nothing you can do about it except recognize that it will happen. When messy things happen and interfere with my schedule or my responsibilities, I find it oddly calming to shrug and say “people are messy.”

2. Projects have a beginning, middle, and end.

This sounds really obvious, but it is fascinating to see that software projects are nearly always managed the same way from start to finish, even though people are doing very different work at the beginning than they are at the end. I have always advocated that the beginning of a project would be best managed with a waterfall methodology, and then, somewhere in the middle of a project, it should be managed with Agile. Expectations about what is getting done, how it is getting done, and how long it should be taking
should always be considered in the context of whether you’re at the beginning, middle, or end of a project.

3. All software resists shipment.

No matter what your release date, there are always last-minute features that become critical and last-minute bugs that are uncovered. All of these things will reset your release timeline. Additionally, there can be noncode, nontechnical activities that slow things down like licensing reviews and export control paperwork. And the bigger the project is, the more people there are that can come up with reasons and roadblocks that force a reset of the release timeline. Don’t be fooled into thinking, then, that after the last line of code has been written, the hard part is done. You have to beat software out the door with a stick.

4. Levels of indirection decay over time.

Levels of indirection are what make it possible to reuse code and build lots of different things with minimal additional work. However, the cost of this sort of architecture is an additional complexity. In cognitive terms, more mental energy is required to keep the levels of indirection in mind when you’re working with the software. And like energy in the real world, entropy will eventually catch up with higher energy levels.

I worked on a project years ago that I thought was really ingenious. From the same source, I could build three different variants of the product in multiple languages (including branding for an OEM product). When I eventually left the company, I trained my replacement on the organization and build process, and that person hated all the levels of indirection. That person insisted that the only way forward was to break up the system into three different source trees that would be kept in sync manually. I was flabbergasted. But all my boss could say was “that’s the person I’ve got to take over the job. You want to stick around and keep it running?”

Pursuing a PIM¹ strategy is always a good idea because it will make your life easier and make you nimbler and more efficient. But be aware that it will require constant care and feeding on a people level. That is, it requires continual training and ongoing communication with your team about what you’re doing and how it supports the team’s goals in order to keep things humming along smoothly. Good software engineering practices require discipline by the developers and stakeholders.

1. See Patterns in the Machine: A Software Engineering Guide to Embedded Development ↩︎

6. Design reviews are better than code reviews.

In my experience, the perennial problem with code reviews is that they miss the forest for trees. That is, code reviews tend to focus on the syntax of the code and don’t really look at the semantics or the overall design of the software. In my experience, code reviews can catch implementation errors, but they don’t often expose design flaws. And design flaws are much harder and more expensive to fix than implementation errors.

In a perfect world, you would want to perform both code reviews and design reviews. But if you’re pressed for time, you’re better off reviewing code by asking “What are the semantics of this class or module? Where does your code implement those semantics?” Or, alternatively, “What is the algorithm for this function? How does your code implement it?”