This posting is the first in a series of postings (okay rants) on what goes wrong with software development, my commentary on why that is, and what can be done to improve the situation.
Disclaimer: I have been an embedded developer all of my professional career so not all of my content is applicable to general software development — YMMV.
You can’t schedule invention
There are a lot of reasons why software projects are late:
- Improperly staffed
- Started late
- Poorly managed
- Impossible requirements for the cost target
- No requirements (i.e. no definition of done)
- Constantly changing requirements
- And the list goes on …
But I am going to focus on the fact that software development is fundamentally a creative activity. Why does that matter? Because you can’t schedule invention!
Let’s step back a bit. If a company undertakes a new software project — the assumption is that the project is intended to fill a need that is not currently being met — or why spend the resources to develop it. Now the new project/software may not be new in the greater world of software or products — but it is new to that company. Furthermore, since it is something new, then by definition there are unknowns. And that is where the creative activity comes in — someone has to discover and provide solutions for those unknowns. If you don’t know the details, then you don’t know what the effort will be, which means you can’t say it will be done on April 25.
Now the term unknowns covers a lot of ground. It can range from: no one has put a man on the moon before, to never having used that particular microcontroller, to what is the regular expression to parse XYZ. Unknowns in this context is a spectrum and has an inverse relationship to accurately predicting the effort required to resolve the unknowns. See the diagram below.
How to go about scheduling the un-schedulable?
On paper this is easy:
- Have a time boxed concept phase (insert your company’s terminology/phrasing here) that one of its major goals is to eliminate as many of the unknowns as possible to before proceeding to a develop phase.
- The goal of the develop phase is to productize what was learned/discovered in concept.
- The supposition being that the develop phase requires minimal invention.
- If at the end of the concept phase, there are too many unknowns (aka too many missing details) — then a new concept phase is started, or the project is tabled.
- At the start of the concept phase the there is targeted delivery time frame for the completion of the project.
- Once the develop phase starts, the targeted delivery time frame can then be refined — with the current state of affairs — into a scheduled release date.
In fact, most companies I have worked for over the years have a New Product Development phase/gate process that encompasses all of the items above. And yet the software projects are still late.
This is where I have seen the phase/gate process go wrong
In the embedded space, software development is a usually tertiary activity, e.g. a HVAC manufacturer sells heat pumps that coincidentally contain a control board with firmware. One side effect of this, is that the company’s phase/gate process and culture is tailored for developing their ‘box’ products. Which means that the business’ expectation of what occurs (and the durations) during the concept and develop phases is based on designing boxes. Furthermore, software development is not considered as an unknown for the concept phase and doesn’t start (think funded/resourced) till the develop phase. This would be okay if the box develop phase was sufficiently long enough to encompass the software’s concept and develop phases — but typically it is not. And to make matters worse the develop phase for a box product can be relatively short when it only consists of generating the drawings, documentation, and build procedures for production.
Another common misstep I have experienced is the assumption — by the powers that be — that a schedule can be created right to left (i.e. a end date is picked and the tasks/durations are worked backwards from the end date). This is wrong in so many ways — but it completely ignores the fact that the details matter and until you know all of the details — a schedule is just wishful thinking.
Summary
When leadership’s expectations or understanding of the software development activities does not take into account the creative/inventive nature of the work — your software project will be late ✋ 🎤.
Another commonly misunderstood facet of creativity is that creativity requires a large amount of “wasted time.” To foster creativity, there needs to be time and space to play around with things, to explore tangential ideas, and to let curiosity lead you away from the matter at hand. By definition, the act of being creative connects new and unexpected things together. But if you’re trying to come up with creative solutions to problems, but all you’re focused on is your software, and your domain space, and your business needs, you’re not going to come up with anything truly creative. This is not to say that you can’t be creative on a deadline. You can, but you have to expect that people looking in at the process will ask “Why are you wasting so much time?” And the answer is, it’s not “wasted time;” it creates the space that will allows us to make connections beyond the ordinary, the habitual, and the commonplace.”