Never Trust the Software Guys: Why is writing software hard?

My first response is a yet more questions. Is the question why is writing software hard about an individual software developer going about implementing a module or sub-system? Or is the context of the question about the difficulties an organization or company face when developing software?  I’ll address both since developers do not work in a vacuum

The maturity and culture of the organization has a huge impact on the success and/or failure of a software project.  A positive organization will remove obstacles, while a negative organization will create more obstacles than it removes.  Since every organization is different, each has a different list of what it does well and what is does poorly. In my experience , it is not that the items in theses list are wildly different, but what items fall into the things we suck at category.  That said, two of the biggest obstacles are ambiguity and indecision.  Read ambiguity as the inability to articulate, define, and capture the requirements for a project.  Further on the bad ambiguity spectrum is to define a business, revenue, and operating plan based on a two-sentence description of product that company wants to develop and the company has no similar products/experience on which to baseline the effort or scope. This approach to software development puts the entire development team behind schedule even before it is formed (Okay this was bit of rant – but it’s frustrating to see history repeat itself again and again).  Ambiguity is almost always accompanied by indecision. It make sense – if you don’t know what you want – how can you decide when its right?  Briefly, indecision is the inability to commit to requirements once they have been defined.  To borrow a well-known quip:

Walking on water and developing software from a specification are easy if both are frozen.

Edward V. Berard

Moving on to the context of an individual developer and purely technical consideration of what the most difficult aspect of writing code is.  <shameless plug>Circa 2000 I created the eXtreme Minimal Kernel open source project</shameless plug>.  XMK is/was a preemptive multithreading real time operating system for microcontrollers.  XMK was designed from the ground up to be very small with respect to ROM, RAM and CPU processing resources.  The fun and exciting part of this project was the challenge of solving the XMK’s mission statement. However, the hardest part was not the design or writing the code – it was validating that it worked!  Especially difficult to prove out was that there was no underlying race condition(s) that crashed the kernel or caused improper behavior.  Testing was further complicated since the thread scheduler and basic primitives had to be validate on hardware targets. By definition of the problem space, the hardware targets did not provide an “instrumentation friendly” environment and this situation was further aggravated by almost no budget for hardware or tools. A side note: the majority of the APL layer of XMK could be and was tested on a PC using functional simulation techniques.  

Okay, so I talked about my personal challenge, now let’s talk about developers in a general sense.  As a manager and team leader where I see many developers struggling or following short is during the analysis and design phase of development.  If the analysis is wrong, the design will be flawed, and if the design is flawed the code will never survive – without significant refactoring – the next added feature.  And one developer’s refactoring is other developer’s rewrite (<editorial>it’s hard to be first to market if every project is a one-off</editorial>).  The reality is that good analysis and design skills are just that – skills.  Not everyone has them, not everyone can acquire them, not everyone wants to invest in acquiring them.  One of my epiphanies as a manager is that not all developers need to be strong analysts or designers, and don’t build a plan or a framework that requires everyone to have good analysis and design skills.