"In fact, my main conclusion after spending ten years of my life working on the TeX project is that software is hard", Donald Knuth said. "I found that writing software was much more difficult than anything else I had done in my life", and "software was an order of magnitude harder".
To put this into perspective, TeX was primarily written by Knuth; he was affected a lot less by many of the activities that make software development hard — most software development is performed:
- with a group of people who collaborate to create the software,
- governed by budgets on money and time,
- developed for people (usually who paid for the work) who want an exacting implementation of what they want (but are unable to express it),
with each of these aspects changing over time.
Letting Knuth speak for the programming aspect of software development, there's still a vast array of hard problems in software development.
According to Fred Brooks, "the hardest single part of building a software system is deciding what to build". Not that you're building "software as a service customer relationship management software" (although getting there is certainly hard too), but figuring out what to build by determining what the various stakeholders of the software development process want, what the cost of those changes are, and deciding which to do, and in what order. And this changes over time — the malleability of software is one of its great strengths if used correctly, and one of its greatest weaknesses when not.
You want to make sure that what you have built is what you think you've built before you release it to the world. Quality (including such things as security, scalability, and performance beyond just reliability) becomes increasingly harder to do the further you progress, especially if you leave them to the end.
Now you actually put it in front of users - packaging it up in a particular way, possibly deploying it, and supporting it. Hoping not to have the whole thing fall apart at this point, and standing ready in case all your careful planning has still missed something.
Then you need to balance fighting bugs, providing support, possibly managing the application in production with hardware and configuration, getting feedback from your users.
And while the fire is still raging, you start from the beginning again. You are better off this time around, since you have real feedback from your users. You are worse off this time around, since this feedback means different things to different people. You have new changes being proposed, which kick off "phase 2" work on items developed in previous versions. You probably made a few short-term decisions that need attention. You realise you made a few design mistakes last time. These might make it into the plan, or they might not. Maybe you'll have time to sneak some stuff in by going around the process...
All the while, you've got people joining the fray who need to learn your code base, your company's methods and processes, and your development team's processes, and people leaving taking away the experiences that have shaped the current environment and with knowledge that will need to be relearned within the team.
The dynamics of a software development team is very different to many other teams — it is often hard to evaluate the effectiveness of individuals, the quality of their work in isolation to the rest of the system, and the other effects their presence has on the team (positive or negative) besides their official output.
Software development and programming itself have their own fractured group of religions each formed around a hierarchy of values. Finding a congruent hierarchy of values naturally occurring amongst even a small group of those constructing software is incredibly unlikely, and leaving this unchecked will increasingly polarise the group.
It's surprising that software ever gets delivered. And a lot of it is not.
There doesn't seem to be a single one-size-fits-all way of developing software. The successes in the field (if you can get anyone to agree about what a success is, but, for example, Google, Atlassian, 37signals, Microsoft, and so forth) have very different approaches. Those that have tried to replicate those approaches have not found universal success; or even statistically better results.
Academic research into software methodologies do not seem to produce any consistent result, either. Some research lauds a particular technique, while other research may find that it does not seem to make a difference.
Given that there does not seem to be a "right" way of approaching software development, what should we do?
There seem to be two activities that are going to improve the output of a software development team.
The first is seeking to improve the software development endeavour. That means understanding what is going on, and trying to change things so that they work better. This means your processes, your tools, and your people. Understand them, and work together on improving things regularly.
The second is seeking a congruent set of development principles — a hierarchy of values, or "religion". "Change the people or change the people" is appropriate here. First, seek to find a congruent set of development principles that the majority of your existing group find acceptable. If you can't find that congruent set, or there are hold-outs, either add more people who share the set of principles, or remove the people who don't share the set of principles.