A Historical View to Organizing Software Development [old obsolete version]
When Alan Turing wrote his book “On Computable Numbers, with an Application to the Entscheidungsproblem” in 1936, he laid the grounds for computers science. The first “universal machines” were developed at Bletchley Park to decipher the Germans’ encryptions during the Second World War.
Between the 40’s and the 70’s computer science became more of a scientific instrument than a well established business technology. It wasn’t until IBM and Apple introduced the PC and Macintosh computers got widely spread outside the scientific institutions and the largest companies. It was now software development really started to take off. A lot of software development companies that are large today where established.
Up until the 70s, programs often were quite simple and were operated only by those how created them. But as systems became larger, it also became more difficult to develop and organize software development.
In 1970, a director at the Lockheed Software Technology Center, Dr. Winton W. Royce, published a paper entitled “Managing the Development of Large Software Systems: Concepts and Techniques“. Dr. Royce presented a method of organizing software development in a more structured way. This technique was inspired by the manner in which engineering field a like civil engineering and manufacturing organized their development.
The basic idea is that everything is done in sequential phases. This means that every phase of a project must be completed before the next phase can begin. First, all the requirements are documented, then everything is designed based on the so called “big design up front” and it continued throughout the stages.
Each phase was handled by specialized groups like business analysts (for defining the requirements), system analysts (for designing the programs), programmers (for developing applications), testers (for testing applications) and deployment personnel (for overseeing operations). These groups communicated mostly in writing, and handed over work from group to group.
Managing software development with the waterfall model is to investigate what the system is supposed to do, make plans so that it does what it is supposed to do, and to stick to that plan. This model had its setbacks: first, people learned a lot from the first system requirements until they went into production and were used by users. This made it difficult to take advantage of what was learned in the various processes.
Another setback was it often took a long time between the requirement phase and the user feedback phase. . If you didn’t figure out what the users wanted or the users themselves didn’t know what they wanted, that meant more time and money had to be spent to change or adapt the system to users’ needs.
In defense of Royce, it would be fair to say that he actually did warn that these things could happen and therefore proposed an iterative way of work. But no one adopted this part of his model. That’s how it came to be called the Waterfall.
When the US Department of Defense needed a software development process, they looked at Royce’s paper and they adopted a part of it (unfortunately they adopted the worst part) and named it DOD-STD-2167 (Department of Defense Standard 2167).
When the NATO later needed a model they thought that if it was the best model the US military could find, then it ought to be adopted. And from there, more and more people adopted the theories of the waterfall. Even if the US Department of Defense changed the standard in 1995, it remained the basis of what the academic world is teaching to this day.
During the 90s, new methodologies were developed as a reaction to the plan-driven waterfall model. Plan-driven methodologies reflect either the waterfall model or the iterative model, like the RUP (Rational Unified Process).
These methodologies, however, had something in common: they focus on a construction and planning plan in the beginning phase and stay with that plan. Plan-driven methodologies were considered bureaucratic, slow, demanding, and inconsistent with the way software developers actually perform effective work.
Many people were very frustrated on the demands presented that developers became more agile to business demands, adapting better to knowledge gained during the project.
New methodologies like XP, DSDM and Scrum conquered the world of software development in the 21st century. Many of the agile methodologies were inspired by “New Product Development Game”, an article written by Hirotaka Takeuchi and Ikujiro Nonaka and published in the Harvard Business Review in 1986. This article is often used as a reference and could be considered the birth of agile methodologies.
To summarize the history of organizing software development, I will use the words of agile guru Martin Fowlers: “from nothing, to monumental, to agile”.