Archive for the ‘Chapter 2: Software Development Organization’ Category

Psychology and System Integration

Comments Off on Psychology and System Integration

The more value a system has for a company the more integration with that system will be performed. When companies grow and processes change due to internal or external changes, more integration is carried out. Integration is a field that developed significantly in the last 10 years. Tasks are more transparent and communication is easier. Technology/Design principals like SOA (Service Oriented Architecture) have been the major accelerators in getting a coherent view through the industry. Here are some recommendations for handling integration projects.

What is Integration?

Integration is making two or more systems “talk” to each other. The interesting thing about integration is that it isn’t two different systems that understand each other, but two different teams that try to agree about how they see the world. Both teams already have a model of the world in their information model.

The Psychology of Who is Integrating Against Who

When integrating two systems, it’s common for team A to feel that they are doing team B a favor because team B is integrating into the team A’s system, and therefore have more to gain from that integration. Team A feels that their information model is the one that team B should adapt to.

Sometimes both teams get the same feeling that they are doing the other team a favor.  This causes irritations as each team tries to present evidence that their view of the world is the correct one.

Is it a problem to adapt to the other systems view of “the world” / information model?

If the other team has had more integration experience (either by building them or integrating to several systems), it is probably a good idea to let their view of the world teach you. While your system might be easier to adapt to and your team is more agile than theirs, the other team has to make sure the information model works for a lot of different systems.

How to avoid this?

If this becomes a problem for both systems, meet with the other systems owner and do an effect analysis together. Which benefit will each system gain through integration?  Are there any cost benefits or new possibilities? If there are new possibilities for any system, include the people who will gain most by these possibilities.

A business developer might see how and if these possibilities might be “harvested”. If there are no new benefits by integrating, no efficiency gain, no cost reduction, or any new possibilities that can be harvested, integration is discouraged. Another alternative is to get upper management / executive support, either enterprise-wide or between the organizations. Present the new possibilities that integration will bring. It’s always good to have “executive blessing”!

How to use this to benefit your team!

If you fear that your project will be the one that has to adapt to the other team, you can manipulate the situation to your advantage. But do it with caution, because these aren’t exactly “kosher” recommendations.

  • Conduct the meeting with the other team in your office or conference room. The other team is in your territory and you’re hosting the meeting.  The meeting host can set the agenda for discussions.
  • Be proactive and send out the agenda before the meeting.  That way, it’s easier to chair the meeting. If you are conducting the meeting, the other team will feel that they report to you.
  • Take down minutes of the meeting and send them to all meeting attendees.
  • If the meeting is via phone or video conferencing, (live meetings) use your account and pay for the facility.

Hidden Stakeholders

Integration projects tend to have hidden stakeholders. Since there are two organizations to be integrated, one has no knowledge of the other. all stakeholders. For the success of the entire project,  it should be made clear that every stakeholder is involved at the beginning of the project. Unfortunately, business people show up only during the testing phase when they should have been present much earlier. In large organizations it also common that there is a specific security department, it becomes very strange when they are not involved early. I has responsible for delivering as system as a supplier to and a week before product delivery the purchaser called and ask if we could have a workshop with the security department and discuss the design and solutions. The purchasor had the prior day accepted all acceptance test so we feelt like this was a bit late in the process to start discussing security design principles might have been better to done this earlier in the project.

The most important success factor in integration projects

The largest factor is relationships – accounting for 53% of success (According to the standish rapport).  For integration projects this becomes more important especially when it is carried out between different organizations, each with their goals and views of the world. Communication and building relationships must therefore be done at every level as well as obtaining upper management’s support. Hold regular meetings – for example every Friday after lunch. Things tend to happen on Friday morning when people promise deliveries or need to prepare.


Solutions are tightly connected to the organization – Conway’s Law

1 Comment »

How to separate different aspects of a project in which the modules are tightly bound together can be a real challenge. If many people are involved in solving a complex project, it can be hard to decide how closely together everyone should work. Shall we have collective ownership or a hard pre-defined contract between modules and different aspects?

 

Mel Conway wrote in his book “How Do Committees Invent?” that “organizations which design systems are constrained to produce designs which are copies of the communication structure of these organizations”. When people are dependent on people in other parts of the organization, a conflict of interest can easily appear. People might share the same view today, but in the future, someone’s boss might set different priorities. When that happens, one team will follow its own objectives and might not prioritize another group’s needs as highly as their own. The further away people are from each other, the more disjointed and incongruous their objectives are likely to be.

 

People working in the same room can easily discuss things spontaneously within the room, or on a short notice attend a design meeting within the room. People on different floors start booking meetings with each other, either officially or in more informal settings such as over a cup of coffee.   People on different offices write memos and specifications to each other have phone meetings and occasionally meet each other in real life. When people are geographically separated from each other, have different underlying motivations or are in a different part of an organization, clearer contract of responsibility and modularization is needed.


Why is software development different from other engineering disciplines’?

Comments Off on Why is software development different from other engineering disciplines’?

Software development is to solve a problem or fulfill a demand using source code for computers.

 

All types of work are based on solving a problem or meeting a need that is more valuable than the money spent for getting this need satisfied. This doesn’t sound that strange, does it? The problem with organization of software development is that it isn’t like any other work, so it is hard to draw conclusions from other industries.

 

I will discuss why software development is unique and why it calls for another kind of organization. There are a lot of factors that make software development a unique field.  This is why it has to develop its own traditions. It is possible to be inspired from other environments like traditional engineering for some aspects; however, when it comes to what we do as a community, our work is no doubt unique.

  • Very few or no physical laws limit us. Software development does not have many limitations that other endeavors have. When constructing and building a bridge, you have to respect the laws of physics. When there aren’t any limitations, a universal theory can’t be developed. The biggest limitation is creativity. When everything is possible, the hardest thing to figure out is how to solve or fulfill a demand in a way that no one else has done.
  • Software development is about design. The final goal in other engineering disciplines is some type of documentation. When a design effort I complete, the design documentation is turn over to the manufacturing team. This is a completely different group with completely different skills from the design team. If the design documents truly represent a complete design, the manufacturing team can proceed to build the product. But in software development the only true documentation that can reproduce the product is the source code, so software development is just about design. (Every thought about compiling source code is called “Making a build”!)
  • Design is about handling complexity. Design is about trying to cope with many variables and making trade-off in order to find a solution that suite all expectations. Software specifications tend to be fluid, and change rapidly and often, usually while the design process is still going on. Software development teams also tend to be fluid, likewise often changing in the middle of the design process. In many ways, software bears more resemblance to complex social or organic systems than to hardware. Few things are so hard to handle then when only your creativity sets the boarders, and then you have infinite parameters.
  • Developing software is cheap. Software development is fairly cheap and marketing a software product is much cheaper compared to other undertakings.  For example, a bug-fix is easy to distribute when you compare it to fixing a construction error of a car engine. This inevitably leads to a lot of different behaviors in our industry, but the most important is that it leads to speeding up the work. Quality is seldom more important than speed. You have to have features in your software product before your competitors create those features and before technology changes (again).
  • Software is easy to reproduce. Because it is cheap, developing it also means that reproducing it is quite easy.  Not many industries can easily reproduce a product.
  • Quick evolution. Every aspect of this industry evolves very quickly. Because it is cheap to develop new ideas and to spread these ideas to others, a much larger community can contribute to the evolution. Everyone wants to gain a competitive advantage so everyone adapts very quickly. New ways of working are constantly being developed.
  • Industry is highly competitive. This is not new in this industry. When combined t with the speed of evolution and the low costs to develop unique features that users appreciate, new features are soon developed by your competitors. They could also add better features.
  • Highly intensive degree of knowledge and information. Because everything in the industry evolves so fast, old knowledge is not valued. Knowledge that was highly valued 15 years ago is now considered as something belonging to the Stone Age.
  • Customers don’t know what they want. Everyone learns new things constantly including programmers and customers who use programs.  When customers learn during the process, they make new demands (like requesting special features).
  • Complexity leads to collaboration. Because information and knowledge evolve so fast, nearly no one is capable of managing the journey from an idea to a successful product to satisfied users.

 

This leads us now to the question, what drives change in software development organization?

Market forces lead to intense competition.  People therefore are always looking to improve and gain advantage. In the past, a company positioned itself along a single dimension.  Today, companies that deliver better products that are faster and cheaper have gained the competitive edge. Companies consistently look for ways to cut the time it takes to market their products, and at the same time look for ways to cut costs in developing new products with similar or better features.

 

If you’re looking to gain speed, cutting costs and raising quality, you have to change. In the 1998 book “Competing on Internet Time: Lessons from Netscape and its Battle with Microsoft”, MIT Sloan professors Michael A. Cusumano and David B. Yoffie, wrote,  “The pressure to release new software products faster and faster has grown over the years… ten years ago, development cycle of 24-36 months were typical, today, a year to 18 months development cycle is normal for software products … Within ‘emerging fields such as electronic commerce and web portal sites competing on the Internet, time demands significant product and feature changes every three to six months’”.

In 1998 this was probably a futuristic statement.  If you asked people then, they probably wouldn’t have imagined a lifecycle from idea to market of up to 18 months. Young developers would think of this statement as coming from the times of the middle ages and the Black Death.

 


Predicting Complexity Leads to Guilt and Guilt Lead to Apathy

2 Comments »

Predicting Complexity Leads to Guilt

When you work as a project manager you are expected to be in control. To control and review things, you must be able to predict things. To be able to predict things you need a plan.

The most common planning and control tools are GANT and PERT diagrams. But for a PERT diagram to work, it needs to be accurate and must rest on the assumption that it’s predictable.

How is this possible in a complex and complicated world? These types of tools give the project manager a sense of control. With this tool the project manager can control the holy trinity of scope, cost and schedule. Every real project manager has full control of these parameters; otherwise, he would be a fake.

But what happens when the project manager starts to experience the noise? A feeling of being out of control takes over. The second effect is that he feels guilty and asks, “With the information and resources I had, did I do my best? I’m not in control and I should be. How and why did this project fail? Is there any way to save my reputation when this project fails?”


When doubt and guilt get hold of a project manager, he does not focus on making the best software; instead he focuses on regaining control. No process or methodology is a crystal ball where in you can look into the future, only because he made an initial plan.

It may sound like I dislike GANT and PERT diagrams. Not at all! They can be very useful, but if they are your sole foundations and tools for organizing and controlling your project, I’d say you could develop an ulcer!

Guilt may Lead to Apathy

When a project manager’s work is based on a reliable process that produces predictable results and the manager feels that he did everything by the book (which must be right), he starts to wonder if the team followed the process.

Maybe the developers didn’t follow the process or aren’t doing what they are supposes to.  This can lead guilt, doubt and blame to spread within the organization. When people feel they are trying hard and doing their best and still fail to live up to their own expectations, they will eventually stop trying to do their best.

If I am confronted with failure even when I do my best, it’s best to just mind my own business. Why care about things that I haven’t been assigned to work on, and get blamed for things failing?

When this happens, things could be overlooked. They fall through and apathy takes over. To control this situation, the project manager has micro manage everything. The traditionally defined software development process is broken. It’s not a predictable tool that leads to repeatable results.

Let’s stop fooling ourselves, software development isn’t civil engineering. We don’t have 10% construction that leads to a 90% predictable production. When projects are complex and noisy, no process can be defined in advance to obtain a repeatable result. The solutions must be an adaptive and empirical process control.

Predicting Complexity Leads to Guilt

When you work as a project manager you are expected to be in control. To control and review things, you must be able to predict things. To be able to predict things you need a plan.

The most common planning and control tools are GANT and PERT diagrams. But for a PERT diagram to work, it needs to be accurate and must rest on the assumption that it’s predictable.

How is this possible in a complex and complicated world? These types of tools give the project manager a sense of control. With this tool the project manager can control the holy trinity of scope, cost and schedule. Every real project manager has full control of these parameters; otherwise, he would be a fake.

But what happens when the project manager starts to experience the noise? A feeling of being out of control takes over. The second effect is that he feels guilty and asks, “With the information and resources I had, did I do my best? I’m not in control and I should be. How and why did this project fail? Is there any way to save my reputation when this project fails?”

When doubt and guilt get hold of a project manager, he does not focus on making the best software; instead he focuses on regaining control. No process or methodology is a crystal ball where in you can look into the future, only because he made an initial plan.

It may sound like I dislike GANT and PERT diagrams. Not at all! They can be very useful, but if they are your sole foundations and tools for organizing and controlling your project, I’d say you could develop an ulcer!

Guilt may Lead to Apathy

When a project manager’s work is based on a reliable process that produces predictable results and the manager feels that he did everything by the book (which must be right), he starts to wonder if the team followed the process.

Maybe the developers didn’t follow the process or aren’t doing what they are supposes to.  This can lead guilt, doubt and blame to spread within the organization. When people feel they are trying hard and doing their best and still fail to live up to their own expectations, they will eventually stop trying to do their best.

If I am confronted with failure even when I do my best, it’s best to just mind my own business. Why care about things that I haven’t been assigned to work on, and get blamed for things failing?

When this happens, things could be overlooked. They fall through and apathy takes over. To control this situation, the project manager has micro manage everything. The traditionally defined software development process is broken. It’s not a predictable tool that leads to repeatable results.

Let’s stop fooling ourselves, software development isn’t civil engineering. We don’t have 10% construction that leads to a 90% predictable production. When projects are complex and noisy, no process can be defined in advance to obtain a repeatable result. The solutions must be an adaptive and empirical process control.


A Historical View to Organizing Software Development [old obsolete version]

1 Comment »

THERE IS A NEW VERSION OF THIS POST ON HERE!

When Alan Turing wrote his book On Computable Numbers, with an Application to the Entscheidungsproblem in 1936, he laid the grounds for computers science. The first “universal machines” were developed at Bletchley Park to decipher the Germans’ encryptions during the Second World War.

Between the 40’s and the 70’s computer science became more of a scientific instrument than a well established business technology.  It wasn’t until IBM and Apple introduced the PC and Macintosh computers got widely spread outside the scientific institutions and the largest companies. It was now software development really started to take off. A lot of software development companies that are large today where established.

Up until the 70s, programs often were quite simple and were operated only by those how created them. But as systems became larger, it also became more difficult to develop and organize software development.

In 1970, a director at the Lockheed Software Technology Center, Dr. Winton W. Royce, published a paper entitled “Managing the Development of Large Software Systems: Concepts and Techniques“.  Dr.  Royce presented a method of organizing software development in a more structured way. This technique was inspired by the manner in which engineering field a like civil engineering and manufacturing organized their development.

The basic idea is that everything is done in sequential phases. This means that every phase of a project must be completed before the next phase can begin. First, all the requirements are documented, then everything is designed based on the so called “big design up front” and it continued throughout the stages.

Each phase was handled by specialized groups like business analysts (for defining the requirements), system analysts (for designing the programs), programmers (for developing applications), testers (for testing applications) and deployment personnel (for overseeing operations). These groups communicated mostly in writing, and handed over work from group to group.

Managing software development with the waterfall model is to investigate what the system is supposed to do, make plans so that it does what it is supposed to do, and to stick to that plan. This model had its setbacks: first, people learned a lot from the first system requirements until they went into production and were used by users. This made it difficult to take advantage of what was learned in the various processes.

Another setback was it often took a long time between the requirement phase and the user feedback phase. . If you didn’t figure out what the users wanted or the users themselves didn’t know what they wanted, that meant more time and money had to be spent to change or adapt the system to users’ needs.

In defense of Royce, it would be fair to say that he actually did warn that these things could happen and therefore proposed an iterative way of work. But no one adopted this part of his model. That’s how it came to be called the Waterfall.

When the US Department of Defense needed a software development process, they looked at Royce’s paper and they adopted a part of it (unfortunately they adopted the worst part) and named it DOD-STD-2167 (Department of Defense Standard 2167).

When the NATO later needed a model they thought that if it was the best model the US military could find, then it ought to be adopted. And from there, more and more people adopted the theories of the waterfall. Even if the US Department of Defense changed the standard in 1995, it remained the basis of what the academic world is teaching to this day.

During the 90s, new methodologies were developed as a reaction to the plan-driven waterfall model. Plan-driven methodologies reflect either the waterfall model or the iterative model, like the RUP (Rational Unified Process).

These methodologies, however, had something in common: they focus on a construction and planning plan in the beginning phase and stay with that plan. Plan-driven methodologies were considered bureaucratic, slow, demanding, and inconsistent with the way software developers actually perform effective work.

Many people were very frustrated on the demands presented that developers became more agile to business demands, adapting better to knowledge gained during the project.

New methodologies like XP, DSDM and Scrum conquered the world of software development in the 21st century. Many of the agile methodologies were inspired by New Product Development Game, an article written by Hirotaka Takeuchi and Ikujiro Nonaka and published in the Harvard Business Review in 1986. This article is often used as a reference and could be considered the birth of agile methodologies.

To summarize the history of organizing software development, I will use the words of agile guru Martin Fowlers:  “from nothing, to monumental, to agile”.