Continuous integration is a way to applying quality control by integrates pieces by pieces of effort into project. When embarking a change, the developer takes a copy of the current code base on which he works on. As other developers submit changed code to the code repository, this copy gradually cease to reflect the repository code. When the developer submits his code to the repository they must first update their code to reflect the changes in the repository since they took their copy. If a lot of changes have happen to the repository, the more work the developer must do before submitting their own changes. If the repository becomes so different from the copy the developer is working on it becomes what is called as “integration hell”, where the time it takes to integrate exceeds the time it took to make their original changes. In a worst-cast scenario, the developer may have to discard their changes and redo the work. In order to avoid “integration hell” the practice of continuous integration is used. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible.
One warning when adapting CI!
The focus of integration and automated builds leads to a false feeling of secure, as long as it builds and all tests passes people tend to be satisfied. But in order to gain all benefits that you strive for, like getting feedback on just work done these test isn’t enough. In order to be as productive as possible you need to really test as close to the development as possible. You will save a lot of time if you find a bug or mistake that you done today or this week, rather something someone may have done to the code base during the last 6 month.
Continuous integration is like one apple a day, it keeps the debug doctor away. But it doesn’t gain its full power if you also complements with frequent testing and verification of the system.
Another important concept in agile software development methodologies is technical debt. When you write code you don’t fully understand a problem until you are finished solving it. This means that your application isn’t designed and implemented the way you would have if you were to do it again. If you continue to add functionality and features to this application without aligning the code to what you have learned and understood about the domain, there could be disagreement. This disagreement between understanding of the domain and reality will lead to a decrease in the rate of development. Ward Cunningham called this technical debt; he did an economic analogy to software development to explain code quality.
When you write code, you make a monetary or time investment to understand and solve a problem. And time is money. You have limited resources at your disposal so you prioritize the use of these resources in solving the problem. To meet deadline, shortcuts can be taken. This means that the job can be completed more quickly, but in doing so, time had to be borrowed. Time, as stated, is money.
When you borrow money, you pay interest until the loan is paid in full. It’s the same in software development. If you don’t re-factor your application and align with newly discovered insights about the problem or fix previous shortcuts, you need to adjust other parts of the system. However, this adjustment is in vain because the other parts don’t align with how it should have been built based on current or newly acquired knowledge.
Every minute spent on the wrong code is time spent in vain. It lowers the speed of solving the real business issues. If you keep borrowing money and not paying it back,this eventually leads to your income going to interest payments, bringing down your purchasing power to zero. It’s the same as developing a program for a long period of time and only adding features without reorganizing them for a better understanding of how they work. This translates into very low productivity.
A friend told me that his latest consultant assignment involved a project with a very large technical debt. He was added to the project very late in the development cycle, and he felt like an archaeologist trying to clean up an artifact that would crumble unless he worked very slowly and cautiously. This project had been running for three years and it was now one year late.
When several programmers pointed out that the problem arose because the project was not aligned with the system and did not take into account the new knowledge and understanding of the domain. Management’s comment was there was no time for refactoring as there were only 10 months left according to their earned value analysis, so the decision was to add the last feature. My friend left after a year and the project was only 90% complete.
What happens knowledge is not synchronized with code?
Management indirectly state that they don’t appreciate quality in the long run. If it’s not appreciated in the long run, it might not be so important in the short run either. If you are not allowed to take pride in your craftsmanship you will not do your best, because it doesn’t matter. Mikael Feathers author/blogger says, “We need to see clean flexible code as an asset and count it as an asset.” Whether it depends on deadlines, dispersed teams, or unfamiliarity with a new technology, it’s easier to let code rot than keep it up to date.
I found a great video with Ward Cunningham when he explains Technical Debt that I have to share with you:
Collective ownership encourages everyone to contribute new ideas to all segments of the project. Any developer can change any line of code to add functionality, fix bugs, improve designs or re-factor. No one person becomes a bottle neck for changes. People what are running their own raise becomes dangerous for the rest of the team. If you hear someone says “That module I dare not touch its Lotta’s code, it’s to complex. Only Lotta can do that!” you got a problem. Your “bus-factor” (How many people that need to be run over by the bus in order to need to redo that knowledge) is 1, if Lotta is run over by the buss all knowledge of that is gone. Collective ownership is quite closed linked to the practice of unit testing and continuous integration. In order for people to dare do change and not have 100% overview of how other modules might be affected on an accidental way a lot of automated unit test helps the developer to handle parameters that he don’t know by heart. And in order for everyone to be able to work and overview the project frequent check-ins shall be done. If mastodon check-ins is done, people don’t learn as much from each other’s code and it’s has a tendency of develop into that one single person is responsible for a specific class.
To be agile and stay agile, you need to change things with confidence without risking change that could result in an unpredictable behavior or in a bug. To do this, verify that previously built features are not affected by newly introduced code. If you don’t create automated tests causing you to do a lot of manual work, you incur unnecessary risks when introducing new features. As a result, your system will become very fragile.
Another argument in favor of several automated tests is that unit testing and test driven development represents the cost of handling bugs. The cost of fixing a bug that has already reached the production environment can be very expensive. Ideally, a bug should never appear.
Practices like unit testing, automated tests, continuous integration, refactoring, TDD, iterations and incremental development and simple design where separation of concerns are applied is great insurance to be able to be agile without becoming fragile. As soon as you start tamper with these practices your risks increase and ability to flexibility decrease.
The Agile Manifest says that people is more important than a process. But what if you don’t have a great team that are smart, disciplined and attentive. If you projects consists of three trolls, a senior fresh out of school consultant from Accidenture and a relatively bright project manager, who happens to be deaf, blind and mute. No amount of coaching can help this team to magically self-organize and deliver a successful product. Then rules, micromanaging and a step-by-step process might be the only way. Smart, disciplined and attentive people simply do their jobs well and don’t get ticket for dangerous behavior. And the agile methodologies build an assumption of that you always got the great team. But the world isn’t perfect, sorry all employees. So self-organizing might not work so good for all teams.
Pair programming is sharing knowledge not only about how the code works, but also why it looks that way. It is sometimes difficult to write explicit instructions on why code looks a specific way and how it’s organized in pair programming. A common question about pair programming is will pair programming slow us down? The answer is yes and no. If the purpose is to spread knowledge, I would reformulate the question to will pair programming slow our learning or speed it up? Another advantage of pair programming is that interruptions tend be minimized.
Sometimes a group’s efficiency is linked to the specific knowledge of one or more people. When an expert becomes a bottleneck, let that expert be the adviser. By spreading the knowledge and letting the team try it first without the expert, it might not be as efficient, but in the long run, the knowledge spreads.
If you are not allowed to make mistakes, you will not understand the whole picture. If the expert isn’t directly involved, the rest of the team has to take more responsibility but still must have access to the expert as an adviser. The expert being fully involved is probably the most efficient approach, but if it limits the team’s performance, it would pay off in the long run.
Agile work uses short cycles so that developed things get too used fast. All code that isn’t used is just an inventory cost and isn’t learned from. The sooner that valuable result can be delivered, the more value can be obtained from those results. This extra value is derived from opportunities such as earlier sales, competitive advantage, early feedback, and risk reduction. In order to learn as fast as possible about the real use and see how it works frequent deliver of the system is a core idea. There is an explicit trade off: the shorter the time to delivery, the smaller the piece of value will be. But, like investing in one’s retirement account, the earlier you start, even with small amounts of money, the better off you are in the long run.
The foundation of Agile methodologies is to work in iterations. This has many benefits:
Benefit # 1: Code that is not used by stakeholders is an inventory cost. If you had a store selling clothes in all models but are not visible to customers, they would never get sold. Their value declines because soon they will be out of fashion.
It’s the same with code. If it isn’t used, you never get real feedback or get the chance to test it. The world around the system constantly changes and soon you will be working on new problems if the code isn’t used. When you move on to new features, this unused code becomes an inventory cost.
A ratio that is used in financial analysis is the stock turnover ratio. Stock turnover ratio = annual sales / average inventory. The average stock is calculated by adding the inventory value at the beginning and end of the year and dividing it by 2. For software development, you could say that the turnover ratio would be “features shipped per year” / “average number of features that are waiting for testing or deployment or not being used by users”.
Benefit # 2: Continuous integration and frequent deliveries generates feedback. Small and fast iterations also mean that the code must constantly be integrated with previous work and frequently delivered to users. This means that the code is tested during integration and also tested by users after it has been developed. If something goes wrong it is better to know about it at once. If you investigate a bug which is the result of work done 6 months ago, the root cause analysis will take some time. But if you get feedback on work done today or recently, you can easily find the root cause.
Benefit # 3: The complexity factor does not become large with small iterations. If you estimate doing two things that are about the same size, you think of the work as a having a linear relationship; that is, if I do twice as much it will take twice as long. If the first thing takes 8 hours and the second one takes 8 hours, you can do it in two days.
But if you have to do both of them in one day with the help of one friend, can it be done in just 8 hours? The answer depends on the dependencies. If both works have dependencies, you need to make sure both parts interact as planned so they don’t interfere or overlap one another.
If you have one thing to work with, you’re dealing with only one thing.
If you have two things, you deal with them, but you also have to deal with the way they interact and interfere with each other.
If you have three things, you have those three things, but at the same time you have three different sets of interaction between pairs of the three, plus you have to deal with the way all three things are combined which leads to seven aspects.
If you have four things, you have 15 aspects, and so on.
Therefore, if you have 10 things that are related and dependent on each other, doing them might be 1000 times more complex (or to be exact, 1023 aspects to deal with). The number of dependencies multiplied by the number of things to do result in the following formula: (Aspects to handle = 2 x “the things” – 1). What do these things consist of? It could be number of users’ stories, number of integrated components, number of departments involved, number of people in a discussion.
Benefit # 4: Business value comes from the possibility of adapting to change. When you start a project, the decision to start it comes from the possibility of reaching an expected business value. When the first version of the system is released, not all of the features are perfectly adapted to users’ needs. The system, therefore, doesn’t reach the expected business value.
The longer the system is used, the more the business learns what it takes for the system to reach expectations; however, because the releases are infrequent, the system falls short of its value. After a few years, the system shall have reached its minimum accepted value.
When an Agile approach is used instead, the first release lacks a lot of expected features but because new versions are frequently released, the system doesn’t fall short of its value over time. Releases are delivered, because new knowledge is used to re-prioritize work. When the system is used for a period of time, expectations tend to rise.
Delivery of Business Value
The most important observation is what this does to the stock turnover ratio, since the delivered value of the system is based on the time the system is used. The delivered business value is the red area, because it creates value when it is used. So using a system early on shows that the value delivered is at least doubled when using the Agile approach.
When the minimum accepted value is reached, business people tend to think they need a whole new system; only this time the original requirement has changed. The priority now is to add features, and to change and adapt the system to new acquired knowledge, re-factoring and updating it with new technologies. In truth, what the business needs is not a completely new system with perfect requirement specifications. What the business needs is to continuously develop their system in new ways.
Code reviewing has two purposes. The first is to make sure that the code that is being produced has sufficient quality. Is it ready for the next step in the process, might be check-in, merged to main track of the source or deployment. Code reviews are very effective at finding errors of all types, identify a poor structure or identify that the source doesn’t meet the business proposal. It’s a effective litmus test for quality of the code.
The second purpose is to spread knowledge, both for the coder and the reviewer. Developers learn when and how to apply techniques to improve code quality, consistency and maintainability. Through thoughtfully evaluating code on a recurring basis, developers have the opportunity to learn different and potentially better ways of coding.
Code reviews can easily start off on the wrong foot, because they are perceived as an unnecessary step that has been forced upon the developers or seen as an evidence that managers doesn’t trust the developers. Code reviews are a proven way to minimize defects. Image source: http://zuskin.com/coding_habits.htm
Refactoring is about reorganization, it comes from mathematics when you factor an expression into an equivalence – the factors are cleaner way of expressing the same statement. Refactoring means equivalence; the original product and the end product must med functionally identical. There are generally two reasons to do refactoring:
Maintainability – it is easier to fix bugs because the source code is easy to read and the intent is easy to grasp. This might be achieved by removing monolithic routines into a set of individually concise, well-named, single-purpose methods. Or it might be achieved by moving a method to a more appropriate class, or by removing misleading comments.
Extensibility – it is easier to extend the capabilities of the application if it uses recognizable design patterns.
In order to ensure that the function is identical is common to create a solid set of unit tests. The test should demonstrate that the behavior of the module is correct. If the test fails, you undo the change. Refactoring shall make your code clearer, cleaner, simpler and more elegant.
What makes sports so engaging is that when someone understands the rules they immediately knows how good a specific throw was. How fun would bowling be if you didn’t have direct feedback on how many pins you have hit? If you instead got a progress report from your project manager once a month, with the conclusions of last month’s game.
But this is quite common in today’s knowledge based work people don’t know if today’s work effort was a complete failure or a success. Three month from know they might get a “bug report” that indicated that something this week wasn’t optimal. How easy is it to get better and realize what lead to success if there is no natural feedback on your work. Feedback is essential for engagement.
This makes the “monitor board”/”agile dashboard” with burndown chart to a great tool. Everyone can see what no one start working with, what someone is working on and how things are going relative to the plan. People tend to understand how things are going and get more engage that people who get reports or review of work they did three month ago.
Through detailed planning in the short term and follow up on a daily basis in a visual way everyone understand how the co-operative game goes. You know when you are in trouble and you know when you’ve made a strike. This creates job satisfaction and commitment.
Everyone are happy when they know they made a strike. You know it’s the best you can do you feel satisfied. It’s the same in work life people feel good about themself if they know they done a good result. As a project manager this is a great tool, because this creates transparency, people know when they need to start focusing on delivery without you being there looking over their shoulder. I use to end my presentations about Scrum with showing the following video, in order to show how powerful visualization and messurement is.
When you have delivered a system you know how that project was built. When a project precedes our ability to change, requirements decrease. At the same time, our understanding of what we are doing increases.
If we prioritize things that add knowledge at the beginning of the project and we minimize the cost of changing our minds, we could speed up the pace of understanding the requirements and therefore lower the risks of misunderstanding.
If we run our software project based on the waterfall model, we choose to increase the growth of our knowledge at the point of delivery. This is the “moment of truth” when we see if everything holds together and we are delivering the expectations of stakeholders. It could go well or it could not, consequently leading to the disappointment of stakeholders. The problem is that the stakeholders acquired new knowledge during the project, and now want the system to do or be something they had not thought of at the beginning.
However, if you frequently deliver software to stakeholders and have the ability to adapt to their feedback, you can speed up your understanding of requirements. When the product becomes visible for the stakeholders, you get acknowledgement on what you have understood. If stakeholders don’t give any feedback and everything seams fine, you have just lowered your risks, as opposed to showing everything at the “moment of truth”.
Another great advantage is that by delivering your work in small parts, stakeholders are not 100% dissatisfied because you gave them the opportunity to give their feedback on the small parts and you have adapted the requirements based on this feedback.
The inspiration for plan-driven methodologies came from other engineering disciplines such as civil and mechanical engineering. In civil engineering, it is less costly to change requirements during the design stage and it is more expensive to adapt to changes when construction has already started.
Therefore a lot of energy is put into the planning phase. When the drawings are specified and finalized, it is reasonably easy to predict the schedule and budget for the rest of the processes, because both the requirements and the technology are known. Once we have the construction plan, the construction is more predictable. The nature of these projects is to resist changes, because it costs too much to make them after construction has started.
Software development is different. There is no guarantee that a good design will make construction predictable. In civil engineering, the time spent on construction design is often less than 10% of the project’s total budget. In software development, it’s more common that over 50% of the total budget is spent on design and understanding requirements. When we adopt the civil engineer’s approach, anything that wasn´t planned causes a problem. We need to reach (B) as planned when the project plan (Time, Money and Scope) was approved.
Construction is less costly in software development. So why not adapt to change as we learn more during the process? Changes in the requirements during the process of developing a system leads to increased information about the system, and thus enabling better decisions because of this additional information. This is how competitive advantage is gained.
If we could use methodology that allows us to change, we will know more without incurring major costs so it would be a mistake not make the required changes. If our methodology allows this, the output (the system or product) is going to be worth more than the product we would have had have from the original design (C) >= (B).
A Difference View of Success
The plan-driven methodologies defined success as the delivery of agreed functions on time and on budget. But agile methodologies promote the idea that there is something nobler to strive for, because sometimes the customer remains dissatisfied even if he got everything he asked for! This is because he got (B) but at the end of the project, he now wants (C).
Customers also adapt but if they change their mind about what they want and you prefer not to make the requested changes, problems could arise.. Either you or the customer will be disappointed. Success for Agile methodologies depends on knowing more which makes it better for us to adapt to new goals. This gives the customer what he wants at the end of the project (C).
If customers don’t know what they want when the project ends, how can you design a system in the beginning when you don’t know what the understanding was? If your design isn’t good enough to predict the rest of the work, it won’t be such a good idea to design too much in the beginning of the project.
And if you cannot establish requirements you cannot make a good design and in turn, you cannot get a predictable plan. The Agile approach is to welcome change so if your assumption is proven wrong, you redefine the goal and set new priorities. At the end of the project journey (C) this new goal will be worth more than the first defined goal (B).
Because the customer and stakeholders are allowed to change as they learn more new things, the value of (C) becomes greater than (B). This of course builds on the idea that you manage to deliver (C) according to time and budget restrictions.
The difference between an incremental approach and an iterative approach is that the incremental approach does not require going back and changing things that are already delivered; the focus is only the requirements that have not yet been implemented. In the incremental plan, you don’t learn by the previous work, and the final product is not based on knowledge acquired during the journey. Here is a visual example of the difference between the incremental and iterative approach: