The Agile Renaissance draws on the knowledge of prior software development eras. Many important lessons in software development were lost in the late 80's and early 90's. But by the close of the millennium then Agile Renaissance had begun with a new focus on iterative development, customer centric requirements and planning, and automated test.

Agile development got its name in January 2001, during a blizzard in Snowbird Utah. A group of influential software developers met to discuss improving the state of software development and comparing experiences and opinions about software development. The term Agile Development was coined and the Agile Manifesto was crafted, it says:

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

Those simple statements have triggered an avalanche of controversy, and misunderstanding covering the entire software development landscape. That avalanche also has triggered people throughout our practice to look at how they work and this has led to many improvements.

Proven techniques brought back into practice and new discoveries were the turning point to this age of software renaissance.

There are many flavors of Agile Development. Probably the two most popular and best known are Extreme Programming, and Scrum.  They are two of the leading drivers of the Agile Renaissance.

Iterative Development

Agile development is based on iterative and incremental development (IID). IID provides regular feedback by breaking the project into iterations that are generally one to four weeks in length. The output of each iteration is working software. Each iteration is like a stand-alone project ending after the fixed time-box, and delivering some executable version of the product.

Iterative development has been around for decades. As far back as 1987 Fred Brooks' as chairman of the Defense Science Board Task Force on Military Software recommended that the waterfall process be replaced with iterative development due to waterfall's history of failure on large DoD contracts. Iterative development has been used successfully on some very high profile projects from the 1950's to present day including: the X-15 rocket plane, project mercury, trident missile submarine control systems, the space shuttle avionics, and the Canadian Automated Air Traffic Control System, to name a few.

The problem with plan driven development, rather than feedback driven planning, is that in the former the fact that new invention is happening is not recognized and acknowledged. You can't schedule inventions and discoveries. You can schedule trials and review milestones, but you can't schedule invention and a lot of software development is invention.

What's wrong with how plans are created and used?

 One of the problems with getting software done is that project planning is often based more on what is desired rather than what is possible. What is desired is very important, don't get us wrong. Dates drive business and dates are often set for very good reasons. For example there is a trade show where your product needs to be showcased, you know that the show must go on, it won't be rescheduled just because your product is late. Or your company may have a competitive threat where time to market is critical or maybe the revenue for the big order cannot be recognized until the product is shipped with software.

Too often the desired date and a few vague "requirements" are the only information provided when starting a project. Model 2 is being retired in 9 months, so we have to have model 3 ready in 8 months with these new features X, Y and Z to get the big sale with Acme. That is critical information, but we also need to know what is possible when planning a project.

Acording to Steve McConnell "Software projects contain too many variable to be able to set schedules with 100-percent accuracy. Far from having one particular date when a project would finish, for any given project there is a range of completion dates, of which some are more likely and some are less." Steve McConnell from Rapid Development.

We need a realistic planning mechanism that allows the business to make informed business decisions. The obscure black art of software planning and development has to be made more transparent, and measurable. The inherent uncertainty has to be recognized and dealt with.

What is Agile Planning?

In Agile planning and iterative development the project timeline is divided into a series of equal duration time boxes called iterations. Requirements are broken down into small pieces of functionality called stories. In a way stories are similar to use cases, but likely more narrow. Each of the small stories gets an relative effort estimate measured in story points. A SWAG is made to estimate the number of story points that can be completed in an iteration. The plan is simple a series of iterations that each contain a set of stories. Once a few iterations have been completed, the SWAG that was used to decide how many stories to include in an iteration is replaced with the actual measured number of points completed by the team. This is the teams velocity. It is used to project how much work will be completed in coming iterations.

The plan becomes more accurate as the team learns more about the project requirements, the solution design and implementation, as well as learning how to work together. The plan is also very flexible, with the content being negotiable for all the future iterations. As market needs and team knowledge change the plan can be adjusted as necessary.

How is work divided on a Agile Team?

There are two main roles in agile development, the customer role, and the developer role. On all but the smallest projects each role is usually represented by a team of people. The customer defines the product and the business goals and the developer builds the product. Customer and developer teams collaborate to deliver products. Each team has their own detailed practices.  To be the most successful using iterative development, the whole organizations should be involved.

Customer Practices

 The customer role is similar in the agile development practices of Extreme Programming and Scrum. Although in Scrum the customer practices are more thoroughly defined. On the other hand, in Extreme Programming the engineering practices are better defined.

The customer has the responsibility to steer the project to successful completion and is responsible for these kinds of things:

• Create a product Vision
• Maintain a product backlog of features
• Split features in to small pieces of work
• Choose the feature content of each iteration
• Define acceptance tests for each iteration's deliverables
• Negotiate delivery dates and content
• Work as a team with development

That is a lot of work! Keep in mind it is not just one person. The customer roles is led by the product owner, usually from product management. The product owner leads a skilled team that knows the product domain and has a vision of the product they expect to be successful in the market. This team is usually made up of marketing, systems engineering and test automation.  One of the customer practices key to the success of the product is creating a rich suite of automated tests.  Who would be better at knowing what to test than those expert in the market's needs.

Agile Engineering Practices

   The engineering team is responsible for working with the customer team to design and build the product. The engineering practices are tailored to support the incremental and iterative planning model. With the software constantly evolving, there is a significant emphasis on test automation, incremental feature delivery, and continuous software design improvement.

One of the most significant problems of teams adopting the agile management practices of Scrum if the lack of technical practices to support the iterative nature of agile development.

The engineering team has these responsibilities:

• Deliver tested software every iteration
• Develop automated unit and acceptance tests
• Evolve a clear architectural model for the software
• Keep the code clean, maintainable and extensible
• Provide and revise estimates
• Work as a team with the customer 

These responsibilities are well addressed by the technical practices of Extreme Programming. When applied together the engineering practices help to keep software internal and external quality high, and provide predictable schedule performance.

Extreme Programming

 Kent Beck, author of Extreme Programming Explained says, "XP is a light-weight methodology for small-to-medium-sized teams developing software in the face of vague or rapidly changing requirements." Simply stated, XP is a set of values, rights and best practices that support each other in incrementally developing software.

XP values Communication, Simplicity, Feedback and Courage.

The team has the right to do a quality job all the time, be honest with each other and to have a real life outside of work. Team members communicate with each other openly and honestly. Problems are solved simply, designs are kept simple. We get feedback by writing tests and showing the customer what was asked for. We need courage to be honest about what is possible, to improve a design when the pressure is on and to challenge the status quo.

XP is a collection of best practices. Some may sound familiar. Some may sound foreign.

Customer Team Member - Teams have someone (or a group of people) representing the interests of the customer. They decide what is in the product and what is not in the product. I'll refer to this group of people as the customer or the customer team in this paper. The customer is responsible for acceptance testing the product. This means the customer team needs skilled testers.

User Story - A User Story represents a feature of the system. The customer writes the story on a note card. Stories are small. The estimate to complete a story is limited to no greater than what one person could complete within a single iteration. If the story is too big to complete in an iteration, split it into smaller stories. If you are used to use cases, a story is like the name of a use case. The details of the use case are in the acceptance tests.

Planning Game - XP is an iterative development process. In the planning game, the customer and the programmers determine the scope of the next release. Programmers estimate the effort to complete each story. Customers select stories and package them into iterations. Stories are prioritized by the customer according to their business value and cost.

Small Releases - Programmers build the system in small releases. An iteration is typically two weeks. A release is a group of iterations that provide valuable features to the users of the system.

Acceptance Testing - The customer team writes acceptance tests. The tests demonstrate that the story is complete. Acceptance tests provide the details behind the story. Tests are defined prior to completing a story. The programmers and the customer automate acceptance tests. Programmers run the tests multiple times per day.

Open Workspace - To facilitate communications the team works in an open workspace with all the people and equipment easily accessible.

Test Driven Design - Programmers write software in very small verifiable steps. First, a small test is written, followed by just enough code to satisfy the test. Then another test is written, and so on.

Metaphor - The system metaphor provides an idea or a model for the system. It provides a context for naming things in the software, helping the software communicate to the programmers.

Simple Design - The design in XP is kept as simple as possible for the current set of implemented stories. Programmers don't build frameworks and infrastructure for the features that might be coming. Frameworks and infrastructure evolve as the application evolves.

Refactoring - As programmers add new features to the project, the design may start to get messy. If this continues, the design deteriorates. Refactoring is the process of keeping the design clean, incrementally. [FOWLER]

Continuous Integration - Programmers integrate and test the software many times a day. Big code branches and merges are avoided.

Pair Programming - Two programmers collaborate to solve one problem. Programming is not a spectator sport. Both programmers are engaged in the solution of the problem at hand.

Collective Ownership - The team owns the code. Programmer pairs modify any source code as needed. Extensive unit tests help protect the team from coding mistakes.

Coding Standards - The code needs to have a common style to facilitate communication between programmers. The team owns the code; the team owns the coding style. Individuals may have to give up their favorite conventions to conform to the team style. The standard is determined by the team.

Sustainable Pace - The team needs to stay fresh to effectively produce software. Creating a product is like running a marathon. We sprint at times during the marathon, but we are careful to reserve our strength so that we can finish the race.

Tweet