One of the common objections to using Agile approaches is, "What if our project is too big?" Indeed, Agile almost invariably equals small teams. However, our experience of the past 2.5 years shows that Agile methods such as XP and Scrum can be successfully applied to large scale projects. In fact, Scrum is the first process with well-documented linear scalability. When you double team size in a well-implemented Scrum, you can double software output, even when the teams are distributed and outsourced.[1][2] In contrast with historical experience large projects can be just as productive as small projects for the first time.[3]
This article discusses some of the highlights of a distributed Scrum project by SirsiDynix (Provo, UT) and Exigen Services (Boston, MA and St. Petersburg, Russia). The project focused on the new implementation of platform and system architecture for a complex Integrated Library System (ILS) called Horizon 8.0, which is best compared to a vertical market ERP system with a public portal interface used by more than 200 million people.[1]
Best current Scrum practice is for local Scrum teams at all sites to synchronize once a day via a Scrum of Scrums meeting. In this case, we describe something rarely seen on large, distributed teams. At SirsiDynix, all Scrum teams consist of developers distributed across different sites. Any team member from any site can work on any team task. While some Agile companies operate in this geographically transparent manner on a small scale, SirsiDynix and Exigen Services have successfully implemented fully integrated Scrum teams with 56 developers in the U.S., Canada, and Russia.
New best practices for distributed Scrum seen on this project include:
While similar practices have been implemented on small distributed Scrum teams, this is the first documented project that demonstrates Scrum hyperproductivity for large distributed/outsourced teams building complex enterprise systems.
Let's look at the project in more detail and see how some of the obvious challenges were addressed.
SirsiDynix is a global leader in technology solutions for libraries. SirsiDynix has approximately 4,000 library and consortia clients, serving more than 200 million people through more than 20,000 library outlets around the world.
SirsiDynix was confronted with the requirement to completely re-implement its flagship product, a legacy library system with over 12,500 installed sites across the globe. The large number of developers required over many years in the midst of a changing business environment threatened to obsolete many feature requirements in the middle of the project. To complicate matters further, the library software industry was in a consolidating phase. Dynix started the project in 2002 and merged with Sirsi in 2005 to form SirsiDynix.
Fortunately, Dynix started the project with a scalable Agile process that could adapt to changing requirements throughout the project. Building a large product and facing time-to-market pressure, Dynix needed to quickly double or triple its productivity to exploit a business opportunity. The local talent pool was not sufficient to expand team size, and salary costs exceeded the constrained R&D budget. The project could only succeed by augmenting Dynix's resources with outsourced teams.
On the other hand, outsourcing is only a solution if Agile practices are enhanced by capabilities of the outsourced teams. The primary driver was enhanced technical capability resulting in dramatically improved throughput of new application functionality. Cost savings were a secondary driver.
Jack Blount, President and CEO of Dynix and later CTO of the merged SirsiDynix company, selected Exigen Services because of its history of successful XP implementations and its experience with systems level software. The combination of high risk, large scale, changing market requirements; merger and acquisition business factors; and the SirsiDynix experience with Scrum combined with Exigen Services success with XP led the team to choose an Integrated Scrums implementation. Jack Blount's past experience with Agile development projects at US Data Authority, TeleComputing and JD Edwards where he had used Isolated Scrums and Distributed Scrum of Scrums models was a key factor in his decision to structure the project as Integrated Scrums.
The project uses a three-tier architecture and uses Hibernate as a database abstraction layer. Oracle 10g, MS SQL, and IBM DB2 support is provided. The JBoss 4 Application server is used with a Java GUI Client with WebStart bootstrap. It is a cross-platform product supporting MS Windows 2000, XP, 2003, Red Hat Linux, and Sun Solaris. Built-in multi-language support has on-the-fly resource editing for ease of localization. Other key technologies are JAAS, LDAP, SSL, Velocity, Xdoclet, JAXB, JUnit, and Jython.
The project uses the Jira issue tracking and project management software. This gives everyone on the project a real-time view into the state of Sprints. It also provides comprehensive management reporting tools. Figure 1 shows the Sprint burn-down chart and a snapshot of Earned Business Value on the project along with a synopsis of bug status.
Figure 1: SirsiDynix Horizon 8.0 Project Dashboard in Jira
One of the obvious obstacles to keeping everyone on the project synchronized is the 10-hour time difference between St. Petersburg and Utah. To create an overlap of three to four hours, the Russian team adjusts its schedule and comes in at 12:00 p.m., while the US team starts its work between 7:00 and 7:30 a.m. local time.
Teams meet across geographies at 7:45am Utah time which is 17:45 St. Petersburg time. Teams have found it necessary to answer the three Scrum questions in writing and distribute the answers by email before the Scrum meeting. This shortens the time needed for teleconference on the joint meeting and helps overcome any language barriers. Each individual reports on what he did since the last meeting, what he intends to do next, and what impediments are blocking his progress. The shifted office hours allow for enough overlap so that both sides of the project can stay online after the Scrum meeting to jointly complete some immediate follow-up items.
The team uses email exchange on the three questions before the daily Scrum teleconference throughout the project to enable phone meetings to proceed more smoothly and efficiently. These daily team calls help the people in Russia and the U.S. to learn to understand each other.
For greater focus and productivity, the Russian team starts its day with a local Scrum meeting. Everyone is using the same process and technologies and daily meetings coordinate activities within the teams.
The duration of the Sprints was chosen to be two weeks: this was as long as SirsiDynix felt it could go without synchronizing and getting the complete picture of what is happening in the project. The team holds a Sprint planning meeting that is the same as an XP release planning meeting in which requirements from User Stories are broken down into development tasks.
The lag time for a Utah Product Owner response to questions on User Stories forces multitasking in St. Petersburg and this is not an ideal situation. Sometimes new tasks are discovered after querying Product Owners during the Sprint with additional feature details.
Code is feature complete and the team provides a demo at the end of each Sprint. If it meets the Product Owner's functional requirement, it is considered done. It is not deliverable code, however, and SirsiDynix wants to strengthen its definition of "done" to include all testing. Failure to do this allows work in progress to cross Sprint boundaries, introducing wait times and greater risk into the project.
SirsiDynix was able to achieve strong central control of teams distributed across geographies by centrally locating ScrumMasters, Product Owners, and Architects. ScrumMasters are all located in Provo, Utah or Waterloo, Canada, and meet in a Scrum of Scrums every Monday morning to coordinate work across teams. Architects are all located in Utah as well, and are directly allocated to production Scrum teams. The Architecture group also meets on Monday after the Scrum of Scrums meeting to control the direction of the project architecture through the Scrum meetings. A Product Owner resident in Utah is assigned to each Scrum team. The Chief Product Owner meets regularly with all Product Owners to assure coordination of requirements. This setup enables SirsiDynix and Exigen Services to get consistent performance across all distributed teams.
At the start of the project, SirsiDynix brought a key senior domain expert to St. Petersburg for a week to train the Russian engineers in the object domain and the library workflow. The client also demoed parts of the product to the Russian team. After that, top product architects come from Utah to St. Petersburg for three weeks to coach the Russian team in how the product works. At several points during the project, key members of the St. Petersburg team travel to SirsiDynix offices for training.
Having people rotate between the US and St. Petersburg helps achieve two goals: to share knowledge and to create personal connections between counterparts on both sides of the Atlantic. Introducing people personally creates a friendlier atmosphere, and work related issues are resolved much faster.
Exigen Services is an XP company and tries to introduce XP practices into all its projects. Pair programming is done on more complicated pieces of functionality. Refactoring is a planned activity for some of the Sprints and is not done in every iteration as in XP. As you can see from Figure 2, some radical refactoring has occurred as the project approached completion of its critical stage, without loss of functionality. Another XP practice, continuous integration, is implemented through hourly builds from the central code repository.
Figure 2: SirsiDynix and Exigen Services: Lines of new Java code in thousands (2003-2006)
his project is still active today, and there is always room for improvement. In fact, even as you are reading this, there are new approaches being tried by the project team to further improve quality and speed (e.g., some elements of test-driven development are being applied). However, both from an independent observer's point of view and from supplier's perspective, we want to offer you the following points for consideration:
Peter Vaihansky is Senior Director at Exigen Services Exigen Services. With headquarters in Boston, MA and development centers in Eastern Europe, Exigen Services provides software development outsourcing management expertise and services to global US and European companies including: Computer Sciences Corporation (CSC), T-Mobile, IBM, Fellowes, Contex, ScriptLogic, WebSideStory, and others. Exigen Services specializes in applying Agile methodologies (XP and Scrum) to distributed, offshore software development projects.
Jeff Sutherland, Ph.D., is CTO of PatientKeeper, an MIT-based startup providing mobile/wireless applications to clinicians. He has been CTO or VP of Engineering in nine software technology companies and introduced improved Agile development processes to each of them. His work on reusable business object components through the Object Management Group and the OOPSLA Business Object Workshop during the last decade has led to new database products, software development environments, CASE/OOAD tools, as well as vertical applications in multiple industries. As an inventor of the Scrum development process, his experience in organizational development has repeatedly enabled high-octane development teams to deliver world-class software products. Learn more about Jeff on his website: http://www.jeffsutherland.com/.
Anton Victorov works as Division Manager at Exigen Services. Anton has been with the company since 2000. He is the Exigen’s first Certified Scrum Master and one of the most outspoken enthusiasts of XP and Scrum. Anton is a frequent speaker at professional conferences, where he delivers presentations on the topics of Agile software development.
[1] J. Sutherland, A. Viktorov, J. Blount, and N. Puntikov, "Distributed Scrum: Agile Project Management with Outsourced Development Teams," in HICSS'40, Hawaii International Conference on Software Systems, Big Island, Hawaii. IEEE, 2007.
[2] Jones, C., 2000, Software assessments, benchmarks, and best practices / Capers Jones, Addison Wesley (Boston, Mass.).