The DevOps Adoption Playbook: A Guide to Adopting DevOps in a Multi-Speed IT Enterprise
Published by
John Wiley & Sons, Inc.
10475 Crosspoint Boulevard
Indianapolis, IN 46256
www.wiley.com
Copyright © 2017 by John Wiley & Sons, Inc., Indianapolis, Indiana
Published simultaneously in Canada
ISBN: 978-1-119-30874-4
ISBN: 978-1-119-31052-5 (ebk)
ISBN: 978-1-119-31076-1 (ebk)
Manufactured in the United States of America
No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 646-8600. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley.com/go/permissions.
Limit of Liability/Disclaimer of Warranty: The publisher and the author make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation warranties of fitness for a particular purpose. No warranty may be created or extended by sales or promotional materials. The advice and strategies contained herein may not be suitable for every situation. This work is sold with the understanding that the publisher is not engaged in rendering legal, accounting, or other professional services. If professional assistance is required, the services of a competent professional person should be sought. Neither the publisher nor the author shall be liable for damages arising herefrom. The fact that an organization or website is referred to in this work as a citation and/or a potential source of further information does not mean that the author or the publisher endorses the information the organization or website may provide or recommendations it may make. Further, readers should be aware that Internet websites listed in this work may have changed or disappeared between when this work was written and when it is read.
For general information on our other products and services please contact our Customer Care Department within the United States at (877) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002.
Wiley publishes in a variety of print and electronic formats and by print-on-demand. Some material included with standard print versions of this book may not be included in e-books or in print-on-demand. If this book refers to media such as a CD or DVD that is not included in the version you purchased, you may download this material at http://booksupport.wiley.com. For more information about Wiley products, visit www.wiley.com.
Library of Congress Control Number: 2016962068
Trademarks: Wiley and the Wiley logo are trademarks or registered trademarks of John Wiley & Sons, Inc. and/or its affiliates, in the United States and other countries, and may not be used without written permission. IBM, the IBM Press logo, UrbanCode, uDeploy, System z, Rational, IBM Watson, WebSphere, Bluemix, InfoSphere, Optim, PureApplication, DB2, SoftLayer, and Blue Box are trademarks or registered trademarks of International Business Machines Corporation in the United States and/or other countries. A current list of IBM trademarks is available on the web at “copyright and trademark information” as www.ibm.com/legal/copytrade.shtml. All other trademarks are the property of their respective owners. John Wiley & Sons, Inc. is not associated with any product or vendor mentioned in this book.
To my wife Ritika, for always motivating me to do more, be more, and never be satisfied with the status quo. And my children, Saransh and Shreya, for being the ones I am motivated to do and be more for.
Sanjeev Sharma is an internationally known DevOps and cloud transformation thought leader, technology executive, and published author. Sanjeev’s industry experience includes tenures as CTO and Worldwide Technical Sales Leader, Acquisition Integration Technical Leader, and IT Architect. As an IBM Distinguished Engineer, Sanjeev is recognized at the highest levels of the exclusive core of technical leaders at IBM.
Sanjeev provides core leadership to drive the adoption of cutting-edge solutions, architectures, and strategies for DevOps and the cloud. His experience as the Global CTO for DevOps Technical Sales at IBM, combined with his deep insight and ability to understand both business and IT needs, drives a unique perspective for any business. This perspective allows Sanjeev to advise and mentor C-level and senior technical executives on achieving DevOps and cloud transformations, across industries and geographies.
Sanjeev is a frequent speaker on the international tech scene, as a cloud and DevOps expert. He regularly publishes articles, blog posts, and videos for leading tech publications and his own blog, at http://bit.ly/sdarchitect. Sanjeev tweets as @sd_architect.
Lee Reid has more than 30 years’ experience in software engineering, architecture, product development, technology innovation, and team leadership in both manufacturing and information technology domains. Lee is an engineering graduate of General Motors Institute (BME) and the University of Michigan (MSE) and holds four U.S. patents. He has recently transitioned into higher education, where he leads IT and is introducing Lean and DevOps practices at St. Norbert College.
Project Editor
Adaobi Obi Tulton
Technical Editor
Lee Reid
Production Editor
Rebecca Anderson
Copy Editor
Marylouise Wiack
Production Manager
Katie Wisor
Manager of Content Development & Assembly
Mary Beth Wakefield
Marketing Manager
Lorna Mein
Professional Technology & Strategy Director
Barry Pruett
Business Manager
Amy Knies
Executive Editor
Jim Minatel
Project Coordinator, Cover
Brent Savage
Proofreader
Kim Wimpsett
Indexer
J&J Indexing
Cover Designer
Wiley
Cover Image
©traffic_analyzer/Getty Images
This book is an effort to put to paper countless conversations and (sometimes heated) discussions and debates on DevOps and IT optimization and innovation that I have had with my customers, co-workers, and peers in the DevOps community. Through these conversations and discussions, dozens of people have contributed to this book, not to mention all those whose blogs, articles, books, webinars, videos, meetings, and presentations I learned from.
The key contributors include my fellow DevOps subject matter experts and technology thought leaders at IBM. These include (in alphabetical order by first name):
Some contributors who were formerly at IBM include the following:
Several key customers, business partners, and experts also contributed, as real-life examples of leaders who led DevOps transformations at their own companies and organizations. Their stories from the trenches are the best sources of lessons learned. In many cases, they were at the other end of the conversations that led to the lessons learned and practices documented in this book. Because I met most of these people in a professional capacity as an IBM employee, I cannot list them all here. I will list the few who also co-presented at conferences, meetings, and webinars with me, or co-authored articles or blogs with me. They include the following (along with their current employers):
I would be remiss to not acknowledge separately Gene Kim, the über-guru of DevOps, with his contributions through his books and his DevOps Enterprise Summit Conference. I personally had multiple opportunities to talk to him one-on-one, including a video interview I recorded in 2014 (https://youtube/6QK2Mt-KPo4).
I would also like to give a special thanks to Lee Reid. I have worked with Lee for more than a decade. He was also my “partner in crime,” leading the Worldwide DevOps architect team at IBM for two years. We developed the DevOps Value Stream Mapping workshop techniques together, and I personally bounced tons of ideas off of him. It was only fitting that I had the opportunity to leverage Lee’s talents and mind, despite his having left IBM for St. Norbert College, by having him be the technical editor of this book. There is no way the book would have made it to its final refined and well-structured form without Lee’s insights, critique, and feedback.
Lastly, I would like to thank the wonderful editing staff at Wiley: Adaobi Obi Tulton, whose skills certainly live up to her Jedi-sounding name, and Marylouise Wiack for her complete mastery of language and prose (and yes, punctuation—my nemesis). The book is light-years ahead of what I originally put to paper because of their hard work and painstaking correction of my meek attempt to put words together into coherent sentences.
Teams that excel do so not just because they have the best members, the best tools, the best training, the best processes, or the best leaders and coaches. They excel because they, as a team, have all of the above but also know what to do when they face various situations and challenges. They have a playbook of potential solutions (plays) for a variety of scenarios.
When faced with a unique situation or challenge, the players and coaches come together as a team to pick the right play from the playbook, and then, most importantly, they execute it. My alma mater, Villanova University, won the national championship when it came down to the final play, with seconds on the clock, because they had plays they had practiced before. They read the situation, picked the right play, and executed with precision to win. Had they not had the play that would catch North Carolina off guard, there may have been a different outcome.
In the same way, IT organizations need plays to execute. For day-to-day application delivery and operations, these so-called plays are captured in their development, delivery, and operational processes. IT organizations that succeed have good processes and execute them with excellence. However, transforming IT organizations is another story. Most organizations struggle with transformations, not having well-defined, winning plays that can overcome cultural and organizational inertia. This book captures a set of proven, repeatable plays for adopting DevOps at the enterprise scale and for transforming a large, complex, distributed IT organization to adopt DevOps.
These plays come from my experience over several years in the trenches as I helped dozens of organizations, of myriad sizes and maturity levels, in a variety of industries and geographical locations, to adopt DevOps. Since the early days of DevOps, as the Worldwide CTO of DevOps Technical Sales and Adoption at IBM, I have had a front-row seat to see the evolution and maturation of DevOps from a set of practices pioneered by startups to a cultural and technological transformation effort in large enterprises. I was a pioneer and thought leader for DevOps at IBM, and I became the face of DevOps to IBM’s clients. This book distills patterns of success that I have observed among hundreds of clients working, struggling, and succeeding at adopting DevOps at organization or enterprise-wide scale.
Adopting DevOps in a small, co-located organization, without a lot of cultural memory, is not difficult. Even in large organizations, small teams—the proverbial two-pizza1 teams—regularly succeed at attaining the business results promised by DevOps. In most organizations, you see many such efforts made, and most succeed. It is taking that success from an individual, isolated team level and scaling it to an enterprise that is a challenge. It is like having a series of small dance squads around the organization. However, these dance squads are all unique; some are doing the salsa, some jazz, some are ballroom dancing, while yet others are doing something my daughter tells me is “hip-hop.” They cannot combine and grow to a massive dance ensemble that can perform at the next halftime show, filling up the entire paying field of a football stadium. For that they need to be dancing not only to the same music but also be performing the same dance form, in unison. Similarly, small unique teams cannot impact the entire organization. These teams need to make the effort to standardize practices, processes, platforms, and tools in order to allow them to be replicated in other parts of the organization.
The organization, in turn, needs to set the right environment for DevOps adoption by sponsoring transformation efforts, by enabling change to rigid legacy processes, and by a top-down push to overcome cultural inertia.
NOTE A bottom-up practitioner-led effort allows extremely productive individual teams to adopt DevOps and thrive. A top-down executive leadership-led effort enables these individual successes to scale.
The engagement of the business is imperative for these scaling efforts to succeed. IT organizations exist to deliver the capabilities a business needs in order to deliver business value to its customers. The business is asking the IT organization for optimization—to be more agile, to be resilient to change, to be more responsive, to do more with less, to be more productive, to increase throughput, to deliver faster, to deliver at higher quality, to be reactive to the market, to accelerate past the competition, to keep up with an ever-changing regulatory and compliance regime, and, yes, to reduce expenses.
In addition, it may also be asking for innovation—to allow the company to enter new markets, to enable exponential growth, to engage and grow the customer base, to be responsive to customers’ needs, and, again, to reduce expenses. Delivering on these asks (hopefully not all of them at the same time) is what drives the need for change. It is what creates the motivation to work toward achieving the benefits that come from adopting DevOps.
NOTE You do not just adopt DevOps because it is cool. You need to have a business reason. The need for agility or velocity is the number-one reason that DevOps exists. The maturing and wide adoption of DevOps over the last few years is a reflection of today’s dynamic marketplace, of customers’ expectations.
Thus, in order for IT to undergo a transformation, this change has to improve and enhance IT’s ability to deliver business capabilities in a manner that, in turn, improves and enhances the business value delivered. Proper partnering between the business and IT is imperative so that the transformation IT undergoes by adopting DevOps delivers what the business needs the most by properly balancing optimization and innovation. Business goals have to drive why IT transforms, which will in turn drive how IT transforms.
This book will categorize DevOps adoption plays as follows:
It will include lessons learned, examples, success patterns, and anti- patterns for each adoption play. Like a sports playbook, this book is designed to deliver certain plays that can be executed for different scenarios and situations—depending upon your organization’s current maturity and state—when it comes to transforming to a high-performance application delivery organization by adopting DevOps. An organization will need to take those plays and execute them in a tactical manner, based on the projects and teams that are adopting DevOps. Just as no battle plan ever survived contact with the enemy, these plays will need to be executed with an action plan or a broader adoption roadmap designed for each organization.
Furthermore, no organization is monolithic or homogenous in nature. Some parts of the organization may be more mature in some areas, but less mature in others. Some teams and groups may have already achieved agility and velocity, whereas others may be experiencing tremendous cultural inertia, all within the same organization, sometimes in the same building; they all need to work together in order to get scale.
An organization may have an innovation lab that is using modern agile and DevOps practices, while core systems teams may still be delivering in a rigid waterfall manner. Thus, these patterns of adoption will apply differently to different parts of the same organization and will need to be customized to suit the needs of various teams. To help with this customization effort, this book will also apply the technique of value stream mapping, used for decades as a component of Lean practices, that can be used to develop a custom adoption roadmap from these plays for an organization’s business goals, current maturity, and capabilities.
We live in an era of massive change. In 1960, the average life expectancy of a Fortune 500 company was 75 years. Today, the average lifespan is just 15 years and declining further. So what gives? You only have to look as far as what is referred to as the Uber effect to understand why so many companies fail. A company led by a founder who is not from the taxi industry, Uber disrupted a centuries-old industry with the touch of a button on a mobile app; they have made cab service part of the on-demand, service economy, where consumers get what they want, when they want it, with no delays. New IT capabilities—accelerated by the intersection of new approaches like Agile and DevOps and technologies like cloud and microservices—are allowing startups armed with no more than services on a cloud and a mobile app to Uber large, established organizations with massive IT investments, valuable infrastructure, and experienced people.
NOTE The fastest-growing transportation company in the world does not own any vehicles (Uber); the fastest-growing hospitality company providing living space for rent owns no property (Airbnb); the fastest-growing media company in the world produces no media (Facebook); the largest encyclopedia in the world has no staff writers (Wikipedia). Disruption is real.
So, ask yourself, is your organization a disruptor or a disruptee? The reality is, most organizations are the latter, putting IT organizations under more pressure today than ever before. Whether it’s the fear of being Ubered by the competition or business demands to pick up the pace, IT organizations face a balancing act of ensuring the optimized operation of core applications and of being innovative. However, the truth is, innovation and maintaining the efficiencies of legacy systems can co-exist. While the prospect of competing with born-on-the-web companies like Uber and Airbnb may seem daunting, adopting DevOps at scale across your organization can enable your IT team to become more agile, efficient, and innovative. Adopting DevOps can put your IT in a position to become the enabler of change at your organization so it can fend off the disrupters; this in turn allows it to become the enabler that lets your organization become the disrupter. In today’s technology-driven world, IT capabilities have become the key differentiators between the disrupter and the disrupted.
Before I begin to delve into the core capabilities and practices that you need to adopt and the various plays you need to execute in order to adopt DevOps in an organization, it is essential that you understand the definition of the term DevOps.
DevOps, like any new technology or tech-related movement that is adopted in industry, has become an overloaded buzzword. Everyone talks about it, not everyone knows what it is all about, and worst of all, many of those who claim to do it are really doing a terrible job. There are some excellent examples of companies that have excelled and are at the leading edge of the DevOps movement—the often-cited Etsy, Flickr, Facebook, and Netflix come to mind. But even here, there is contention and debate as to what is truly the best approach to DevOps. Netflix says what they do is NoOps, with developers taking over Ops responsibilities. Yet others counter that such a situation would lead to anarchy.
Such debate is to be expected as the industry refines what DevOps is as it evolves. As I will discuss at length in this book, there are different approaches to adopting DevOps, and each organization should look at adopting the right capabilities and practices of DevOps, based on their individual risk-value and business drivers balance. In fact, this adoption needs to start at a project level and then be scaled across the enterprise, leveraging techniques that will be described in this book.
As I mentioned previously, there are as many definitions of DevOps, or at least opinions of what DevOps really is, as there are blogs and tech “experts.” There is the perspective of DevOps where the developer is king; DevOps where continuous delivery is the driver; DevOps where it all hinges on the cloud and one cannot have true DevOps without a cloud; and DevOps where DevOps equals microservices. So, let’s start with the definition listed on a (fairly) neutral source—Wikipedia (Wikipedia, 2016):
DevOps (a clipped compound of development and operations) is a culture, movement or practice that emphasizes the collaboration and communication of both software developers and other information-technology (IT) professionals while automating the process of software delivery and infrastructure changes. It aims at establishing a culture and environment where building, testing, and releasing software, can happen rapidly, frequently, and more reliably.
It is important to note that the Wikipedia definition has also evolved over time, as DevOps has matured. For comparison, here is the definition listed on Wikipedia in 2013:
DevOps (a portmanteau of development and operations) is a software development method that stresses communication, collaboration and integration between software developers and Information Technology (IT) professionals. DevOps is a response to the interdependence of software development and IT operations. It aims to help an organization rapidly produce software products and services.
This evolution of the Wikipedia definition is indicative of the evolution of DevOps and how the industry views DevOps. Other than the replacement of the esoteric portmanteau, which had everyone looking it up on Dictionary.com, the key points to note are as follows:
Of course, I would be remiss not to mention the most concise definition of DevOps ever written; this was seen on a T-shirt at the O’Reilly Velocity Conference in 2013:
DevOps—taking the SH out of IT!
A sports team does not just have the players who take to the field on game day; it also has everything from coaches, assistant coaches, team management, team executives, trainers, doctors, nutritionists, physiotherapists, equipment managers, all the way to ball carriers and water servers. All are essential, and all need to excel in their roles and how they work together as a team, for the team to perform at its highest capacity. The same way, DevOps is not just about development and operations practitioners. It requires all the stakeholders in the application delivery pipeline to transform how they work, how they collaborate and communicate, how they work together like a high performance team.
This book is for all the team members in an organization who are stakeholders in the application delivery pipeline—from line of business owners, to analysts, architects, designers, developers, testers, quality assurance (QA) practitioners, automation engineers, infrastructure engineers, operations practitioners, database administrators, system administrators, documentation writers, project managers, product owners, all the way to c-suite executives. These roles may vary by organization, and many will need to evolve and transform what they do and how they do it as the organization adopts DevOps. This book is designed to benefit them all.
The application of each play discussed will impact each stakeholder role differently—some will see significant change in their role and how they interact with others, and some will see none at all. Just all the ball carriers and water servers are typically not impacted by which plays a team runs, but they are still stakeholders who can impact team performance if they do not perform as expected. The same way certain roles are supporting roles in IT too. Other roles, like key stakeholders who directly work with artifacts and processes that are a part of the application delivery pipeline, will benefit significantly from the plays in the book. They are the players and the direct supporting staff who play the game or support those who do, enabling them to perform at peak performance capacity.
Chapter 1 is an overview of DevOps. It documents the evolution of DevOps from its origins to today. It defines and describes all the common practices and capabilities that make up DevOps. It sets the stage with the broad definition of DevOps and of a DevOps transformation, which are used as the premise of this book.
Chapter 2 is focused on the leaders on the team: the coaches, the team captain, the senior players who form the core of the team. It focuses on how to assess the playing conditions and the competition to develop and select the right set of plays—the playbook for the team. It is for the IT management, project and program managers, product owners, team leads, senior practitioners, DevOps coaches, and anyone who aspires to be one of them.
Chapter 3 provides guidance on how to build a business case for a DevOps Transformation, allowing for the right sponsorship and investments to ensure success.
Chapters 4 through 6 are the actual plays. They are categorized as follows:
Chapter 7 is a chapter dedicated to the executive leadership driving the DevOps adoption. Like the general managers and executives of a sports teams, executives make the executive decisions and set the direction and culture of the organization. They are the ones who need to make the decision to undertake a DevOps transformation. They need to make the necessary investment and sponsor the transformation. They will need to know how to make the business case and determine the return on investment for such a transformation. They need to drive the transformation, from the front, across the enterprise.
The book also has one appendix. It is an example of a DevOps Transformation adoption roadmap, developed for a fictitious bank by delivering a value stream mapping exercise.
The book purposefully has tried to remain tool and platform agnostic. While several examples of tools, platforms, and technologies—both commercial and open source—are made throughout the book, they are done to as current examples of tooling and platforms available in the market to enable automation. Tools are necessary to automate processes, making them fast, repeatable, scalable, and error free. However, tools and platforms are continuously evolving and getting replaced by newer and better ones. It is hence a futile effort to recommend or even document available tools and platforms. The goal is to highlight capabilities, while remaining as tool and platform agnostic as possible to remain relevant even as tools and platforms available evolve.
Individual commitment to a group effort—that is what makes a team work, a company work, a society work, a civilization work.
—Vince Lombardi, legendary American football coach
There is nothing that transcends culture, language, and geographic borders than sports. If you have any doubts, just watch the reruns of the recently concluded Olympics in Rio. Analogies from sports are also very relevant to application development and delivery, as they are both team events. While developing or delivering a new application or service may not require the physical conditioning an Olympics gold medalist does, they do require the leadership, communication, collaboration, and trust that any team sport needs.
I also have a personal passion for sports. Right from my childhood I grew up in a household that had a love for sports. My maternal grandfather was an Olympian and national sports figure in India. He played for the Indian National Hockey team in his youth and later was a sports executive with the Indian National Football (Soccer) team at the 1952 Helsinki Olympic Games. He also had the opportunity to run with the Olympic torch for the 1964 Tokyo Olympics, as the torch passed through India. He remained a sports executive for domestic soccer tournaments for several decades after that, including when I was a child. Growing up with an Olympic torch in the family home gives one a high respect for sportsmen and women.
Major Lachhman Singh running with the ‘64 Tokyo Olympics Torch (Source: Singh Family Personal Collection)
In the book I have taken examples, quotes, and players’ and coaches’ experiences from multiple sports and mapped them to the Plays of DevOps Adoption. The parallels are intended to make the plays more relatable and understandable and hopefully the book more interesting to read.
This books comes with a companion website where I will continue to post updates and new content, including case studies, presentations, videos, and outtakes from the book. Check it out at http://devopsadoptionplaybook.com.