Hello, all my fellow DevOps practitioners and enthusiasts!  I recently wrote a book review about The Phoenix Project, a fictional novel that follows an organization’s DevOps transformation, thereby enabling it to escape an impending doom (going out of business) and begin winning in their market once again.  Towards the end of the novel, one of the characters asks the VP of technology to do him one favor–to “write a book, describing the Three Ways and Parts Unlimited.  Call it The DevOps Handbook and show how IT can regain the trust of the business and end decades of intertribal warfare.”  And wouldn’t you know it?!  Gene Kim, Jez Humble, Patrick Debois, and John Willis were already writing The DevOps Handbook!  Where The Phoenix Project was a fictional novel, The DevOps Handbook starts with a technical breakdown of the underpinning principles of DevOps (“The Three Ways”). It continues with a prescriptive method for implementing DevOps within your organization.

The core chronic conflict

Before discussing the principles of DevOps, the Handbook first covers a significant problem with many organizations that directly impedes their ability to continue winning in their industry.  As it turns out, there is a core, chronic conflict within the majority of Technology organizations that most businesses don’t even realize they’ve created.  Development teams are incentivized to be nimble and rapidly respond to customer needs by regularly developing and deploying new services and features.  However, the IT Operations teams are incentivized to provide stability and reliability for the customers and consumers of production software and services.  Without realizing it, the business has created a disastrous recipe where the Development and Operations teams are fundamentally at odds with each other. Devs want to increase the speed of change (by deploying new features), and Ops wants to prevent instability in production (which happens when things change).  When a Dev team misses a delivery deadline because a new feature isn’t deployed to production successfully, they almost always blame the Ops team for blocking their progress.  Whenever the Ops team is firefighting production incidents that directly impact customers (at all hours of the day or night), they will blame the Dev teams for pushing new changes to production without proper testing.

This culture of blame only causes further distancing between the teams, and silos are quickly put in place to try and “protect” either of the teams from blame.  The Dev teams will begin to increase their velocity to prove that it isn’t their fault new features aren’t being delivered to the customer.  And Ops teams will devise layers of process and change management to ensure that ANY outage or customer impact can be directly tied back to the new features that were deployed.  Before you know it, the entire organization begins in a downward spiral, and the company’s goals are continually being missed because the Technology value stream isn’t optimized and prioritized.  And it has been proven that when IT fails, the entire organization fails because this day in age, every organization needs to function as a technology company.  So, how do we mitigate this downward spiral and implement the practices required to achieve the organization’s desired outcome?

The three ways of DevOps

The First Way revolves around the principles of the flow of work and emphasizes systems thinking to achieve the global goal of delivering value to the customer.  In most cases, this means that a product, service, or new feature is deployed to production so a customer can evaluate it.  Since workflows from Dev teams to Ops teams, it is important to think about the entire delivery process and not only optimize any single work center.  For instance, if a Dev team can produce more features than the Ops team can deploy, then the customer doesn’t get any added value from the entire system.  Systems thinking focuses on optimizing each of the work centers so that flow is increased and requires that silos are broken down.  This typically means that the Dev and Ops teams need to begin implementing a close-knit collaboration with a genuine interest in understanding the entire system.

When the Dev and Ops teams begin collaborating, they can quickly identify hardships in each other’s daily work.  These hardships create waste, whether that be wasted resources, time, or work, that adds no additional value to the customer.  Making these visible makes it possible to put in mitigations to prevent these hardships, which ultimately optimizes the flow of work so the value can be delivered more rapidly.

The Second Way targets the importance of implementing and amplifying feedback loops throughout your entire system.  Where the goal of The First Way is to understand the system, the purpose of The Second Way is to improve the overall quality of the products and services that are developed.  This is the foundation for continuous improvement and “shifting-left” in a Technology organization, where developers are immediately notified if a new code change has caused an issue.  This is done by implementing quality gates in the form of a pipeline of automated processes, where any failure is raised so it can be corrected before ever moving further in the delivery lifecycle.

This is very similar to Toyota implementing the Andon Cord at each of its manufacturing work centers that can be pulled if any worker spots a problem or has an idea for improving the overall system.  When this occurs, the issue is swarmed and fixed immediately to prevent that issue from flowing downstream to other work centers, and the worker who pulled the cord is thanked!  Applying this concept to software development, when a pipeline stage (or quality gate) has an issue, the entire build process breaks, and the issue must be fixed.  This increases everyone’s confidence that the same problem will not arise in a production environment.  In theory, this will not only improve the quality of software being developed but should also reduce or even prevent production outages that are directly related to software changes being released, since most issues are detected much earlier in the development process.

The Third Way focuses on organizational culture and enabling continuous experimentation, which leads to improvements built on the learning and knowledge gained from those experiments.  Let’s face it; we all work with highly complex systems, and understanding the impact of any change in complex systems is almost impossible.  But, that doesn’t mean we can’t create some safety while working within such complicated systems.  This is where an organization needs to create a culture that doesn’t reprimand failures but views failing as an opportunity to improve the reliability of the overall system (if we’re going to fail, let’s fail fast).  Previously, Dr. Ron Westrum defined a model with three distinct types of organizational cultures.  When applied to technology, it has been determined that an organization that has implemented and adopted a “Generative (performance-oriented)” culture fosters continuous experimentation and improvement because blame is eliminated, and fear of failure is abolished.  When an outage occurs, instead of seeking someone to blame, a genuine inquiry can help produce countermeasures to prevent the issue from happening again.  This is bubbled up through the entire organization so that everyone can learn from failure.

This unlocks an organization’s potential for innovation by allowing people to take risks, test out hypotheses, and, most importantly, to practice regularly within a safe environment without the fear of failure.  New ideas are actively sought and tested.  Teams are encouraged to introduce chaos into their running systems to discover the potential for new types of failures and fill those gaps to increase the overall resiliency of their complex systems.  But none of this can be accomplished unless the value is placed on organizational learning, and the entire workforce feels a high sense of trust while trying to improve their daily routines.  As we all know, failure is inevitable.  But creating a sense of safety and viewing failure as an opportunity for global learning will ensure that your organization can continue to evolve and keep pace with the continually changing landscape of any market.

DevOps transformation

After understanding the underlying principles that create the foundation for the DevOps movement, many are left wondering how they should get started.  The DevOps Handbook covers exactly how and where to start, including how to visualize your value streams and how to identify innovative teams to gain some traction with small successes that will help expand the DevOps philosophy across the organization, building critical mass.  Of course, in every organization, you’ll have the early adopters of new technologies as well as the laggards or skeptics, which will need to be converted after first creating the silent majority within the organization.  The Handbook also provides guidance on the different organizational archetypes and why most organizations need to transition from optimizing for cost and move towards optimizing for speed (enabling market-oriented teams).  This valuable information lays out the path for your DevOps Transformation, including how Conway’s Law effects team communication and coordination, as well as how each team member should avoid overly specializing and move towards being “T-Shaped” generalists.

One of the strengths of The DevOps Handbook is the real-life case studies embedded throughout the book.  Companies like Nordstrom, Etsy, Amazon, Target, and CSG, just to name a few, have adopted and implemented DevOps values to turn their organizations around.  At CSG, after their transformation was complete, deployments became so frictionless and routine that Ops resources would spend free time playing video games, and the value was constantly being delivered to the customer in half the time.  Changes weren’t feared because they worked to create a safe system where deployments could take place easily, and rollbacks worked seamlessly.

In addition to case studies that prove how DevOps can benefit your organization, The DevOps Handbook discusses the technical practices of setting up your continuous delivery pipelines, the benefits of automated testing and trunk-based development, and even the differences between deployments and releases using feature flags.  From direction on database changes to blue/green deployments to the use of immutable infrastructure to create production-like environments, The DevOps Handbook provides an invaluable exploration into IT organizations and delivers the resources needed to transform your workforce and become a world-class technology organization completely.

The one question that remains, however, is, “Why should we adopt DevOps in the first place?”  The Handbook makes it abundantly clear that technology has already proven to be a differentiator, and the companies that don’t embrace technology as a competitive advantage will fall victim to disruption in their industry.  Blockbuster was disrupted by Netflix.  The book store and retail industry have been and continue to be disrupted by Amazon.  Uber built an app and disrupted the Taxi industry.  There is an ongoing struggle for survival, and those organizations that adopt DevOps principles and patterns correctly will have adapted themselves to be able to out-innovate and out-perform the competition whenever customer needs and market demands illicit change.  Which, as we all know, happens ALL THE TIME.

Get in touch with us to learn more about how Beyondsoft can help your organization accelerate innovation and begin your DevOps Transformation.

DevOps transformation is more than just adopting tools and automation, although these are critical parts. One of the foundational pieces of a successful DevOps transformation is creating a healthy, generative culture that encourages learning, collaboration, and continuous improvement.

Another critical piece for enabling DevOps transformation is having a comprehensive understanding of the value stream across the entire application lifecycle. Value stream mapping plots the multitude of workstreams, including handoffs, wait times, and critical steps. Value stream mapping enables organizations to uncover opportunities for optimization, automation, and continuous feedback loops that ultimately empower developers to deliver quality, reliable products faster.

Below, I’ll share how we helped one of our customers, a major automotive manufacturer, build a generative culture, empower applications and operations teams, and create long-term, sustainable change and value through DevOps transformation.

Automotive manufacturer embraces DevOps

Our customer needed to shift to a DevOps culture and engage application vendors in a new way of development. The previous development lifecycle processes created a wall between app teams and the operations vendors, resulting in a communications void that led to longer lead times to remediate issues and deliver innovation to the customer.

Our charter was to streamline the development lifecycle, help the customer’s teams become DevOps practitioners, and onboard them onto a DevOps platform. 

Value stream mapping

As part of our engagement, we embedded ourselves into the culture and worked within each of the teams to perform value stream mapping. Value stream mapping is all about plotting out the entire delivery lifecycle: every workflow, every vendor, every handoff. And with around 20 vendors, more than 60 microservices, multiple tools, and multiple environments, there were plenty of moving parts.

Once we completed the value stream mapping, we began scouting out opportunities to automate, consolidate, and remove friction. We found plenty.  

For example, we learned that while some developers were producing code quickly, it took an average of three days for operations to set up a test environment—or even longer if they were contending with a production issue. Here we found an opportunity for deployment automation, including the necessary handoffs. This not only saved time, but it also created an accelerated feedback loop.

By uncovering opportunities such as this and building in automation, we streamlined processes and set up teams to innovate faster and deliver more stable products.

DevOps Dojo training program

A key part of cultural transformation is connecting people, breaking down silos, and helping key players understand the importance of DevOps. One way to do this is through a custom DevOps Dojo training program. DevOps Dojo training creates a safe environment to onboard teams to help them understand and buy into the DevOps vision.

In fact, hosting a DevOps Dojo training session with all the teams and vendors in one room was one of the first things we did. We walked the teams through the underpinning principles of DevOps transformation, including the three ways of DevOps. By helping them envision and align with what needed to be accomplished and how it would help them, we laid the foundation for greater cooperation and collaboration. The Dojo training helped each team understand the bigger picture and how their workflows impact other teams.

These kinds of sessions can be a real eye-opener. We found that a lot of the vendors weren’t thinking holistically. They were just focused on their areas. They realized that even if they’re churning out code quickly, if their code isn’t passing quality gates or if it isn’t flowing into an optimized process, they’re not generating value. This kind of understanding helps breaks down walls and makes it easier to collaborate.

DevOps platform onboarding

Once we finished value stream mapping and identified optimization and automation opportunities, we established an onboarding strategy for the application teams. We set up the DevOps tools and technology and automated access levels and role-based access controls. We got the teams up and running on a scalable DevOps platform with a standard set of reliable, resilient, self-healing tools. And we conducted DevOps Dojo training sessions to help them fully on board.

Today our customer has transformed into a true DevOps organization. They’ve accelerated their development lifecycle and are running an optimized CI/CD pipeline that is injected with quality gates, giving them and their customers greater confidence in overall product reliability.

Get in touch with us to learn more about how Beyondsoft can help your organization acceleration innovation through DevOps.

It’s a common scenario.

The development team is working hard to code new products, features, and updates. They’re incentivized to get code out the door as quickly as possible. They’re coding and testing on their individual machines, which could have any number of unique configurations.

Meanwhile, the operations team is working hard to keep the production environment stable. They’re incentivized to keep systems online and running smoothly so there are no customer impacts. Which means they often block things (like code) from going into production and causing instability.

Enter the Wall of Confusion.

Having thoroughly tested code on their machines, the development team throws their code over the Wall of Confusion to the production team. The production team tests the code, which does not operate as expected in the production environment. The production team throws the code back over the Wall of Confusion to the development team.

The development team gets frustrated because the product worked fine in their environment—so why can’t the operations team get it to work? The operations team is frustrated that the development team can’t build code that runs as it should in production.

This back and forth may happen several times before the code gets deployed into production, wasting time and money while escalating tension between the two teams—both of whom are doing the best they can

So how do you break down silos and topple the Wall of Confusion? Below, I’ll outline how DevOps practices can help by creating a consistent platform along with automated practices that accelerate continuous integration, continuous improvement, and continuous delivery while saving time and money.

CI/CD enables DevOps

First, let’s dispel the myth that you can deliver quickly OR maintain stability. DevOps provides the principles for building a highly available, cloud-based platform with automation tools for deployment into every environment. A CI/CD pipeline injects automated quality gates to give you confidence in the reliability and security of the code you’re deploying.

Embracing DevOps also means breaking down monolithic apps into microservices that can be updated and released into production without impacting any other teams. These smaller services enable developers to deploy with greater agility. So rather than waiting to release a massive update every three to six months (or longer), teams can deploy smaller updates multiple times a week (or even multiple times a day).

Finally, by establishing a stable DevOps platform, all developers work on a standard platform that provision production-like environments on demand. The platform offers a standard set of tools that are reliable and resilient, self-healing, and always available for app teams to leverage when building their CI/CD processes.

Beyondsoft helps leading automotive manufacturing company to modernize through DevOps automation

This use case is a classic example of how a scalable, enterprise DevOps solution can accelerate and automate continuous deployment, integration, and improvement.

The automotive leader wanted to break down silos and make innovation more accessible. But their system and processes created long development cycles and yielded increasing amounts of technical debt. Their teams couldn’t modernize in a standard way and projects languished in limbo for extended periods of time. To solve for this, Beyondsoft worked with the company to create a standardized set of automation tools by provisioning an enterprise-scalable DevOps platform, allowing their application teams to easily create continuous delivery pipelines. We helped them unify their development teams onto the highly available, cloud-configured DevOps platform and ensured they were enterprise scale ready. We then onboarded more than 120 different apps.

Features of the DevOps solutions include:

• Automation in the cloud: Even if an availability zone goes down, their platform doesn’t. The solution is highly resilient and highly available. Thanks in part to the cloud, there is no single point of failure. Everything from infrastructure provisioning to app configuration is automated. If the company decides to make a change to the tools or infrastructure, they just update the code. Plus, as they expand, the platform scales with them.
• Container-first strategy: The DevOps solution embraces containerization to further increase availability and resilience. If one containerized tool has an issue, the platform is self-healing and will automatically replace it with no impact to the application teams.
• Seamless ecosystem integration: The DevOps solution seamlessly integrates with their system as well as on-premise systems.
• Application language agnostic: No matter which coding language a developer uses, the DevOps platform can handle it with standardized pipeline patterns.  

  Everything from code builds to code scans to code deployment can be done securely from the DevOps platform. Artifactory provides a universal, binary repository management system that enables developers to deploy into any environment without code changes. 

  Today, this automotive company is enjoying all the benefits a scalable DevOps platform has to offer:

  • Shorter, faster release cycles
  • Better quality code and smoother deployments
  • Minimal downtime for the developers and their customers
  • Fail-fast environment
  • Amplified feedback loops via CI/CD pipelines and visibility into technical debt
  • Task automation
  • Increased efficiency and productivity
  • And no more silos!
• To learn more about how Beyondsoft can help you create an enterprise scalable DevOps/DevOps platform to accelerate innovation, give us a shout.

At Beyondsoft, our job is to not only solve customers’ biggest problems, but we hope to provide them with the tools and knowledge to continue making the right decisions for their business. This is why we decided to start a book review series with some of our experts. This week we sat down with Beyondsoft DevOps Architect, Jose Gonzalez and talked to him about The Phoenix Project, A Novel about IT, DevOps and Helping Your Business win. 

You’re a DevOps Architect here at Beyondsoft. Can you tell us more about your role and why you choose DevOps?

I’ve just celebrated my two year anniversary with Beyondsoft in February. My role is focused on applying DevOps principles to projects to help accelerate delivery and increase the overall quality of client products and solutions.  My aim is to ensure that we can truly shift-left and catch the problematic bugs and issues BEFORE we ever impact a real user.  In a sense, I’ve been a DevOps enthusiast for a very long time and have always believed that IT is much more than a cost center to an organization, so I feel like DevOps chose me.

You recently read the Phoenix Project, A Novel about IT, DevOps and Helping Your Business Win. Can you give our readers a quick summary of the book from your eyes?

If anyone has ever been on an Operations team, worked on-call support roles, or simply been part of an IT initiative, you NEED to read this book.  It all starts with a fictitious organization called “Parts Unlimited” that is being disrupted by its competitors and losing market shares very quickly.  An internal business project (Project Phoenix) is the organization’s only hope for success.  Project Phoenix has been in development for three years and there has been a twenty million dollar investment to bring this new product to market.  The problem is that the organization is WAY over budget, they missed their delivery deadlines, and there is very low confidence that the IT department can ship the product to help close the gap with competitors and regain their competitive advantage. 

The novel follows Bill Palmer who is promoted to the VP of IT Operations and tasked with making Project Phoenix a success.  As Bill starts to realize that the Development team has taken all of the available time before the ship date, he does his best to persuade the CEO to delay the launch and paints a picture of disaster if the organization continues.  It turns out the Marketing team has already purchased ads in the paper announcing Project Phoenix to the world, and a delay is now impossible. 

As you can imagine, the executive decision to push forward without heeding the warning leads to a complete catastrophe during the release of Project Phoenix.  Not only were the Operations teams ill prepared for deploying the code to Production, the Development team was still creating new “builds” to be shipped the day of the Production deployment.  Things quickly head south as a critical database re-indexing job has slowed to a halt and the release has also taken out all of the Point Of Sale (POS) systems across all the Parts Unlimited retail stores. 

The release, which should’ve completed in 6 hours on a Friday evening for a Saturday morning release turns into an extended deployment event, where Operations and Developers are working 24 hours into the middle of the following week before successfully deploying.  But, the system is so frail that it requires Ops resources to constantly reboot servers due to a memory leak.  And the product seems to be an overall failure because the features that were promised weren’t ready for the release and the organization encounters bugs in their product that end up displaying customer credit card information to other customers!  It’s a nightmare from the start.  The CEO tells Bill that IT is being held responsible for this entire disaster, and the Board members of the company are now considering outsourcing ALL of IT in the next 90 days.  Essentially, Bill has 90 days to turn the company around before he and the rest of his teams are all out of a job.

Luckily, Parts Unlimited is currently courting a potential new Board member candidate named Erik Reid.  After Bill and Erik are introduced, Erik quickly becomes an eccentric mentor that wants to apply manufacturing principles to the work of IT.  While Bill is a skeptic (because there is NO way that IT work can relate to Manufacturing), Erik continues to impress the “Three Ways” upon Bill, slowly helping Bill realize that IT work is very similar to manufacturing.  And optimizing the entire system’s flow only comes from assessing the “Four Types of Work” and applying the “Theory of Constraints” to identify and alleviate the bottlenecks that are preventing the IT organization to be viewed as a differentiator and an asset to Parts Unlimited.  If you don’t already know, Erik is laying down the foundation for embracing DevOps as a methodology for delivering quality solutions and building capabilities that help excel the organization to a prosperous future.  In the end, Bill becomes Erik’s student and completely turns around the organization with the ability to deliver features on a regular basis without fear of failure.  Parts Unlimited can now respond to the market and customer needs as fast as those needs change. 

What did you like most about the novel? Was there anything you read that you couldn’t wait to talk about?

Honestly, this book really sets the foundation for the DevOps movement that started in 2009.  With references to many important pieces of literature as well as the iconic John Allspaw “10+ Deploys Per Day” presentation delivered at the O’Reilly Velocity conference in 2009, I can attest to the praise that this book has already garnered.  It can turn skeptics into believers of the benefits that a DevOps transformation can begin delivering.  It strikes a chord with anyone in IT, because I’m 100% sure that you will be able to identify with at least one of the many characters in the novel. 

One of the biggest constraints in the book is a character named Brent, who is simply overworked because of his comprehensive knowledge of the entire organization’s IT systems.  Essentially, no changes, projects, or improvements to the systems can take place without his involvement.  He is a perfect illustration of the “Bus Factor” where if he was hit by a bus, the entire IT department and organization’s future might be in risk. 

I have known a few Brent’s in my life, and I have been Brent in previous roles.  One of the first realizations made by Bill is that Brent is a constraint to the flow of work because a single person cannot be in ten places at the same time.  So he works to elevate his constraint and protect work that comes to Brent.  Brent’s involvement in the real work to save the company is much too important to have his time completely consumed by so many different individual requests.  The “Theory of Constraints” is definitely a subject matter that organizations should embrace and incorporate into their IT department.

How does the book relate to what you do for our customers every day? How did it change the way you approached your job?

Customers and clients typically deal with the struggles of Parts Unlimited.  Technical Debt has made it difficult to make changes to existing code and fragile infrastructure has helped to pit the Developer and Operations teams against one another.  Developers want to deliver new features for the business but the Operations team NEEDS to keep stability of their systems so there isn’t any impact to the business’s customer base. 

By building a Continuous Delivery pipeline that keeps work flowing through the entire system and amplifying feedback loops for bugs and security vulnerabilities encountered, the Development team is empowered to iterate quickly on new features and the Operations team has confidence that the changes being made will NOT cause impact to their systems.  The old adage of delivering quickly OR remaining stable is thrown out the window because you can now accelerate innovation AND remain stable.

That said, the only way to truly achieve this outcome is through collaboration.  The Ops resources help automate their infrastructure provisioning, so environments can be created on-demand vs. having long wait times for the Dev teams.  Developers get a set of quality gates that force them to deliver high quality solutions to production-like environments.  The pipeline patterns are automated which reduce human error because code runs through a consistent, repeatable process.  Implementing this is my primary objective on any project or at any organization.  This the only way to deliver value to customers and stay on the cutting edge of market trends and customer demands.

So, now that you have finished The Phoenix Project, what book are you reading next?

Towards the end of “The Phoenix Project”, Erik asks Bill to do him one favor.  He says, “I want you to write a book, describing the Three Ways and Parts Unlimited.  Call it The DevOps Handbook and show how IT can regain the trust of the business and end decades of intertribal warfare.”  It turns out that Gene Kim had already begun collaborating with Patrick Debois (who coined the term “DevOps”), John Willis, and Jez Humble to write “The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations”.  So I guess you should keep an eye out for a future review of “The DevOps Handbook”.

Thanks to Jose for taking the time to explore The Phoenix Project, and to talk about how it applies to some of the real world scenarios he and his team face daily. If you’ve read the Phoenix Project, be sure to comment what you thought below!  For more information on our DevOps service offerings, contact us today!