Skip to content

How Three Principles Transformed a Bank’s Technology Landscape

Implementing Agile and DevOps principles is difficult in almost all organizations. Imagine scaling the three principles of DevOps to transform product delivery and quality in the world’s largest bank

When I joined JP Morgan Chase in 2014, I was greeted with this quote by the Lead Development Manager:

“We work in a highly regulated environment, we have tight deadlines, and we have teams across the world. To ensure that we don’t make mistakes, we need to build, test, and deploy manually. Our environment and situation are special, nothing like you might have experience elsewhere.”

I was hired as a Transformation Coach to help the team, the product owner, and the stakeholders adopt better Product Management, Agile, and DevOps principles and practices. Curiously my working title was a Program Manager. I actually had no clue what a Program Manager really does.

As a coach, my job was to observe and help the team despite their current beliefs.

“A common disease that afflicts management the world over is the impression that our problems are different – They are different but the principles that will help to improve quality of product and service are universal in nature” – W. Edwards Deming

Background

The product, was regulatory in nature, had to be released with new attributes, rules, and workflow once every quarter. Working backwards, we started a new test release cycle a month before release in the UAT (User Acceptance Test) environment. The team worked on requirements gathering, development, QA testing every sprint (each sprint was two weeks long). Our larger team was split into six individual teams, two in New York, one in Glasgow, and three in India.

Work distribution

Cumulatively, the teams were working on six streams of work. Every 1.5 weeks, the developers integrated the separate code bases and merged them manually. They then tested them before involving the QA people in the team. 

Each manual merge would take two days. Promoting code from the development environment to the QA environment required a lot of coordination. Sometimes the database would be ahead of the code. Sometimes the code base worked well in the developer’s workstation or in the dev environment. Once the team moved it to QA, it would not work anymore. After several hours of debugging, the team identified a fix and applied it manually to finally complete the merge.

Testing was performed by the members in the Scrum teams that had QA expertise. They identified bugs and then debated fixes with the coders back and forth like Ping-Pong balls. 

In addition to the development challenges across many streams, testing cycles, and system integration, the team also struggled with requirements gathering. Product management was chaotic. Several handoffs between the stakeholders, the product management team, and the business analysts within the various Scrum teams.

I observed these behaviors and challenges both in New York and in India. I started creating an idea of how to help these teams get better by using DevOps practices and principles. 

In DevOps there are three core principles, and they are known as the “Three Ways”:

The First Way: The Principle of Flow

Lead time is the time from which an individual request is made until that work gets delivered. Cycle time is the time from when the team starts working on the new request until it’s delivered. Watch my video on lead time vs. Cycle Time

Large, complex organizations that work on tightly coupled, monolithic applications and use limited integration environments. Typically requiring months of lead time. Their flow of work is slow, and they have limited ability to adapt quickly to changes in requirements.

“One of the major contributing causes of chaotic, disruptive and catastrophic software releases is the long lead time required to get to the releases, caused by siloed work performed by development and operations” – DevOps Handbook by Gene Kim

Increasing Flow

Many successful companies like Amazon, Netflix (https://netflixtechblog.com), Meta, and others have been able to increase the flow of work by limiting the work in progress, by delivering work in small batches, by reducing the handoffs that are typical with siloed functional areas in a large organization, and by making the hierarchy flat. This makes them different than traditional large complex organizations.

Ideally, developers receive fast, constant feedback on their work, which enables them to quickly and independently implement, integrate, and validate their code and to deploy their code into production. They do this continuously, with small batches of work deployed. Fast feedback is the goal and smooth flow of work from development to operations allows to deliver value to the customers quickly.

To implement this principle, work needs to be visualized using value stream mapping or impact mapping. I considered visualizing the whole system as one value stream, but quickly realized that no one would be open to this large-scale change.

“Choosing a value stream for DevOps transformation deserves careful consideration” ~DevOps Handbook by Gene Kim

Implementation Steps

I first chose to ensure that we could configure and create server environments on the fly. This was the low-hanging fruit to help the team reduce the overall delays and hence the lead time. We created different configurations and a script that would pick the required configuration and build the server environment on the fly. By versioning the configuration, we ensured that we could completely destroy an environment and build it from scratch in an automated way.

Seeing the positive impact of the infrastructure improvements, I was able to secure budget for a full-time engineer to support other value streams. With the additional capacity we were able to build a continuous integration and delivery pipeline using Jenkins [ https://www.jenkins.io ] and Gitflow [ https://nvie.com/posts/a-successful-git-branching-model/ ]. 

Our next challenge was to get the databases in each environment in sync. We used a tool to keep the databases in sync, destroy and load the database in every environment on the fly. [https://www.liquibase.org/]

Finally, we took a proactive approach to build unit and automated functional testing. We built an entire delivery pipeline weaving the development, the testing, the database, and the infrastructure so we could deploy on the fly in every environment.

The Second Way: The Principle of Feedback

Failure is inherent and inevitable in complex systems, whether in manufacturing or technology. When a problem is detected in one component, it is often difficult to isolate the other components. It also defies a single person’s ability to see the system as a whole and understand how all the pieces fit together. So, creating fast feedback loops is critical to achieving quality, reliability, and safety in the technology value chain.

We hooked up the tools we used for versioning, for quality testing, and for continuous integration and deployment, to an open source telemetry solution called Hygiea. [https://github.com/hygieia/hygieia]

This started providing Developers and Operations staff real time view into the entire delivery cycle and allowed us to enable quality gates so that we could stop the pipeline on the first sign of trouble. Similar to an Andon Cord used by the Toyota manufacturing teams to stop the production when there is a problem. https://psychsafety.co.uk/psychological-safety-79-the-andon-cord/

The Third Way: The Principle of Continual Learning and Experimentation

This principle encourages a culture of continuous learning and experimentation within DevOps teams. By constantly trying new things and measuring the results, teams can improve their processes and deliver better products faster. 

The Third Way also supports a culture of sharing: Lessons learned from experiments and failures are openly shared within the team and organization. This helps to prevent others from making the same mistakes and accelerates collective learning.

We applied the principles learnt from the book The Geek Way by Andrew McAfee where everyone in the team could raise issues and suggest ideas that were taken up by the collective team. We encouraged experimentation.

Our local discoveries spread. We documented and promoted the learning from this work across the firm. This led to other teams hiring engineers and adopting similar ideas. DevOps started to become a company-wide practice. 

In a couple of years, the firm created a continuous integration and delivery product that every team adopted firm-wide.  

Watch JP Morgan’s Product Leader discuss the firm wide CI/CD Tool Suite.

Firm wide, the three ways of DevOps principles have transformed product delivery and quality in every team in the world’s largest bank

References:

12 Factors that change how you achieve operational excellence

Software Craftsmanship

DevOps Certification Courses

Join the conversation

Your email address will not be published. Required fields are marked *