ci/cd pipeline

Ancestry was experiencing issues with performance, staying in budget, efficiency for some time, and decided to begin a digital transformation beginning with their CI/CD pipeline. This even included stopping the release of code altogether to standardize all their systems! They soon realized that their issue was not only outdated pipelines but also how they approached developing these infrastructures – learn more about what changes were made to pipeline infrastructure as well as best practices you can apply in your enterprise so you can achieve better productivity, fewer headaches, and happiness where things work.

1) The CI/CD tools proliferation was creating a fragmented organization

Multiple teams were using their own processes, each with their own set of tools for continuous integration and deployment. This made it difficult to govern costs, availability, or productivity across the board because there wasn’t an overarching plan guiding how things should be done.

With each team adopting new processes (e.g., canary deployment) for their discrete tasks, the company as a whole couldn’t scale fast even if single teams were able to do so. As a result, it became near-impossible to govern for top-priority goals: cost and availability.

2) Messy, Old CI/CD Pipelines were slowing down the company

The issue at the company was not only about their outdated pipelines but also the approach they were taking to develop these infrastructures. There was a lot of room for improvement. Especially when it came to continuous integration with source code changes where every team relied on old school large deployments instead of incremental updates, which made things slow down significantly, some teams did have best practices in place, such as full tests before releasing any branch or piece of software, but once again, this wasn’t across the board, so there was still plenty left that needed fixing.

3) Comparing Canaries

At the time, two weeks was considered a benchmark for excellence on Jenkins. But most of your teams took considerably longer to release new products and services. When you multiply this across 10 systems at once within an organization with 70 different teams, there’s obviously something wrong with how things are going. The question becomes: “how do we fix these issues?”

Ancestry started this digital transformation thinking it would be great, but it turns out this was not the case. They panicked and thought, “what can we do now?” So Ancestry made a deliberate decision that really takes guts: the CEO basically said, “Look, everybody stop, we’re not releasing any more software until you standardize everything.” 

We have these comparing canaries because you have to deploy a canary every time you deploy a new system. Instead of deploying a new canary, they just standardized on the same system. Likewise, they got rid of multiple pipelines of Jenkins infrastructure and went with Harness. With this transition, they were able to reap some significant benefits in return.

The Fix

Ancestry stopped development and spent all effort migrating to a common CI/CD platform (Harness) to tackle the problem plaguing their company. However, the impact wasn’t as great as feared because it was no different from rolling out a new canary release. With this brave decision, some teams were able to get up-to-speed quickly while others struggled with change; however, the long-term benefits of having a standardized infrastructure around which global changes are now easy has proven to be worth the short-term pain. The net results are better productivity, efficiency, fewer headaches, and happiness where things work! Tune in to our next blog on how Ancestry implemented COaaS to their pipeline to improve performance, reliability, and cost.

For more information on Ancestry’s cloud optimization journey, click here!