Amazon vs Google: Contrasting Approaches to CI/CD

In the world of software development, Continuous Integration and Continuous Delivery (CI/CD) play a crucial role in ensuring efficient code deployment. Two tech giants, Amazon and Google, have adopted vastly different approaches to CI/CD, each with its own strengths and challenges.

Monorepo vs Microrepos: The Foundation of Divergence

At the heart of this contrast lies the fundamental difference in repository structure. Google employs a monorepo system, where over 120,000 engineers share a single repository without branches. Amazon, on the other hand, utilizes thousands of microrepos, with each service typically having its own repository.

Google’s Pre-Submit Prowess

Google’s monorepo approach necessitates a robust pre-submit testing infrastructure. With such a vast number of developers working on a single repository, the potential impact of a bad check-in is enormous. To mitigate this risk, Google has invested heavily in pre-submit testing capabilities

Ephemeral Test Environments: Google has developed multiple infrastructures to run end-to-end integration tests from local dev environments or during code reviews. These tests are conducted against ephemeral, hermetic test environments, allowing developers to catch issues before they affect the entire codebase

Smart Test Selection: Given the scale of Google’s monorepo, running all possible tests for each change is impractical. The company has invested in smart test selection and deflaking mechanisms to optimize test execution

Amazon’s Post-Submit Excellence

In contrast to Google, Amazon excels in the post-submit phase of CI/CD. The microrepo structure allows for more contained environments, enabling faster deployment to production

Rapid Deployment: Amazon’s philosophy emphasizes getting code changes to production within hours. This approach is facilitated by the microrepo structure, which allows for more independent and localized changes.

Built-in Blast Radius Reduction: The microrepo system provides an inherent mechanism for limiting the impact of bad check-ins. Issues are typically caught and blocked before they can affect other microrepos.

The Trade-offs

Both approaches come with their own set of challenges and benefits:

Google’s Challenges:

• Complex pre-submit infrastructure
• Slower post-submit deployment
• Difficulty in identifying culprits in failed deployments

Amazon’s Challenges:

• Less robust pre-submit testing
• Complexity in managing code flow between microrepos
• Potential issues with dependency version resolution

Conclusion

The debate between monorepo and microrepo approaches is ongoing, with each system having its proponents. Ultimately, both Amazon and Google have adapted their CI/CD processes to work effectively within their chosen repository structures. The key takeaway is that investing in both pre-submit and post-submit infrastructure can lead to significant improvements in developer productivity and code quality.

Read more such articles from our Newsletter here.