Salesforce’s Leap to Automated End-to-End Testing

In today’s rapidly evolving digital landscape, software has become the backbone of modern life. Companies like Salesforce, a leader in software and SaaS solutions, rely on swift innovation to maintain their competitive edge and address emerging markets. This trend extends beyond the tech industry, as businesses across various sectors increasingly define their products and services through software.

The Rising Tide of Consumer Expectations

The democratization of information and global access to goods and services has leveled the playing field for businesses of all sizes. This shift has created new service expectations among consumers, who now gravitate towards providers offering speed, innovation, or price advantages. These expectations permeate into professional environments, influencing how individuals interact with partners and suppliers.

As consumer demands evolve, the pressure to innovate through software intensifies. Fast iteration has emerged as the key to successful innovation, involving rapid releases, feedback assessment, and agile course corrections. To achieve this accelerated pace, all phases of the software development lifecycle must move in tandem, with testing playing a crucial role in preventing bottlenecks.

Salesforce’s Community Cloud: A Testament to Scale

Salesforce’s Community Cloud stands as a testament to the company’s massive reach and influence. With over 500 million users, more than 40 sites published daily, and approximately 50,000 live customer sites, the platform’s performance is critical. Any bug, functionality issue, or downtime can have immediate and far-reaching consequences on customer satisfaction.

The Agile Testing Quadrants Model

At Community Cloud, the team adheres to the Agile testing quadrants model, initially introduced by Brian Marick and later refined by Lisa Crispin and Janet Gregory. This model divides testing into four quadrants, helping teams plan various test types and allocate necessary resources.

While the Community Cloud team had successfully automated thousands of tests across Quadrants 1, 2, and 4, they relied on manual testing for Quadrant 3. Their goal was to extend automation to cover key scenarios within this quadrant, particularly focusing on end-to-end (E2E) UI functional testing.

Challenges in Automating Q3 Testing

E2E UI functional testing represents the ultimate test of an application’s real-world performance. These tests serve as the last line of defense against bugs reaching production and impacting user experiences. However, they are also the most challenging to automate due to their comprehensive nature, the need to cover frequently changing applications, and the inherent unknowns of real-world environments.

The Community Cloud team faced several challenges with their existing Selenium-based automation framework:

1. Unpredictable results from thousands of daily tests
2. Difficulty in obtaining a high-level view of critical use cases
3. Flaky tests that broke with minor code changes
4. Time-consuming troubleshooting and test maintenance

The Quest for Improved Test Automation

To address these challenges, the Community Cloud team established specific criteria for their ideal test automation solution:

1. Test Stability: E2E tests needed to focus on software functionality rather than test maintenance.
2. Ease of Use: The solution had to accommodate various user types, including QA Automation Engineers, Software Engineers, and non-technical users.
3. Fast Authoring: Test creation needed to keep pace with the team’s rapid release schedule.
4. Completeness of Function: The solution had to support all application features, desired browsers, and integrate seamlessly into existing pipelines.

Selenium vs. Testim: A Comparative Analysis

The team conducted a proof of concept comparing six extensive E2E tests in both Selenium and Testim environments. Testim, an AI-based test automation startup, offered a Chrome extension for recording functional and GUI end-to-end tests, along with AI-based Smart Locators for improved element identification.

The comparison revealed significant advantages of Testim over Selenium:

Embracing Testim: Solution and Benefits

Based on the comparative analysis, the Community Cloud team decided to implement Testim. This decision marked a paradigm shift for Salesforce, a company traditionally focused on coded automation. The transition to a codeless automation tool like Testim required some convincing, but users quickly adapted and recognized its value.

The implementation of Testim yielded several significant benefits:

1. Successful automation of complex E2E tests in Q3
2. Increased detection of major P1 bugs (nearly 50%)
3. Quick onboarding of both technical and non-technical users
4. Reduced time spent on test maintenance
5. Centralized view of critical path testing status
6. Significant reduction in automation planning time

Conclusion: A New Era of End-to-End Testing

The Community Cloud team’s journey to improve end-to-end testing led them to discover a more efficient and effective solution in Testim. By outperforming Selenium in key areas such as test stability, troubleshooting, environment stability, and usability, Testim has proven to be a valuable asset in Salesforce’s quest for accelerated releases, increased quality, and reduced costs.

As Salesforce continues to evaluate and roll out Testim to additional cloud teams, the company is poised to maintain its position at the forefront of software innovation and quality assurance. This success story serves as a testament to the power of embracing new technologies and methodologies in the ever-evolving landscape of software development and testing.

Read more about the topic here.

Read more such articles from our newsletter here.