E2E QA Automation
Summary
I designed and built E2E infrastructure from zero to reduce the manual checks that connected QA records could not remove. I automated critical workflows and built isolated k3s environments to check basic functionality at the PR stage, before deployment to the test server.
- Role
- InitiatorDeveloper (POQA)
- Outcome
- One-person QA+31% tickets verified per daypost-deploy bugs 7.0% → 3.9%
The Inefficiency of Being Tied to Repetitive Basic-Feature QA
When I joined, three POs, including me, shared feature verification. For each release version, QA alone usually took a week and sometimes two. Reviewers relied on memory and experience to choose QA checks, and manually repeating basic actions such as logging in, viewing, creating, and updating data took considerable time.
Even a change to a single feature could affect other features and workflows in different ways. Despite the time spent on QA, it was difficult to check every impact, and post-release hotfixes were frequent. Repeating basic-feature QA left too little time to examine exceptions in new and changed features. I needed to change the workflow to reduce repetitive checks and allow enough time to verify newly added or modified features.
Managing QA Work from Google Sheets to Jira
I first connected Google Sheets and Jira to manage QA work in one place. When a bug description is written in Google Sheets, Apps Script automatically calls the Jira API to create an issue containing that description along with the test case’s screen path, actions, expected results, and permissions.
Previously, whenever a bug was found during QA, I had to manually enter the feature and screen being checked, the actual result (AS-IS), and the expected result (TO-BE) in Jira. I changed the workflow so that details entered in Google Sheets were sent to Jira tickets, reducing repeated switching between the tools and duplicate entry. I also polled the Jira API to synchronize ticket status in real time and displayed content summaries in the sheet. This made it possible to check and manage the progress of tickets being handled by developers from Google Sheets alone.
I could now manage verification scope and bug-fix status in one place, but each new release still required manual QA of the same basic features. Connecting the tools reduced duplicate entry, but did not reduce the time spent checking basic functionality. To make time to review policies and exceptions in new features, I also needed to automate repetitive basic-feature QA.
Building E2E Test Infrastructure from 0→1 for Repetitive QA Automation
I automated recurring checks of basic features with E2E tests, freeing time to review policies and exceptions in new features myself.
I built E2E infrastructure to run recurring checks of basic features through test code. I scoped automation around login, viewing, saving and editing, along with essential contract, schedule and collection workflows. To select test cases, I reviewed existing test cases, past Jira bugs, changed code and connected workflows together. New policies and exceptions requiring human judgment remained part of my manual QA.
With no automation infrastructure in place, I designed the tests and their execution structure from scratch. Using Playwright and TypeScript, I built shared code for login, navigation, input, test data creation and cleanup. I connected scenario execution order, failure handling, CI runs and result sharing so the same tests could be run again for each release.
Integrating Cross-Platform User Workflows into a Single E2E Verification Flow
The platform has three parts: Admin for managing contracts, schedules, and billing, the Driver app for carrying out assigned schedules, and the Customer app used by client companies. I began implementing E2E tests with Admin and Driver, where core workflows span both platforms. Checking each screen in isolation was not enough to verify the complete workflow. I also needed to check that data created in one platform carried correctly into the next task on another.
I separated login states by role for Admin, Customer, and Driver so they could be reused across tests. For workflows spanning Admin and Driver, I used Playwright project dependencies and teardown to run setup, execution, verification, and cleanup in order. I also verified that data created in Admin carried correctly into field operations in Driver.
I also verified billing after collection in the same workflow. After checking the execution history and task list in Admin, I generated an invoice from completed tasks. The test code independently calculated expected amounts from the contract terms and work quantities, then checked them against the invoice’s pre-tax amount, VAT, and total charge. To verify the saved result, I queried the API again to check the state and amounts rather than relying only on the success message shown on screen.
A Reusable E2E Automation Environment Under the Same Conditions
At first, I ran E2E tests using existing data on the test server. If that data changed, tests could fail even when the feature itself was working correctly. To address this, I configured each run to create the contracts and schedules it needed and record their unique identifiers in a file. I added a mutex to protect the identifier file when multiple pipelines ran concurrently. I configured the tests to delete the generated data after each run and rerun the entire cleanup sequence if a test failed midway, reducing leftover data.
Expanding E2E Automation with Per-PR Isolated k3s Environments
I added E2E checks to individual feature PRs so developers could review the results and fix issues before deployment to the test server.
Running E2E tests on the test server reduced repetitive QA, but verifying a developer’s code still required merging and deploying it first. Once changes from several developers were combined, it was difficult to distinguish each change’s bugs and effects on existing features. I moved verification to the PR stage so developers could identify problems during implementation.
I created a separate verification environment for each PR and posted the results on that PR. Developers can compare a failure point with their changes to determine whether the expected result changed intentionally or an implementation error affected a related feature. They can fix bugs before deployment to the test server or update the E2E test code to reflect the intended behavior and expected results.
I built a k3s cluster on an in-house server and used Helm to deploy Frontend, Backend, and DB as containers, creating per-PR verification environments. I used a separate namespace and DB for each pipeline so that data from different PRs would not mix. I then configured Playwright to test the app for that PR instead of the test server.
Initially, building from branch names meant a new commit arriving mid-build could change the version under test. I pinned Frontend to the PR commit and Backend to the commit for the selected version, then tagged images and DB snapshots per run. This made it possible to identify the code, environment and data behind each test result.
Keeping Failure Details Available After Tests Finish
I needed to retain reports and logs so the results could still be reviewed after the tests finished. I saved the test exit code to a file and kept the container waiting until report copying was complete, so it would not exit before the result files were copied. Once copying was complete, I removed the Pod, but retained Pods with failed report copies and environments where tests had failed for diagnosis.
To analyze test results, find bugs and errors, and fix them, I needed to collect a range of logs to identify the cause of a failure. The problem could lie in the product or in the E2E test environment. To investigate the cause, I collected Playwright reports, Backend logs, Pod logs, Kubernetes event logs, and Helm status. I stored these reports and logs in GitHub Actions artifacts and posted results and links on the PR so developers could review the test results.
After downloading the reports, developers can open a web report for each platform. Selecting a failed test shows the error message, expected result, failing code line, test steps, and response results for each retry. The image below shows an actual E2E test run where schedule creation timed out because a particular menu did not appear. Through this report, developers can identify where the failure occurred and compare the expected screen with the actual behavior. They can also check Backend logs and Pod events from the affected environment to identify and fix the cause more quickly.
During tests, I needed to see how far they had progressed. Previously, CI logs did not show progress until the test Pod finished, making it difficult to tell whether a test was taking longer or had stopped. I therefore added updates every 30 seconds showing the platform under test, execution stage, elapsed time, completed test count, and retry status. Retries update the result for the same test ID so that completed tests are not counted twice.
I checked actual runs to verify the sequence from image builds through isolated deployment, test execution, report upload, and PR reporting. For schedule tests whose results depend on dates and times, I aligned browser and Backend timezones so both used the same date basis. I corrected state handling so that report preparation would not be treated as a test timeout.
Sharing Results and Failure Details through Slack and PRs
I configured results to be shared after CI runs so developers could review them and respond. E2E runs against the test server combine Admin, Customer, and Driver results and send them to Slack. Per-PR k3s verification posts results through the PR status and comment.
Slack notifications include counts of passed, failed, flaky, and skipped tests, per-platform durations, the branch and commit, and a link to the GitHub Actions run. When a test fails, its file and name, retry count, and error message are sent to the same notification thread. I made it possible for developers to quickly review the failed test and error details before opening the full logs.
+31% QA Throughput with One QA Owner, Post-deploy Bugs Down from 7.0% to 3.9%
I now handle QA on my own and complete verification before each release in about three days on average.
When I joined, three POs, including me, shared verification, and QA alone usually took a week per release and sometimes two. E2E tests now check core contract, schedule and Driver workflows in roughly an hour, reducing time spent checking the same screens manually. I can use that time to verify policies and exceptions in new or changed features, and their effects on related workflows.
The comparison covers regular releases before the introduction of E2E testing (2.13.0–2.14.2) and after (2.15.0–2.16.1). Automating basic features enabled me to verify more features, and as a result, the number of tickets handled in QA over equal-length periods increased from 712 to 992, up 39%. Average QA volume, calculated as tickets divided by actual QA days with hotfixes excluded, also increased from 22.3 to 29.2 per day, about 31% more.
Even as more tickets were verified, the post-release bug ratio fell from 7.0% to 3.9%. I used E2E tests to repeat basic-feature QA and verified new features and exceptions myself. For bugs found during QA, I tracked the developer’s fix status in Google Sheets and Jira and verified them again after the fixes. This reduced the burden on other POs of repeatedly checking basic features and helped improve product stability through pre-release basic-feature verification.
| Metric | Before E2E testing | After E2E testing |
|---|---|---|
| Verified tickets | 712 | 992 |
| Average QA volume (tickets/day) | 22.3/day | 29.2/day (up ~31%) |
| Post-release bug ratio | 7.0% (50 / 712) | 3.9% (39 / 992) |
Baseline Checks in Code, QA Focused on New Features and Exceptions
Starting with verification items written manually in Excel, I connected Google Sheets and Jira and built infrastructure to verify recurring basic features with E2E tests. I then set up per-PR isolated k3s environments to move verification to the PR stage. This allowed developers to review results and failure details on the PR and make fixes before deployment to the test server.
I now also use the RAG platform I built when preparing QA for new and changed features. I use it to find requirements, earlier bugs, and changed code, then review the material myself to define role- and state-specific checks, exceptions, and regression scope. E2E tests perform the recurring basic-feature checks, while I use browser QA to directly verify policies and complex combinations that automation alone cannot assess.