Accelerating software delivery with agentic QA automation using Amazon Nova Act – Part 3

As quality assurance (QA) automation adoption grows inside an organization, a new problem appears that has nothing to do with writing tests. A platform team stands up test automation, then a second team wants to join, then a dozen more. Soon hundreds of test cases belong to different products and squads, and everyone can see everyone else’s tests. No one can answer a basic question at a glance: is this application’s test coverage healthy right now? Test authoring stops being the bottleneck. Organizing tests and controlling who can touch them becomes the work.

A second, related question follows close behind: who owns this application and its tests, and who is accountable when coverage slips? Without a clear owner, health problems have no one to land on.

In the first post , we introduced QA Studio, a reference solution built with Amazon Nova Act . Teams use it to define tests in natural language and run them through AI-powered visual navigation. In the second post , we added test suites and a CLI for batch regression testing in continuous integration and continuous delivery (CI/CD) pipelines. In this post, we cover two capabilities that help QA Studio scale across an organization. The first, applications , organizes tests and reports health metrics. The second, team-based user management , controls who can see and change what.

An application in QA Studio represents the system under test: a website, a web app, a service front end. It’s the container that gives a collection of tests, a home, and an owner. Use cases (the individual tests from Part 1) and test suites (the regression collections from Part 2) belong to an application, and they inherit its ownership. Instead of one flat list of every test in the deployment, each team works inside the applications they own.

Creating an application takes a name, a base URL, and an optional set of environments such as development, staging, and production. From the application page, teams create new use cases directly in context, add existing ones, and group them into test suites. This is also where QA Studio reports on the health of the tests it runs.

Figure 1: The application detail page, showing pass rate, executions, and failures with the use cases and test suites tabs

Tests produce results. Understanding those results over time is what tells a team whether the development lifecycle for an application is healthy, not only whether a single build is ready for release. QA Studio records the outcome of every execution as Amazon CloudWatch custom metrics, dimensioned by application and by environment. It turns these metrics into a health view on the application page.

The top of the page answers the immediate question with three figures: the pass rate over the selected window, the total number of executions, and the current failure count. Alongside them, QA Studio computes a health score of 0–100. The score starts from the pass rate and subtracts a penalty for flaky tests. That way, an application whose tests pass but flip between pass and fail doesn’t present a falsely perfect picture. A team scanning a portfolio of applications can tell the stable ones from the ones that need attention without opening each result.

The metrics tab moves from the current snapshot to trends. Teams choose a 7-day window with hourly granularity or a 30-day window with daily granularity, and QA Studio charts executions, successes, failures, and average duration across that period. Because the underlying metrics carry an environment dimension, teams filter the view to a single environment. For example, they can compare how staging behaves against production for the same set of tests.