Pick what you look at
Three controls sit at the top of the page. Each choice is kept in the page’s link, so you can share exactly the view you see.
In Release mode a fourth control, Compare with, picks the release to compare with. See Release mode.
A line under the controls says what the numbers count.
Which tests
- A folder or a tag holds the test cases in it today. A test case moved to another folder yesterday counts under its new folder, with all its runs, including the ones from before the move.
- A team holds the test cases it owns today: a test case’s own team, or else the team of its folder or the nearest folder above it. The list shows every team in your organization, so a team that owns nothing in this project shows no test cases.
- A selection counts differently in each mode. In Period mode it counts only the runs it launched: a test case run directly, or by another selection, is left out. In Release mode it holds the test cases in the selection today.
- Archived test cases are left out, as on the Test cases page.
Break down by
Camera model is listed but not available yet: runs don’t record a camera model.
Period or release
The two modes count different things.Period mode
Pick a window: 7 days, 30 days, 3 months, 1 year, or Custom range. A custom range is up to 366 days and ends today or earlier. Days start and end at midnight in your time zone, so a teammate in another time zone can see a run near midnight on a different day.What counts
- A run is an attempt and its retries. Its result is the result of its last attempt. Its first try is its first attempt.
- Pass rate is passed runs divided by passed and failed runs. A timeout is a failure.
- Environment errors, cancelled and skipped runs are shown but not counted. They say nothing about whether the test passes.
- Every run counts, whoever started it: Peeps, your CI, a person recording a manual test, or results you uploaded.
The summary cards
Newly failing, newly flaky and fixed
These three cards compare test cases, not runs, and only test cases that have a counted run in both periods. A test added this period that fails from its first run isn’t newly failing, and one that stopped running isn’t fixed. When no test case ran in both periods, the three cards show — and No comparable results. A test case is flaky in a period when either of these happened:- A retry cleared a test failure. A run failed and then passed on a retry. A retry that clears an environment error doesn’t count.
- Its results flipped twice. Its results switch between passed and failed at least twice on the same configuration, such as passed, failed, passed. For runs from your CI, the same Playwright project counts as the same configuration.
A test case flaky now is neither newly failing nor fixed, so no test case counts in two cards.
The breakdown
The section under the cards splits the view by your Break down by choice. Each row shows its pass rate, how many runs (or, in Release mode, results) it counted, how many failed, its Cases with a failure, and its Change against the period or release compared with. In Period mode, except by cause, a row also shows its Flaky cases and a Trend: one pass rate per day or week of the period.Worst first
Broken down by Folder or Team, the rows come worst first:- The most test cases with a failure.
- Then the lowest pass rate.
- Then the most flaky test cases.
- Then the name.
Open a folder or team from its row
Click a folder’s or a team’s row to open it: Which tests changes to that folder or team, and the view keeps your runs, your period or release, and your breakdown. The numbers you then see are the row’s. Back returns you to the list, and you can share the link or open it in a new tab. A team’s row opens only when Which tests is All test cases. Under a folder, tag or selection, a team’s row counts only that part of the team’s test cases. These rows don’t open:- The row for the folder’s own test cases, since that folder is the view you’re looking at.
- No folder and No team.
- Folder rows under a tag, a selection or a team, since the folder holds more test cases than the row counts.
Why they failed
Why they failed lists the failures after retries by cause, largest first, and who each cause goes to:
A failure Peeps hasn’t analyzed is listed as Not analyzed. A failure a person recorded in a manual test run is Failed (manual).
App bugs by owning team counts the app bugs by the team that owns each test case today, with how many runs failed, how many test cases they belong to, and how many of those analyses were low confidence. Test cases with no team are under No team.
The Issues panel
The Issues panel, beside Why they failed, lists the view’s failures as issues: failure groups from Peeps’ analysis, each with its issue in your tracker and how many failures it holds. Above the list it counts the issues by where they stand in your tracker, such as open, done, or not tracked.- In Period mode, an issue is runs that ended failed with similar failure text, so a failure a retry cleared isn’t an issue. An issue opens its latest failed run, where you can create a tracker issue.
- In Release mode, an issue is one of the release report’s failure groups. An issue whose tracker issue is done but that failed again reads done, failed since; one with no failure since reads done, needs retest. An issue opens its page in the release.
The test cases table
Test cases, least stable first lists each test case: the lowest pass rate first, then the most failed, then the most saved by a retry. Its last column is the test case’s issue in your tracker, with +N when it has more, or Not tracked when a failing test case has none. A test case that didn’t fail shows —. In Release mode a Fails on column names the configurations a test case fails on.Release mode
Release mode shows one release, counted the way its release report counts it: each test’s latest result per configuration. So the page, the release report and the Releases list show one pass rate for a release. The pass rate is a whole percent.- Errors count as failures. Skipped, blocked, retest and N/A results are shown but not counted.
- No first try and no flaky tests. A latest result has no retries behind it.
- The release is the one you pick. With none picked, the page opens on the newest release that has results. A release created with no results yet is listed as No results yet.
- Compare with starts on As the release report does: the nearest earlier release that shares a result with this one. Pick another release to compare with it instead. When the two releases share no result, no change is shown.
- Which tests picks within the release’s latest results. Results not linked to a test case are left out of every count, and the Not linked to a test case card counts them when Which tests is All test cases.