Rules behind the numbers
The rules that several BesTest reports share: finished results, the newest result, the retest rule, day bands, archived items, time zones and rounding.
Several reports use the same rules. This page states each rule once. Each report page links here.
Finished, open and to run
| Word | Meaning | Where |
|---|---|---|
| Finished | The result is Passed, Failed, Blocked or Skipped. | Every report except Test Execution Time |
| Finished in Test Execution Time | The latest run is Passed, Failed or Skipped. A Blocked test case is work left. | Test Execution Time |
| Open | The run is In Progress or Not Executed. | Every report |
| To run | A test case has no finished result, or its newest result is Skipped. | Requirement Coverage and Risk, Work Item Traceability, Traceability Matrix |
| Retest result | Passed, Failed or Blocked. Skipped is no retest. | Defect Analysis, Test Execution Results, Test Execution Explorer, Test Execution Time |
The newest result
A test case can have many runs: reruns in one cycle, and runs in many cycles. Each report says which run counts.
- The latest run in a cycle. In one cycle, the runs of a test case form a chain: a rerun follows the run that it replaces. The last run of the chain is the latest run in that cycle, also when it has not run yet. Test Execution Results ("in cycles", the cycle rows) and the Test Execution Explorer (Latest result) use this run.
- Latest result in Test Execution Results. One run per test case over all the picked cycles: the finished run with the latest run time. A test case with no finished run counts by its open run in the newest cycle.
- The result of a test case in Requirement Coverage and Risk, Work Item Traceability and the Traceability Matrix. In each cycle, the newest finished run of the chain counts: a rerun wins over the run that it replaces. Over the cycles, the finished run with the latest run time counts. A newer run that is not finished never replaces it. It shows as a clock mark, "waits".
Runs between
"Runs that finished on these days count, both days included. Runs that are not finished yet always count."
The presets are Any day, Last 7 days (today and the 6 days before), Last 30 days (today and the 29 days before), This month (from the 1st to today), Last month and Custom days. A data set keeps a preset as a preset, so "Last 7 days" moves with the calendar.
Test Execution Results, the Test Execution Explorer and Defect Analysis read Runs between. Test Execution Time has no date range. Requirement Coverage and Risk, Work Item Traceability and the Traceability Matrix read cycles, not a date range.
The retest rule
Defect Analysis, Test Execution Results ("Still fails") and the Test Execution Explorer ("Defect retests") use one rule. Test Execution Time uses it for "ran again".
A retest of a defect on a test case is a run of that test case that does not carry the defect, with the result Passed, Failed or Blocked, on a day after the last run of the defect on that test case. For a fixed defect, the run must also be on or after the day Jira resolved it. Skipped is no retest. A run that did not pass and has another defect linked is that defect's failure, not a retest.
The newest retest decides. Pass, then fail: still fails. Pass, fail, pass: passes. The first retest counts only for "Median from the fix to the first run".
A run of the defect with no day counts as today. A finished run with no run time is never a retest.
Example
A run finds a defect on Monday. Jira resolves it on Wednesday. The test case runs again on Thursday: Passed. It runs on Friday: Failed, with no other defect linked. The verdict is "Fixed, but still fails". When it runs on the next Monday and passes, the verdict is "Fixed and passed on retest".
A defect over several test cases: any test case that did not pass wins, then any with no retest, then passed. Done with a resolution in "Closed without a fix" is "Closed, not fixed", with no retest. See the verdict table on Defect Analysis.
Resolution groups
A Done defect is "Fixed" or "Closed without a fix", by its Jira resolution ID, never by its name. By default, Jira's Won't Do and Duplicate (and the older Won't Fix, Duplicate, Incomplete and Cannot Reproduce) are "Closed without a fix". Done with no resolution counts as fixed. Only Jira admins and space admins can change the groups, and they are the same in every space of the site. Defect Analysis, the Test Execution Explorer and Test Execution Results read them.
Day bands
| Band | Days |
|---|---|
| 0-7 | 0 to 7 |
| 8-30 | 8 to 30 |
| 31-90 | 31 to 90 |
| 90+ | over 90 |
Each upper edge is in its band: 7 days is "0-7". Defect Analysis uses the bands for the days open. Requirement Coverage and Risk uses them for How old is the evidence.
Archived items
- The execution numbers count the runs of a test case that was archived later: Test Execution Results, the Test Execution Explorer, Defect Analysis and Test Execution Time.
- The coverage numbers leave archived requirements and archived test cases out: Requirement Coverage and Risk, Work Item Traceability and the Traceability Matrix.
Days and time zones
- A day is a calendar day in your day zone: the time zone of your Atlassian profile. With no zone in the profile, it is UTC. It is never the zone of your browser.
- Working days (Test Execution Time only) are Monday to Friday. The report does not know holidays.
- A week starts on Monday.
Rounding
- A rate, for example Pass Rate or Completion, rounds to the nearest whole percent, with two exceptions. A rate that is not exactly 100% never shows 100%: 199 of 200 is 99%. A rate that is not exactly 0% never shows 0%: 1 of 100,000 is 1%. With nothing to divide by, there is no rate. It never shows as 0%.
- The shares of one whole, for example the slices of Defects by priority in Test Execution Results or of Test results in Work Item Traceability, add up to 100%. A part above 0 never shows "0%": it shows
<1%. A part that is not the whole never shows "100%". - Hours in Test Execution Time are whole hours on screen that add up to the total on screen. Excel has the exact values.
The 100-cycle cap
A source that holds more than 100 cycles keeps the 100 newest cycles, by the day they were made, and the report says how many it left out. Requirement Coverage and Risk, Work Item Traceability and the Traceability Matrix cut only picked cycles: Every cycle has no cap.
When Jira does not answer
The parts that come from the BesTest database show at once. The numbers that come from Jira, for example the status of a defect, show a placeholder, then "Could not read the defects from Jira." with Retry. A number that the report could not read never shows as 0.
A click on a number
A click on a number or on a part of a chart shows the rows behind it. In the "⋯" menu of the top line, Display sets what a click does:
- Open a list dialog: "Shows the rows in a dialog over the report". This is the default.
- Filter the table: "Sets the table filters and shows the table".
The setting "applies to every report. Saved in this browser."
Limits
A report that reads more than its limit shows no numbers and says why, with a hint to narrow the sources. It never cuts a list without a notice. Each report page lists its limits.