A test suite is a set of claims about the system. Until the claims are checked against a specific build, they are intentions. TestExecute is where they become evidence.
The design decision worth explaining is how results are categorised, because most tools get it wrong in a way that quietly corrupts every number downstream.
Four results, not two
A result is one of: not executed, pass, fail, or blocked.
Most reporting collapses this into passed and failed, and pushes everything else into one of them. That collapse is where the damage happens.
- Not executed means nobody has checked yet. It is not a pass and it is not a fail. Counting it as either invents information.
- Pass means the claim held against this build.
- Fail means the claim did not hold. This is a statement about the product.
- Blocked means the test could not run. This is a statement about the environment, not the product.
Treating blocked as a failure inflates the defect picture and sends people hunting for a bug that does not exist. Treating it as a pass is worse: it reports coverage that was never gathered. Keeping it separate is the only honest option, and it makes the blocked count a useful signal in its own right, because a run with thirty blocked tests is telling you something about your environment that no pass rate would.
A pass rate calculated over executed tests only. Ninety percent of forty executed tests, out of a suite of two hundred, is not ninety percent confidence in anything. Always read the pass rate next to what was never run.
Blocked requires a reason
Marking a test blocked is not a single click. It requires a category and notes, and both are mandatory.
The categories are the ones that actually recur: environment unavailable, dependency not ready, test data missing, prerequisite failed, build not deployed, and other.
Requiring the category is a deliberate friction. Blocked without a reason is a black hole in the record: three weeks later nobody can tell whether the environment was down for an hour or the feature was never testable at all. With categories, a pattern across runs becomes visible, and "test data missing" appearing in every cycle is a process problem worth fixing rather than a recurring inconvenience people work around.
A run is scoped to a build
A run carries a name, an environment, and a build version, and it is either in progress or completed.
The build version is the part people skip and the part that matters most. A passing result without a build reference cannot answer the only question that counts at release time, which is whether the evidence applies to the thing you are about to ship. Evidence from three builds ago is not evidence about this one.
Evidence attached to results
Results can carry evidence: a note, or a URL pointing at whatever supports the finding.
This matters most on failures. A failure recorded as "did not work" costs a developer an hour reproducing it and frequently ends in "cannot reproduce", which is a defect resolution that means nobody learned anything. A failure with the conditions and a link is a defect somebody can act on immediately.
From execution to defect
A failing result is not automatically a defect. The judgement about whether a failure represents a product problem, a test problem, or an environment problem belongs to the person who ran it.
When it is a real defect, flagging it from execution carries the context across: which test, which run, which suite, and what failed. That linkage is what lets a coverage report answer what a failing test actually costs rather than just how many are red.
Key takeaways
- Four results, not two, because blocked is a statement about the environment and not about the product.
- Blocked requires a category and notes, which turns a black hole in the record into a pattern you can act on.
- A run without a build version cannot support a release decision.
- A failing result becomes a defect by human judgement, not automatically.
Reading a completed run
Look in this order, which is roughly inverse to how dashboards present it.
- What was not executed. The gap is the risk, and it is invisible in a pass rate.
- What was blocked, and why. If a category dominates, that is your next process fix.
- What failed, and on which paths. Two failures on the payment flow matter more than nine on a settings screen.
- The pass rate. Last, and only in the context of the three above.