“Not Tested” Is Also a Result: Why QA Teams Should Not Ignore Missing Evidence

For a long time, QA teams were often asked one main question before release: “Did you test it?”

The expected answer was usually simple: “Yes, we tested it.”

But modern software delivery has changed. That answer is no longer enough.

Today, teams need to know what exactly was tested, which areas were covered, what failed, what passed, what was skipped, what changed during the release cycle, and what risk remains.

This is the shift from test execution to quality evidence. Test execution shows that testing happened.

Quality evidence helps the team make a decision.

Why “we tested it” is no longer enough

Software products are more complex than they used to be. A single user action can involve the frontend, backend, database, third-party services, payment providers, email systems, analytics tools, permissions, notifications, and mobile responsiveness.

• A small change can affect several connected flows.
• A redesign can make old test cases outdated.
• An API update can break functionality that looks unrelated.

AI-assisted development can speed up code creation, but QA teams still need to validate that the product works correctly for real users.

That is why QA results need context. When a team says “we tested it,” stakeholders may still need to ask:

• What exactly was tested?
• Which version was tested?
• Which environments were used?
• Which critical flows were covered?
• Which test cases failed?
• Which bugs are still open?
• Which areas were skipped because of time?
• What is the remaining release risk?

Without answers to these questions, the testing activity does not fully support confidence in the release.

Execution is an activity. Evidence is decision support.

Test execution is necessary. QA teams need to run tests, check features, verify fixes, explore risks, and report defects.

But execution alone is not the full value of QA.

The real value appears when testing results help the team make better decisions. Quality evidence helps answer questions like:

• Is this release ready?
• What is the biggest known risk?
• Which areas need retesting?
• Can we release with this defect?
• Do we need more time?
• Which feature is unstable?
• What should be prioritized next?

This is especially important for QA leads, product managers, project managers, developers, and stakeholders who need a clear picture of quality before release.

What QA teams need to prove in 2026

QA teams do not need to prove that they tested everything. That is not realistic.

They need to prove that testing was thoughtful, structured, and connected to product risk.

A strong QA process should make several things visible.

First, it should show scope. The team needs to know which features, flows, user roles, devices, browsers, or integrations were included in testing.

Second, it should show priority. Not all tests have the same importance. Critical user journeys, revenue-related flows, security-sensitive areas, and recently changed features usually require more attention.

Third, it should show execution results. Passed, failed, blocked, and skipped tests all tell a different story. A release with many skipped critical tests is not the same as a release with skipped low-priority checks.

Fourth, it should show defects and retesting status. It is not enough to find bugs. Teams need to know whether fixes were verified and whether regression risk was checked.

Fifth, it should show the remaining risk. No release is completely risk-free. Good QA work helps the team understand which risks are acceptable and which ones need action.

Why scattered QA evidence creates problems

Many teams still keep QA evidence across multiple places:

• test cases in spreadsheets;
• bug details in a tracker;
• screenshots in chats;
• release notes in documents;
• testing updates in Slack;
• final status in a meeting;
• context in someone’s memory.

This may work for small teams for a while. But as the product grows, this becomes harder to manage.

The team may lose track of what was tested. Test cases may become outdated. Test results may be hard to find. QA reports may require manual work every time. New team members may struggle to understand previous testing decisions.

The problem is that testing is not visible enough.

Good test management makes QA evidence easier to trust

A structured test management process helps QA teams keep testing work clear and reusable.

When test cases are organized, test runs are created for specific releases or milestones, and results are tracked consistently, the team gets a much better picture of quality.

This helps QA teams answer practical questions:

• What should we test for this release?
• Which tests are assigned?
• Which tests are completed?
• What failed?
• What still needs attention?
• What was covered in the last regression run?
• Which test cases need updates?
• What can we show to stakeholders?
• This is not only useful for QA.

It helps the whole product team work with better information.

Quality evidence matters more when teams move fast

Fast delivery creates pressure. Teams want to release quickly. Product teams want progress. Developers want feedback. Stakeholders want updates. Users expect stability.

In this environment, QA teams need to avoid two extremes.

The first extreme is testing everything equally. This is usually impossible and inefficient.

The second extreme is testing only what is obvious. This creates risk because important connected flows may be missed.

Quality evidence helps teams stay balanced.

It shows what was tested, why those areas mattered, and what the results mean for the release.

How TestCaseLab supports quality evidence

TestCaseLab helps QA teams organize test cases, test suites, test runs, milestones, and reports within a single structured workflow.

Instead of keeping testing activity scattered across documents and chats, teams can manage test cases, execute runs, track progress, and keep results visible.

This helps QA teams move from “we tested it” to a clearer message: “We tested these areas, these scenarios passed, these issues were found, these fixes were verified, and this is the remaining risk.”

That kind of visibility supports better release decisions.

QA is becoming more evidence-driven

The role of QA is not only to find bugs. QA teams help protect user experience, product stability, business logic, and release confidence.

In 2026, this means QA needs to provide more than execution. It needs to provide evidence that the team can trust.

Not perfect evidence.

Not endless documentation.

Not unnecessary bureaucracy.

Just enough structure to make quality visible. Because the most useful QA work is not only the testing that happens.

It is the testing that the team can understand, discuss, and use to make better decisions.

‍

QA reports often focus on two statuses: Passed and Failed.

Passed means the tested behavior worked as expected. Failed means something did not meet the expected result and needs attention.

But there is another status that deserves more respect in QA reporting: Not Tested.

At first glance, it may look neutral. Nothing passed. Nothing failed. Nothing happened.

But in real release planning, “Not Tested” is not empty information. It tells the team that there is no testing evidence for that area yet. And that matters.

“Not Tested” means the team lacks evidence

When a test case remains Not Tested, it does not automatically mean there is a problem with the product.

It may simply mean the team did not have enough time. The scenario may have been out of scope for the current release. The environment may not have been ready. The required test data may have been missing. Another issue may have blocked the feature.

All of these are valid reasons, but the status should still be visible.

The risk starts when Not Tested cases are ignored, hidden, or treated as if they do not matter. In that situation, the team may believe the release is covered better than it actually is.

A release report with 90 passed tests can look strong, but if 40 important tests were not executed, the picture is incomplete.

QA reporting should not only answer: “What passed?”, “What failed?”

It should also answer: “Where do we still lack evidence?”

Why Not Tested cases matter before release

Every release decision is made with some level of uncertainty. QA does not remove all risk, but it helps the team understand it. That is why Not Tested cases are important. They show where the team cannot confidently say, “This was checked.”

For example, imagine a release where the main happy path passed, but several permission-related tests were not executed. The report may still look mostly green, but the team lacks evidence that users with different roles can access only what they should.

Or imagine that payment tests passed in one region, but localization and currency-related cases were not tested. The release may work for one group of users but fail for another.

The product may still be released. Sometimes that is the right business decision. But it should be an informed decision.

There is a big difference between:

“We tested this, and it passed.”

and

“We did not test this, but we understand the risk and accept it.”

Not Tested is not the same as low priority

One common mistake is treating Not Tested cases as automatically unimportant. That is not always true.

A test case can remain Not Tested because it was low priority. Still, it can also remain Not Tested because QA ran out of time, the environment was unstable, requirements changed late, or a blocker prevented execution.

This is why QA teams should not only count Not Tested cases. They should understand why they were not executed.

The reason behind the status matters. If a low-risk visual check was not tested, the impact on the release may be small. If a critical checkout flow, permission rule, data migration, or security-related scenario was not tested, the situation is different.

Not Tested cases should be reviewed with context, not ignored.

How QA teams should work with Not Tested status

A useful QA process makes missing evidence visible.

Before release, QA leads and project teams should review Not Tested cases and ask what they mean for the current release.

The review does not need to be complicated. The team can focus on three questions:

1. Why was this test not executed? Was it out of scope, blocked, skipped because of time, or waiting for another dependency?
2. What is the risk if this area remains untested? Could it affect users, revenue, security, compliance, or a critical business flow?
3. What should happen next? Should the test be executed before release, moved to a later milestone, marked as blocked, or accepted as a known risk?

This simple review helps teams avoid false confidence.

It also makes communication clearer. Instead of saying, “Most tests passed,” QA can say, “These areas passed, these failed, these were blocked, and these were not tested, with the following risk level.”

That is a much better foundation for release decisions.

Why visibility matters in test management

When test results are scattered across spreadsheets, chat messages, tickets, and personal notes, Not Tested cases can disappear easily. This creates a reporting problem.

The team may remember defects because they are visible. Passed tests may be counted because they look positive. But Not Tested cases may receive less attention because they do not create an immediate alert.

A structured test management process helps prevent this. When test cases are organized into test runs, and each case has a clear status, QA teams can see the full picture:

🔸 Passed shows confirmed behavior.

🔸 Failed shows discovered problems.

🔸 Blocked shows where execution could not continue.

🔸 Not Tested shows where evidence is still missing.

All four statuses are useful. Together, they give the team a more honest view of release readiness.

Where TestCaseLab fits in

TestCaseLab helps QA teams keep test execution visible and organized.

Teams can create test cases, group them into suites, plan test runs, track results, and review statuses such as Passed, Failed, Blocked, and Not Tested.

This helps QA leads and teams understand not only what was checked, but also what still needs attention.

For manual testing teams, this visibility is especially important. Manual QA often deals with changing priorities, limited time, unstable environments, and last-minute release decisions. Without clear statuses, it becomes easy to lose track of what was actually tested.

With structured test runs and reporting, Not Tested cases do not disappear.

They become part of the release conversation.

‍

Why rerunning one failed test may hide a much larger reliability problem

One automated test fails.

The team reruns it. It passes.

The pipeline turns green, the release continues, and everyone moves on. Problem solved?

Probably not. 🎭

A successful rerun proves only that the failure did not happen twice in a row. It does not explain why the first failure occurred, whether the same condition affected other tests, or when it will return.

Yet flaky tests are still often handled as isolated problems: one failed script, one investigation, one retry, one ticket.

Recent research suggests that this mental model may be wrong. Flaky tests frequently fail together because they are reacting to the same unstable dependency, environment, shared state, network condition, or timing problem.

The test that caught your attention may be only the loudest member of a much larger group.

Flakiness is not always isolated

A flaky test produces different results without a relevant change to the application or test code. It may pass in one run, fail in the next, and pass again immediately afterwards.

That inconsistency makes the test difficult to trust. But the deeper problem is that teams often analyse each inconsistent result separately.

A 2025 empirical study examined 10,000 test-suite runs across 24 Java projects. The researchers identified 810 flaky tests and found that 75% belonged to a cluster of tests whose failures occurred together. The average cluster contained 13.5 flaky tests. Intermittent networking problems and unstable external dependencies were the predominant shared causes.

The researchers described this pattern as systemic flakiness.

Instead of one unreliable test having one local defect, several tests may be different symptoms of the same underlying instability.

That changes the investigation question.

Instead of asking:

Why did this test fail?

The team should also ask:

Which other tests failed under the same conditions, and what do they share?

What a flaky cluster can look like

Imagine that five automated tests fail during the same nightly run:

• a customer cannot complete checkout;
• an administrator cannot open an order;
• a report does not show the latest transaction;
• an email confirmation is not generated;
• an API test times out while retrieving the order.

Investigated separately, these failures appear unrelated. They belong to different features, pages, and test classes.

However, all five tests depend on the same order-processing service.

If that service starts slowly, loses a database connection, or fails to process a queue message on time, each test may fail differently. One produces a timeout. Another receives incomplete data. A third cannot find the expected record.

Fixing the timeout inside one test will not fix the service problem.

Increasing another test’s wait time may make the suite look healthier while allowing the same instability to remain.

This is why flaky-test analysis needs context beyond the latest error message.

A passing rerun is not a diagnosis

Retries have a legitimate purpose.

They can help a team determine whether a failure is reproducible. They can collect additional logs and provide another data point. In some cases, a controlled retry may temporarily prevent a known infrastructure issue from blocking every delivery.

The problem begins when the retry becomes the resolution.

Automatic retries can turn a red pipeline green without restoring confidence in the test suite. They can also make the cost of flakiness less visible because the pipeline eventually passes.

Over time, the team learns to interpret the first failure as noise.

That creates a dangerous habit. A real product defect may produce a failure that resembles the familiar flaky pattern. When engineers are used to clicking “rerun,” they may dismiss the signal before investigating it.

A retry should therefore answer a question, not close the problem.

For example:

• Does the test fail only on a specific worker or environment?
• Does it fail when executed with other tests but pass in isolation?
• Do several tests fail during the same runs?

The result of a rerun should become evidence for the investigation.

Look for co-occurrence

Test history becomes much more useful when teams stop viewing it as a list of individual pass and fail results.

Start by identifying which failures repeatedly appear together.

Two tests do not need to display the same error message to share a cause. One may time out while another receives incorrect data. The useful signal is that both failures begin during the same runs or under the same conditions.

Compare:

• the build and deployment version;
• environment, browser, worker, container, or device;
• test data, account, user role, and shared dependencies.

This may reveal that several apparently unrelated failures happen only in one test environment, after a specific deployment step, or while using the same external service.

A 2026 study of 649 OpenStack projects found that cross-project flakiness affected 55% of the analysed projects. The researchers identified race conditions in CI, inconsistent build configurations, and dependency mismatches as primary causes of inconsistent flakiness. The findings show how instability can extend beyond one test file or even one project.

The investigation boundary should therefore follow the shared resource, not the folder containing the failed test.

Check when the pattern started

A flaky test may have existed for months, or it may have appeared after yesterday’s infrastructure update.

Those situations require different investigations.

Compare recent runs with a known stable period. Look for the first point where the failure pattern becomes visible, then review what changed around that time.

The trigger may be:

• a dependency or runtime upgrade;
• a change in test parallelisation;
• new test data or cleanup logic;
• an altered deployment sequence;
• an API rate limit or network configuration;
• a test that now leaves shared state behind.

The test itself may not have changed. Its environment may have.

This is especially important when several tests begin failing at approximately the same time. Editing every affected test separately may create more complexity while leaving the trigger untouched.

Group by shared cause, not identical error

Teams often search for duplicate stack traces when looking for related failures.

That is useful, but it is not enough.

The same unstable dependency can produce several errors depending on where each test is when the problem occurs. A slow service might cause a timeout in one test, missing data in another, and an unexpected status code in a third.

Useful grouping dimensions include:

Shared execution context: the same environment, runner, browser, operating system, container, or time window.

Shared dependency: the same API, database, queue, file system, authentication provider, or third-party service.

Shared state: the same account, test data, cache, feature flag, transaction, or cleanup process.

The goal is not to prove immediately that the failures have one cause. It is to create a hypothesis worth testing.

Run the suspected group under controlled conditions. Change one factor at a time. Execute the tests in isolation, in a different order, on another worker, or with a stable replacement for the external dependency.

A cluster becomes actionable when the team can connect co-occurrence to a shared condition.

Do not “fix” flakiness by hiding it

Some common responses reduce visible failures without improving reliability.

Longer waits may hide a race condition. Additional retries may reduce the chance of a red pipeline. Quarantining tests may protect delivery speed. Relaxing an assertion may stop a test from failing.

Each action can be reasonable temporarily, but none should happen without a recorded explanation.

Before changing the test, ask:

1. Did the test reveal a real reliability issue in the product, infrastructure, or environment?
2. Will this change remove the cause or only make the failure less visible?
3. How will we know whether the problem returns?

Quarantine should be a managed state, not permanent exile.

A quarantined test needs an owner, a reason, evidence, and a review point. Otherwise, the suite gradually loses coverage while the dashboard becomes easier to keep green.

Different systems produce different types of flakiness

There is no universal list of flaky-test causes that applies equally to every product.

A 2026 study of fixed flaky-test issues in SAP HANA found that concurrency-related problems were the most common category in that specific system, representing 23% of the analysed issue reports. The researchers also noted that different test types face different flakiness challenges.

That distinction matters.

A browser-based end-to-end suite may struggle with asynchronous UI behaviour, external services, and dynamic selectors. API tests may be more sensitive to rate limits, shared data, or eventual consistency. Unit tests may still become flaky because of concurrency, time, randomness, global state, or inconsistent dependencies.

Teams should use general cause categories as investigation prompts, not as automatic diagnoses.

Build a useful flakiness record

Flaky-test investigations often fail because the relevant context disappears after the run.

A screenshot shows the visible error but not the environment state. A stack trace identifies where the test stopped but not what happened in another dependent service. A defect ticket says “passed after rerun” but does not record which other tests failed at the same time.

For every suspected flaky result, preserve enough information to compare it with future failures:

• build, environment, and execution time;
• error, logs, screenshots, and relevant response data;
• related failures, dependencies, and test data;
• rerun conditions and results;
• current owner and investigation status.

This does not require an elaborate observability platform before the team can begin.

Consistent test-run records are already enough to reveal basic patterns: the same tests failing together, the same environment appearing repeatedly, or a cluster beginning after a particular release.

How TestCaseLab supports the investigation

A test management tool cannot correct a race condition or stabilise an external API.

It can, however, preserve the history and context needed to recognise that several failures may belong to the same problem.

With TestCaseLab, teams can keep automated and manual test cases organised in structured test plans, execute repeatable test runs, and record the result of each check. Testers can attach evidence, document observations, connect defects, and use fields or tags to capture useful context such as the affected component, environment, platform, or risk area.

The Run Again capability allows teams to repeat an existing test run without rebuilding the same test selection manually. This is useful when reproducing a suspected flaky pattern under controlled conditions.

Reports and result history can then help the team compare runs instead of treating each failure as a new isolated event.

TestCaseLab provides the QA layer that connects test cases, execution history, evidence, defects, and investigation decisions in one structured workflow.

That shared record is especially valuable when the pattern spans several tests and requires collaboration between QA, developers, DevOps, and product teams.

Fix the system, not only the script

Flaky tests damage more than the reliability of automation.

They change team behaviour.

Engineers stop trusting failures. Testers spend time repeating executions instead of investigating risks. Pipelines consume additional resources. Releases slow down—or continue despite warnings that no one takes seriously.

Google researchers studying flaky tests across 428 projects found that effective root-cause support depends not only on detection accuracy but also on fitting investigation tools into developer workflows and making debugging information easy to use.

That is the practical lesson.

Finding a flaky test is not the end of the task. The team needs enough context to turn an unreliable signal into an understandable problem.

The next time one test fails and passes on rerun, do not ask only whether the pipeline is green again.

Ask:

What else failed with it?

When did this pattern begin?

Which environment, dependency, data source, or setup process do these tests share?

The fastest way to fix several flaky tests may be to stop treating them as several problems.

‍