Is 100% Test Automation Really a Good Idea?

The idea sounds perfect

Picture a release day where nobody clicks through the same screens again. Every regression check runs on its own. The pipeline goes green and the build ships.

It's easy to see why teams want this. Automated checks run fast and run the same way every time. They remove a lot of boring, repetitive work. They give developers feedback in minutes instead of days, and they make frequent releases possible.

So "100% test automation" sounds like the finish line. Automate everything, and quality takes care of itself.

But I keep coming back to one question: does having more automated tests actually mean we have better QA?

I don't think it does, at least not automatically. That's what I want to think through here.

I'm not arguing against automation. I'm arguing against treating the percentage as the goal.

What does "100% automation" actually mean?

When someone says "we want 100% automation," they can mean very different things:

• Every regression test is automated.

• Every documented test case is automated.

• All critical user journeys are automated.

• Manual regression has almost disappeared.

These are not the same goal. The first might be reasonable. The second might mean automating a lot of old test cases nobody has looked at in years. The fourth might be a healthy outcome or a warning sign, depending on what replaced the manual work.

There's one distinction I think matters more than the others: 100% automated testing is not 100% product coverage.

Automation checks what we told it to check. If nobody thought of a scenario, no script covers it. A fully automated suite can be completely green while the product has a serious problem in a place nobody looked.

What QA people are debating right now

One thing I keep seeing in QA discussions is that people talk past each other. It usually isn't "automation vs manual." It's closer to "how much, and where?"

The pro-automation view is strong. Manual regression is expensive and slow, and it doesn't scale as the product grows. Automation gives faster feedback. Many QA engineers also point out that automation skills are now expected in the job market, so avoiding them is a risky career choice.

The cautious view is just as reasonable. Don't automate everything simply because you can. Big suites get hard to maintain. Flaky tests create noise. The number of tests can turn into a vanity metric. And when the team is busy keeping scripts alive, exploratory testing is often the first thing to get squeezed out.

I don't think either side is wrong. Test automation costs time, tooling and people to build and maintain. The real disagreement is about where automation gives enough value to justify that cost.

Even among automation people, this isn't a new argument. A well-known Microsoft engineering blog post from years ago pushed back on the "100% automation" mantra. The author argued that scripted automation does a poor job of finding new defects after the tests are designed, and that GUI automation can be costly to maintain when it's built before the interface settles. The same post also said that some areas, like baseline and build-verification suites, are good candidates for very high levels of automation. So even a critic was saying "be smart about it," not "don't do it." microsoft

Vendor content makes similar points. For example, ACCELQ argues that aiming for 100% automation can drain budget and doesn't produce bug-free software. That's a vendor view, so read it with that in mind, but it matches what many practitioners say.

The "1,000 automated tests" problem

This is the part I think gets skipped most often.

Imagine a team has 1,000 automated tests. That sounds great. It sounds mature.

Now ask some questions:

• How many are still relevant to the product as it is today?

• How many are flaky?

• How many are duplicates, testing the same thing in slightly different ways?

• How many break every time the UI changes?

• How many actually cover an important business risk?

• When one fails, does anyone investigate, or does someone just re-run the build?

• How long does the whole suite take to run?

I'm not saying any specific number is bad. I'm saying the headline number tells you almost nothing without these answers.

Flaky tests are a good example because there's real data. Google has written publicly about this. Across their tests, about 1.5% of test runs gave a flaky result, and almost 16% of tests had some level of flakiness. They also noted that it's common to ignore real failures in flaky tests because there are so many false alarms. A separate Google analysis found that larger tests were far more likely to be flaky: about 0.5% of small tests, 1.6% of medium tests and 14% of large tests.

That's Google's environment, not yours, so I wouldn't apply those numbers directly to any other team. But the pattern is worth noticing. Once people stop trusting failures, the suite stops doing its job.

Test count is not quality. A smaller suite of reliable, meaningful tests can be worth more than a huge one nobody trusts.

Where automation makes a lot of sense

I want to give automation proper credit, because in the right places it's excellent.

Repetitive work. If the same check runs hundreds of times, automate it. Doing it by hand every time is a waste of a skilled person's attention.

Regression. Anything you need to re-verify after every release is a strong candidate. This is the classic use case for good reason.

Critical flows. Login, checkout, payment, order creation and key API workflows. If these break, everyone notices, so you want to know within minutes.

Fast feedback. When automated checks run on every build, developers find out about a regression while the change is still fresh in their heads. That's hard to get any other way.

Stable checks. Functionality that doesn't change much is easier to automate and cheaper to maintain.

Other good fits include API testing, data validation, cross-browser checks and high-volume testing. In these areas, automation does things humans can't do at the same speed or scale.

What I don't want to hand completely to automation

Some work I wouldn't want to hand over completely:

• Exploratory testing. Learning, designing tests and running them all at the same time.

• New features. Especially when behavior is still being figured out.

• Usability and UX. Whether something is confusing or annoying doesn't show up as a failed assertion.

• Unexpected user behavior. People use products in ways nobody planned for.

• Changing requirements. Scripts written against moving targets get stale fast.

• Strange combinations of actions. The odd sequence a real user stumbles into.

• Business logic questions. "Is this actually the right behavior?" is different from "does this do what the spec says?"

• Problems nobody thought to write a test for.

I want to be careful here. I'm not saying automation can never handle edge cases or unexpected paths. People are doing interesting work with randomized and generated testing. The question is whether automation is the best tool for discovering these things, as opposed to checking them once we know about them.

Here's how I'd put it: automation is excellent at checking known scenarios again and again. Human testers can also explore unknown scenarios and challenge assumptions.

On exploratory testing specifically, the general description is that it combines learning, test design and execution, and that it depends heavily on the tester's domain knowledge. That domain knowledge is exactly what's hard to script. Wikipedia

The real mistake: automating the wrong things

The most common mistake I see is a straight line from:

Requirement → Test case → Automation

with nobody stopping to ask, "Is this actually worth automating?"

A few examples. None of these are absolute rules.

• A test that runs on every build is a strong candidate. The effort pays back quickly.

• A test for a UI that changes every few days may cost more to maintain than it saves.

• A one-time investigation probably isn't worth automating. You'd spend more building the script than you'd get back.

• A critical payment workflow is a strong candidate. The cost of missing a break is high.

• A brand-new feature whose behavior is still changing may be better served by exploratory testing first. Build the large automation suite once the behavior settles.

The Microsoft post I mentioned earlier cites an estimate that an automated test may need to run roughly 15 to 17 times to break even on the cost of building it. Treat that as a rough rule of thumb, not a law. But the idea behind it is useful: every automated test has a cost, and it has to earn it back.

What QA teams are actually doing instead

Here's the process I see sensible teams moving toward. It isn't a formal framework. It's closer to common sense.

1. Identify the risk. What could seriously hurt users or the business? Start there.

2. Find the repetitive checks. What has to be tested again and again?

3. Automate valuable, stable checks. Don't automate to push the test count up.

4. Keep exploratory testing. Give testers real time to investigate the areas scripts won't reach.

5. Review the suite regularly. Remove obsolete tests, duplicates, consistently unreliable tests and low-value tests. A suite needs pruning like a garden does.

6. Investigate failures. A red test should start a question, not just a re-run until it turns green.

7. Measure useful outcomes. Instead of counting tests, look at meaningful coverage, defect detection, feedback speed, reliability, maintenance effort and release confidence.

None of this is exciting. But it works better than chasing a percentage.

Then AI enters the conversation

This part is moving fast, and a lot of QA engineers are experimenting with it. People are using AI to:

• generate test cases

• write automation code, including Playwright and Selenium tests

• create test data

• suggest edge cases

• prioritize which tests to run

• help analyze failures

I find this useful. I also find it raises a new version of the old problem.

AI can make it very easy to create thousands of automated tests.

But creating more tests doesn't automatically mean creating better ones. Some commentators warn that large AI-generated suites can create an illusion of completeness and false confidence if human review and risk analysis are reduced. Another industry piece describes teams shipping more tests while spending more time fixing them, and argues that the useful measure is signal-to-noise ratio, not how many tests ran.

There's some early research too. One small Thoughtworks experiment, with three QA engineers and nine user stories, compared AI-generated test cases against manual ones. It found strong performance on basic scenarios, but ambiguity in about 27% of the generated cases, so it recommended structured human review before using them. It's a small study, so I wouldn't generalize from it. But it fits what many people report in practice: AI is good at the first draft, and a human still has to judge it.

So here's the question I think we should be asking:

If AI can generate 10,000 tests, should we actually run all 10,000?

I think the conversation is shifting. It started as "Can we automate this?" Now it's more like "Should we automate this?" And I suspect the next version is "Is this test valuable enough to run at all?"

What this means for QA engineers

I don't want to make big predictions about jobs. Nobody knows exactly how this plays out.

What I can say is that the most useful QA engineers I can picture have both a testing mindset and automation skills. That usually includes:

• manual and exploratory testing

• test automation

• API testing

• some programming

• CI/CD basics

• debugging

• product understanding

• risk-based thinking

• the ability to work with AI tools critically

I wouldn't tell anyone to drop manual testing. Good manual and exploratory work is what tells you what is worth automating in the first place.

Modern QA is getting broader. It isn't simply "manual" or "automation" anymore.

How I would approach automation

If I were building a test strategy today, I wouldn't start by saying we need 80% or 100% automation.

I'd start with questions:

1. What are the biggest risks?

2. What breaks most often?

3. What do we test over and over?

4. What needs fast feedback?

5. Which tests are stable enough to automate without constant repair?

6. Where do we still need a person to explore?

7. Which of our existing automated tests aren't giving us value anymore?

The answers would shape the strategy. And I'd expect the answers to change over time, which means the strategy needs a regular review too.

Conclusion

I don't think 100% automation should be the finish line.

The goal of QA isn't to produce the largest possible number of automated tests. The goal is to help the team find important problems, reduce risk, get useful feedback quickly and release with confidence.

Automation is one of the best tools we have for that. But it's still a tool, and using more of a tool doesn't automatically mean you're using it better.

So here's what I'd like you to think about:

If your team had 100% automated test coverage tomorrow, what would you still want a QA engineer to test manually?