The Pentest you Bought May Not be a Pentest at All

As penetration tests have become a standard line item for compliance and a strong, defensible way to demonstrate security posture, the demand for these services has exploded. So has the number of providers willing to sell them.

The problem is that not everything being sold as a “penetration test” actually involves penetration testing. Some engagements, offered at a fraction of the cost of a genuine assessment, amount to little more than an automated vulnerability scan, an unsupervised AI agent run, or a collection of tool output poured into a report template.

So what separates a real penetration test from an imitation? What does the low-cost, automated version miss? And how can you tell which one you bought?

Read more: Red Team vs. Pentesting: What’s the difference and why it matters for your business

‍

Not All Pentests are Created Equal

Low-cost “pentests” can take several forms. The technology and level of human involvement may differ, but they share a fundamental gap: insufficient depth, coverage, or accountable expertise driving the assessment.

1. The Relabeled Vulnerability Scan

A scanner checks your assets against predefined rules and known vulnerability patterns, then exports a list of findings. Scanning is valuable for continuous monitoring, but a scan alone doesn’t establish the depth of a penetration test, nor does it determine what is exploitable. It can miss exploitable weaknesses, particularly those that depend on how your business logic works.

2. The Unsupervised AI Agent 

AI agents can perform reconnaissance, test for common vulnerabilities, and attempt exploitation. However, without expert oversight, they may report false positives, miss vulnerabilities that require business context, disrupt live systems, or access data outside the authorized scope.

3. The Templated Tool Report

Findings copied from automated tools, severity ratings taken from public databases, and generic remediation advice dropped into a polished template. It looks thorough, but the underlying system was never actually attacked. There’s no manual validation, no attempt to chain weaknesses together, and no evidence that anyone tested whether the reported issues could actually be exploited in the context of your application. You get a document that looks like a penetration test, without the actual adversarial testing that makes a penetration test valuable.

Read more: Pentesting or Vulnerability Scanning: Which should you choose?

‍

What a Real Pentest Delivers

A real penetration test is a controlled attack carried out by experts who think like the adversary. It answers three questions leadership actually cares about: what could an attacker reach, what impact could it have on the business, and what should be fixed first?

Getting there takes actual judgment and expertise that no tool supplies on its own. An expert-led engagement includes:

• Scoping and threat modeling: Identifying what matters most to your business and where an attacker would realistically focus.
• Understanding the application: Learning user roles, workflows, and intended behavior, then determining which behaviors are intentionally allowed, unintentionally exposed, or simply insecure by design.
• Manual validation: Validating findings with evidence from your environment and clearly stating any limits on exploitation
• Attack chaining: Combining individually minor weaknesses into a realistic path to compromise.
• Contextual risk rating: Scoring severity based on your environment, not a generic database entry.
• Actionable reporting: Step-by-step reproduction, business impact, and prioritized remediation guidance.
• Retesting: Confirmation that fixes actually resolve the issue. 

Just as important, a real pentest documents which assets and controls, what resisted the attempted attacks, and where testing was limited. Knowing where you are strong is half the value of the exercise.

The Gaps that Only Hands-on Testing Reveals

Some serious flaws are difficult for tools to identify because they depend on understanding what your application is for.

• Revenue: A checkout that lets a customer change the price, claim multiple refunds for the same purchase or skip a payment entirely.
• Customer data: One customer reads another’s records by manipulating values in a request. Broken object-level authorization ranks first in the OWASP API security top 10 (2023).
• Chained attacks: Several minor issues that together hand an attacker full control of an account or system. Rated one at a time, each may appear low-risk in isolation.
• Context: The same weakness is trivial on a marketing page and critical in front of payment data. Tool output alone may not capture the business context. An expert can.

The real cost of a superficial test goes beyond the engagement itself. It can create false assurance, leading leadership to make decisions based on a report that never tested what actually matters. It can also waste engineering time as teams spend hours triaging unverified findings while genuine risks remain undiscovered. Finally, when a customer, auditor, or security incident exposes the gap, you may end up paying for a proper test anyway, often under much greater time pressure.

This isn’t About AI and Automation

Automation has never been the problem – in fact, strong pentest teams use scanners and AI agents every day to cover ground quickly and catch common issues reliably.

Where the issue lies is who is in charge of the process. An expert directs the tools, checks every result manually and spends their time on what tools cannot do: understanding your business, finding the flaws that live in its logic, and judging what each one means for you.

What this breaks down to is that you aren’t paying for the tools. You’re paying for the judgment that turns their output into an answer.

Five Questions to Ask Your Vendor

If you’re currently in the market for a high quality pentest, make sure you do your due diligence - ask these questions of each potential vendor before you sign, and pay close attention to their answers (or, in some cases, non-answers). If you’re already working with a pentester, hold your last report up against them:

1. Who will do the testing, and what qualifies them?
2. How are automated and AI-generated findings validated, and by whom?
3. Will you test our business logic and access controls with our real user roles?
4. Will the report show how you attacked us, what held up, and what to fix first?
5. Is a retest included to confirm our fixes?

If a vendor cannot answer clearly, you have a reason to question the depth of the assessment. A pentest is only worth what it tells you about your real risk. Before you file the next report as done, make sure someone actually tried to break in.

‍