After Mythos: Why Bug Bounty Programs Need Harder Evidence Now

A few weeks ago I wrote about Anthropic Mythos and Project Glasswing. Back then the point was mostly the big picture: if models really become better at finding old vulnerabilities, combining exploit paths, and understanding whole codebases, vulnerability research changes.

The new Sophos article about bug bounties in the Mythos era now shows the operational side of that shift.

It is no longer only about whether AI finds better vulnerabilities. It is about whether bug bounty programs, security teams, and engineering organizations can still distinguish fast enough between junk, plausible claims, and real security issues.

The real problem is not only AI slop

When people talk about AI and bug bounties right now, they quickly think of slop: automatically generated reports, half-understood error messages, invented exploit chains, non-reproducible claims, and a lot of text with very little substance.

That is real. And for maintainers, product security teams, and triage people, it is brutally annoying.

But it is only one side of the story.

The other side is more dangerous: good researchers can use the same models to find real vulnerabilities faster, test hypotheses across larger codebases, and systematically walk through variants of a pattern. What used to take days or weeks of manual work may, in the future, land in the queue within hours.

That shifts the problem. In the past the question was often: are we getting enough good external findings? Today the question is becoming: can we recognize, validate, prioritize, and turn the real findings into fixes quickly enough while the noise rises at the same time?

Why Sophos is an interesting source here

The Sophos article is not just a generic comment about AI. Sophos looks back at its own bug bounty program and gives concrete numbers.

According to Sophos, its public program has been running on Bugcrowd since December 2017. Sophos writes that, by the time of the article, 1,343 vulnerabilities had been rewarded, across 7,091 total submissions, with a total payout of 599,695 US dollars.

For 2025, Sophos lists, among other things:

• 59,400 US dollars paid out for more than 52 reports
• about 420 participating researchers
• up to 80,000 US dollars for Intercept X Endpoint under certain conditions
• up to 50,000 US dollars for Sophos Central
• up to 50,000 US dollars for Sophos Firewall
• 13 valid Sophos Firewall bugs in 2025 with 21,500 US dollars in total payout
• 13 valid Sophos Central bugs in 2025 with 11,650 US dollars in total payout

Those are not astronomical numbers, and that is exactly why they are interesting. They show how much sorting work sits behind a reasonably mature program. Thousands of submissions do not automatically mean thousands of relevant security issues. And AI will probably not make that ratio more relaxed.

It will make it harder.

Reproducibility becomes the entry ticket

The most important consequence, in my view, is banal but uncomfortable: a security report without clean reproducibility becomes less valuable.

Not because triage teams are lazy. Because their time is getting scarcer.

If a report claims to show remote code execution, an auth bypass, or a critical data exposure, it has to prove cleanly:

• which version is affected
• which configuration is required
• which privileges the attacker needs
• which steps reproducibly lead to the result
• which logs, requests, traces, or screenshots support the claim
• which impact has actually been demonstrated
• where the line sits between assumption and proof

That sounds strict. It is. But that is exactly what becomes necessary if security teams are not supposed to drown in plausible-sounding text.

An AI-generated report can sound very convincing. It can quote code, use CVE-like language, and imitate a clean structure. That does not make it true. Conversely, a short, dry report with a good proof of concept can be extremely valuable.

The new currency is not wording. The new currency is evidence.

AI makes authorization bugs especially unpleasant

One point in the Sophos article feels especially practical to me: AI can help scale a discovered authorization bypass across larger scope surfaces.

That matches what you see in real SaaS environments anyway. Authorization is rarely one switch. It lives in roles, tenants, object IDs, subdomains, API versions, admin surfaces, mobile endpoints, legacy routes, and half-migrated features.

If a researcher finds a pattern, AI can help test variations systematically:

• Does the bypass affect only one endpoint or a whole endpoint family?
• Does it work only inside one tenant or across tenants?
• Does the same logic exist in old and new APIs?
• Are admin and user roles really separated everywhere?
• Can an object be loaded directly by ID even though the UI would not show it?

That is exactly where AI becomes dangerously useful. Not as a magical hacker, but as an accelerator for boring, broad, systematic testing.

And that is bad for organizations whose security depends heavily on nobody having enough time to test the boring variants.

Bug bounty is not a PR inbox

Many companies still treat bug bounties half like an image topic: we have a program, therefore we are open, modern, and security-minded.

That is no longer enough.

A bug bounty program is a production system. It needs clear rules, good triage, technical reproduction, product closeness, engineering ownership, and incident response integration. Otherwise it simply becomes a public inbox where external researchers, AI slop, and real attackers share the same entrance.

Sophos brings two uncomfortable points together in the article.

First: good researchers help. External perspective, different thinking patterns, and continuous pressure are valuable.

Second: a system with money and trust can also be abused. Sophos points to experiences around Pacific Rim, Asnarök, and Personal Panda, where temporal proximity between active exploitation and later bug bounty reports at least raised questions. Sophos explicitly does not say that every such report was malicious. But the operational point remains: a bug bounty program must not be built naively.

Concretely, that means:

• Reports belong in correlation with telemetry.
• A new finding should be able to trigger retrospective threat hunts.
• Triage has to know whether similar exploit attempts were already visible.
• Researcher reputation helps, but it does not replace technical verification.
• Safe harbor is important, but it is not a substitute for abuse detection.

That is the sober reality: bug bounties are part of Secure by Design, not a replacement for it.

Harder evidence also means harder responsibility

There is a tension here that should not be moderated away.

Historically, researchers are often told: please stop early enough. Show the bug, but do not go too deep. Do not touch customer data. Do not move laterally. Do not perform destructive actions.

That is right.

At the same time, the burden of proof will rise. If AI creates large volumes of plausible but false reports, programs will demand more evidence. Then the difficult question becomes: how can a researcher prove impact more strongly without sliding into dangerous behavior?

The answer cannot be: “just do more.” The answer has to become more controlled:

• clearer rules of engagement
• dedicated test environments
• safe reproduction paths
• agreed boundaries for impact proof
• better sandbox and lab systems
• automated replays of proofs of concept

For larger vendors this becomes almost mandatory. Anyone seriously paying high bounties should also have the infrastructure to validate reports cleanly and quickly.

For smaller projects it is more bitter. They get the same slop wave, but not the same resources. That is exactly why some projects will reduce financial bug bounties or shut them down entirely. Not because they do not take security seriously, but because operating the program itself becomes a burden.

What admins and MSPs can learn from this

You could say this only affects vendors and Bugcrowd programs.

I do not think so.

The same mechanism also hits MSPs, internal IT teams, and security owners. Wherever external or internal findings have to be assessed, pressure rises:

• Scanners deliver more findings.
• AI assistants explain findings more convincingly.
• Developers bring AI-generated security notes.
• Customers ask about CVEs before they know the context.
• Management wants to know whether a risk is real or merely loud.

The practical answer is not to ignore everything. The answer is a harder validation process.

For me, that includes at least these questions:

1. Is the problem reproducible?
2. Is the affected scope clear?
3. Is there a realistic attacker path?
4. Is the impact technically proven or merely claimed?
5. Are there logs or telemetry that could show exploitation?
6. Is the fix a patch, a configuration change, a workaround, or just a bandage?
7. Do we need to search retrospectively for previous exploitation?

That sounds like more work. It is. But it is better work than reacting frantically to every well-written report.

Why this fits Mythos

The decisive point about Mythos, for me, was never only: “wow, a model finds bugs.”

The point was: if these capabilities become more real, the time between finding, understanding, reproducing, and exploiting goes down. That is exactly where bug bounty programs get hit. They sit at the intersection of research, attack potential, engineering, and responsibility.

Sophos says something similar in its own article: the question is not how to stop AI submissions. The question is how to preserve trust and signal when good research and noise can both be produced at machine speed.

To me, that is the cleanest summary of the problem.

Not every organization needs its own large bug bounty program. But every organization needs better mechanisms for checking technical claims. The flood of security information is not getting smaller. It is getting faster, louder, and better written.

My take

I think the Sophos article is a useful follow-up to Mythos because it moves the debate from the model room into the operations room.

Mythos is the spectacular signal. Bug bounty triage is the workbench where we see whether security processes can keep up.

My thesis is simple:

• If you only collect more reports, you will lose.
• If you demand reproducible evidence, you will prioritize better.
• If you connect bug bounty, telemetry, engineering, and incident response, you will react faster.
• If you see AI only as a text generator, you underestimate its value for systematic security work.
• If you do not filter AI slop, you burn the time of the people who need to solve real problems.

That sounds less glamorous than a frontier model finding zero-days. But that is exactly where it will be decided whether the security gain reaches defenders or whether everyone just drowns in more noise.

Until next time,
Joe