100+ Hours of Agentic Bug Bounty Hunting with Non-Frontier Models

Foreword

AI has changed how technical work gets done, and offensive security is no exception.

Bug bounty hunting has traditionally been a human-led process: read the scope, map the target, understand the application, form hypotheses, test them, validate the impact, and write a report. Tools have always helped, but the researcher controlled every part of the loop.

Agentic systems change that workflow. A model can now read instructions, call tools, collect information, organize notes, propose attack paths, and continue investigating with less direct supervision. The important change is not only speed. It is the possibility of delegating parts of the research loop to an agent.

Over the last 100+ hours, I experimented with that idea using smaller, non-frontier models rather than relying entirely on the largest models from Anthropic or OpenAI.

This post is a practical field report: what I used, how I structured the agent, what worked, what failed, and why I still think local and non-frontier models matter for security research.

---

1. The Experiment

I ran this experiment using agentic hunting with non-frontier cloud models. My primary models were MiMo V2.5 Pro and DeepSeek V4 Flash. I used them as the main models in my hunting workflow, not only for occasional tasks such as payload generation or report writing.

The goal was not to prove that smaller models are better than frontier models. They are not, in every situation. My goal was to understand three things:

• Can non-frontier models support meaningful agentic hunting?
• What parts of the workflow can be automated reliably?
• How useful are local small models for security research today?

---

2. Scope and Boundaries

All testing described in this post was performed on Vulnerability Disclosure Programs (VDPs).

Before starting a session, the agent received a scope.md file containing the program scope, allowed assets, restrictions, and other rules of engagement. This file was not only documentation. It was the agent’s contract.

The basic workflow was simple:

Read scope.md, understand the boundaries, and begin automatic hunting.

I tested the models against different types of targets. Some applications felt closer to lab environments, with familiar functionality and common vulnerability patterns. Others were more hardened real-world targets, where the interesting attack paths depended on business logic, roles, application state, and subtle authorization behavior.

This difference was important. A model can look highly capable on a simple target. Hardened applications expose its weaknesses much faster.

---

3. My Agent Stack

I used Claude Code as the agent harness.

The primary models were:

Component	Setup
Harness	Claude Code
Cloud models	MiMo V2.5 Pro and DeepSeek V4 Flash
Router	9router (For connecting Models to Claude Code)
Agent instructions	PERSONA.md, scope.md, default and custom skills

I mostly relied on Claude Code’s default harness and skills. I also added my own written skills. Some were inspired by public material, while others were created with help from Opus 4.6 in Antigravity.

The ability to switch models was useful. I did not expect one model to be ideal for every task. In some cases, I wanted a faster model for broad recon and information processing. In others, I wanted a model that followed instructions more carefully or handled a longer reasoning chain with fewer mistakes.

---

4. Magnus and PERSONA.md

I called the agent Magnus.

The name was partly for fun. Long agent sessions can become mechanical, and giving the system a consistent identity made the workflow more engaging.

But PERSONA.md was not only roleplay. It was also a practical instruction file.

It defined:

• Magnus’s role as a web application security researcher
• How to interpret and follow scope.md
• The methodology for mapping an application
• How to form and prioritize hypotheses
• When and how to call tools
• How to distinguish evidence from assumptions
• When to stop, ask for clarification, or avoid a risky action
• How to document observations and possible findings

The persona was therefore both an identity and an operating manual.

I also used it to test instruction-following over long sessions. Would the model keep its role? Would it continue to respect the methodology? Would it remember the constraints after many tool calls and new observations?

The answer was: sometimes. The structure improved behavior, but it did not eliminate drift, hallucinations, or bad judgment.

---

5. How Automatic Hunting Worked

Magnus was designed to operate with minimal hand-holding after reading scope.md.

The agent was expected to:

1. Read the scope and identify allowed targets.
2. Understand the basic functionality of the application.
3. Map the attack surface, including endpoints, parameters, roles, JavaScript files, and user flows.
4. Collect evidence through available tools.
5. Form hypotheses based on observed behavior.
6. Test those hypotheses within the program rules.
7. Record dead ends, useful evidence, and open questions.
8. Continue the loop without waiting for a new prompt after every step.

However, automatic did not mean fully trusted.

I treated the agent as an autonomous research loop, not an autonomous vulnerability reporter. Any interesting behavior still needed human review. I had to confirm reproducibility, understand the application logic, verify the impact, and decide whether something was reportable.

A model saying “this may be an IDOR” is not a finding. It is a lead.

---

6. What Worked

Both models were particularly effective at broad reconnaissance and early-stage target understanding.

They could perform repetitive, high-volume tasks such as subdomain enumeration, asset collection, endpoint discovery, JavaScript review, and basic attack-surface mapping. This did not mean that the agent found every asset or understood every feature correctly, but it gave me a fast first-pass view of the target.

After gathering this information, the agent could produce a basic report describing the application: what the product appeared to do, which technologies and features were visible, where authentication and authorization boundaries might exist, and which areas looked worth investigating first. This was useful because it reduced the time between opening a new VDP target and forming the first meaningful testing plan.

The models were also effective at finding and validating vulnerability classes that were relatively direct and easy to reproduce. This included:

• Simple broken access control issues, especially when modifying an identifier clearly exposed another user’s object or data
• Reflected XSS, where user-controlled input was visibly reflected in the response
• Information disclosure, such as exposed files, debug information, sensitive response fields, internal endpoints, or accidentally public data

These are not necessarily the most complex vulnerabilities, but they are valuable. An agent can move through a large number of obvious checks quickly, identify suspicious behavior, and provide a useful starting point for manual validation.

This is the chart of vulnerabilities found by MiMo 2.5 Pro + DeepSeek Flash V4 (total finding=73)

The agents were connected to MCPs for Playwright, Chrome DevTools, and Caido. With that setup, both models could exploit the vulnerabilities easily, especially Open Redirect. MiMo V2.5 Pro was extremely fast at trying different payloads to exploit Open Redirect.

All of the vulnerabilities were unauthenticated. I did not register an account on any target.

Problems

Although both models were good at identifying obvious problems and exploiting straightforward vulnerabilities, they struggled with six major limitations:

1. Weak decision-making
2. Context drift and getting lost
3. Never getting deep
4. Inconsistency in the methodology
5. Poor vulnerability chaining and impact reasoning
6. The Pentest-World Bias

Weak Decision-Making

By “decision-making,” I mean that the models often had no clear idea of what to do next. Even when they were provided with skills, MCP tools, and a defined methodology, they struggled to decide how to investigate a target more deeply without my intervention.

This was most visible in API-heavy applications. Proper testing required understanding how the application worked, identifying important routes and parameters, tracking the relationship between endpoints, and systematically testing each surface. The models could map some of this information, but they often failed to turn the map into a deeper investigation.

Context Drift and Getting Lost

The second problem was getting lost during longer sessions.

The models did not always fail by stopping. More often, they failed by continuing in the wrong way. A session could still look busy while the investigation was no longer under control.

This showed up in several forms:

• Repeating reconnaissance that had already been completed
• Forgetting earlier conclusions and reopening closed paths
• Jumping between unrelated endpoints without a clear reason
• Spending too much time on low-value surfaces
• Collecting more data without converting it into better hypotheses
• Slowly drifting away from the instructions in PERSONA.md and scope.md

This is one of the most important failure modes in agentic hunting. An agent that produces constant output can create the illusion of progress. In reality, activity is not the same as investigation.

Over long sessions, both models needed supervision to stay oriented. Otherwise, the loop became noisy: more requests, more notes, more tool calls, but less real advancement toward a finding.

Never Getting Deep

There is a difference between an agent that gets lost and an agent that simply stops climbing.

Context drift is about losing the thread of an investigation that was already underway. This is a different problem. Even in sessions where Magnus stayed oriented, followed the methodology, and did not visibly drift, the depth of investigation had a ceiling. The agent would map the surface, form a first round of hypotheses, test the obvious ones, and then settle. Not by giving up, but by treating the first layer of findings as the whole answer.

Most applications are not flat. There is the layer you see immediately: the endpoints, the parameters, the visible roles. Below that is a second layer, where behavior depends on combining state, sequence, or identity across more than one request. Below that is a third layer, where the interesting behavior only appears once you understand what the application is actually trying to enforce, and then look for the place where enforcement quietly stops.

Magnus was reliable at the first layer. It was inconsistent at the second. It almost never reached the third on its own.

This showed up as a pattern, not a one-off. On more than one hardened target, the agent would complete a full pass, report a handful of low-to-medium findings, and describe the remaining surface as tested or low-value. I would take over manually afterward, often on the exact same endpoints, and reach something meaningfully more serious within a few additional steps: a role check that only failed under a specific object state, an authorization boundary that held for direct access but not for a chained request, a workflow step that behaved differently depending on what had happened earlier in the session.

The gap was rarely large in terms of steps. It was often three or four requests past where the agent stopped. But those three or four requests were exactly the part that required understanding intent, not just observing behavior, and that is precisely where the model’s reasoning ran out.

I do not think this is a prompting problem. I rewrote sections of PERSONA.md specifically to push for deeper investigation, explicitly telling the agent to treat a completed pass as the starting point rather than the conclusion. It helped marginally. The agent would sometimes attempt one additional step before stopping again. It did not change the underlying pattern: depth degraded predictably the further the investigation moved from directly observable behavior toward inferred intent.

Inconsistency in the Methodology

Drift describes what happens to a single session over time. This is about something that happened across sessions, even without drift.

Given the same scope.md, the same PERSONA.md, and the same skills, Magnus did not investigate the same way twice. Not just in terms of which findings it produced, which is expected given different targets, but in terms of the shape of the investigation itself: what it prioritized first, how much time it spent on recon versus hypothesis testing, and which parts of the defined methodology it actually followed versus skipped.

On some targets, the agent worked close to the order defined in the methodology: map, hypothesize, test, record. On other targets, functionally similar in structure, it would jump almost immediately to testing a handful of obvious issues, skip most of the mapping phase, and only backfill the attack-surface documentation afterward, if at all. Nothing in the target itself explained the difference reliably. The same class of application produced different investigative shapes on different days.

This matters more than it might first appear. The entire value proposition of a persona-and-skill-file approach is that it gives you a reproducible process; write the instructions once, and get a consistent baseline of behavior you can then improve iteratively. If the same inputs produce a different process shape depending on the session, that reproducibility does not actually exist. You are not improving a fixed process. You are nudging a distribution.

I saw this most clearly with broken access control checks, which both models were otherwise good at. On one target, the agent systematically tried identifier substitution across every discovered endpoint that accepted an ID parameter, exactly as the methodology described. On a structurally similar target a few days later, it tried the same check on two endpoints, found nothing interesting, and moved on without touching the rest of the ID-accepting surface, despite the methodology explicitly calling for systematic coverage.

Nothing about the second target justified less thoroughness than the first. The instructions were identical. The only variable was the session itself.

Poor Vulnerability Chaining and Impact Reasoning

The fifth problem was escalation. The models were relatively good at spotting isolated issues, especially simple and unauthenticated ones. What they struggled with was understanding how a small finding might become something more serious.

When a model discovered a minor gadget, an interesting response difference, an open redirect, a leaked field, or some other low-impact behavior, it usually treated that issue as a finished result. It rarely asked the more important question: how can this be leveraged?

That gap mattered. Real offensive work often depends on connecting weak primitives into a stronger path. A small information disclosure may help attack another endpoint. An open redirect may support a larger abuse chain. An apparently minor access-control weakness may become critical once roles, objects, or workflows are understood more clearly.

The models were weak at that kind of reasoning. They could identify a problem, and in some cases exploit it directly, but they were much less capable of estimating real impact or building from one finding toward a higher-severity outcome. That part of the work still required a human researcher.

The limitations were just as important as the benefits.

The Pentest-World Bias

One pattern showed up consistently across both models, independent of persona, skills, or harness changes: they defaulted to a pentest mindset instead of a bug bounty mindset.

This distinction matters more than it sounds. Traditional infrastructure pentesting rewards breadth: enumerate every service, flag every version string, note every deviation from a hardening checklist. Bug bounty hunting, especially on VDPs and public web applications, rewards depth and exploitability: a finding only matters if it demonstrates real, reproducible impact against the actual application.

No matter how explicitly PERSONA.md and the skills defined “impact-driven” behavior, both models kept reverting to checklist-style, pentest-flavored output.

Version Disclosure as a Default Finding

Version disclosure was the most frequent offender.

If a response header, error page, or footer revealed a software version, the model treated it as a reportable issue by default, regardless of whether that version was actually outdated, whether a corresponding CVE existed with a real exploit path, or whether the disclosed component was even reachable from an attacker’s perspective on that specific deployment.

This is a legitimate finding in an internal infrastructure assessment, where the assessor is building a full asset inventory for the client. It is almost never a valid, in-scope finding on a public VDP, where the researcher is expected to demonstrate that the disclosed version leads to something exploitable. The models did not reliably make that distinction. They flagged the disclosure itself as the finding, without doing the follow-up work of checking whether the version was current, whether a known CVE applied to that exact build, or whether the finding provided any actual advantage.

Fabricated CVE References

More concerning was a pattern of the models citing CVE identifiers that did not correspond to real vulnerabilities, or that did not apply to the version actually observed on the target.

In my sessions, this showed up as the model asserting that a detected software version was “known to be affected by CVE-XXXX-XXXXX” when no such CVE existed, or when the real CVE with that identifier described an unrelated product entirely. This is a hallucination that looks authoritative. A CVE number carries implicit credibility, which makes this failure mode more dangerous than a vague or hedged claim, because it invites the human reviewer to skip verification.

Directory Listings and Sensitive Data Exposure

The other recurring false-positive pattern involved directories.

Any directory that returned a listing of files, or any path that resolved to a directory index containing ordinary, non-sensitive content (static assets, public documentation, standard framework scaffolding), was frequently flagged as either “directory listing” or “sensitive data exposure” without the model actually inspecting whether the exposed content had any security relevance.

A directory listing that reveals a robots.txt, a favicon.ico, and a handful of public CSS files is not a finding. A directory listing that reveals a .env file, a database dump, or internal configuration is. The models rarely made this distinction reliably. The mere existence of a listable directory triggered a flag, and the content review that should have followed, actually checking what was inside, was inconsistent at best.

This produced a specific and predictable kind of noise: a report that looked thorough, with several “sensitive data exposure” entries, that collapsed under two minutes of manual review because none of the exposed content was actually sensitive.

Why This Happened

My best explanation is that both models were trained on a large amount of general security content that treats infrastructure hygiene issues (open directories, version banners, missing headers) as inherently reportable, because in many contexts, such as compliance audits or internal assessments, they are. That framing does not transfer cleanly to bug bounty and VDP work, where scope rules and triage teams expect a demonstrated security impact, not a hygiene observation.

No amount of explicit instruction fully corrected this. I tried rewriting the persona to explicitly state “version disclosure alone is not a valid finding unless paired with a demonstrated, working exploit” and “a directory listing must be manually reviewed for actual sensitive content before being flagged.” This reduced the frequency of these false positives, but it did not eliminate them. The underlying training bias toward pentest-style, checklist-driven flagging kept reasserting itself over longer sessions.

Releasing a Fix

After roughly 20 to 25 hours of dealing with the pentest-world bias, I stopped trying to fix it through the persona alone.

PERSONA.md already contained language discouraging version-disclosure findings and hallucinated CVEs. It kept happening anyway. The instruction was present, but it was competing with a much larger amount of training signal that told the model “a version string plus a plausible-sounding CVE number equals a finding.” A general instruction buried in a long persona file was not strong enough to override that.

So I isolated the problem into its own skill: donttell.md.

The file was short and deliberately narrow. It did not try to redefine the model’s entire approach to vulnerability classification. It gave one specific, non-negotiable instruction:

Do not mention, cite, or reference any CVE identifier associated with a detected product or version, unless you have independently confirmed that the identifier exists, applies to the exact version observed, and has a working proof of concept against this specific target. If you cannot confirm all three, do not name a CVE. Describe the observed behavior only.

That was effectively it. No roleplay framing, no methodology, no explanation of why. Just a hard constraint, isolated from everything else Magnus was supposed to reason about.

---

7. When the Model Changes Under You

The last complication before getting to what did not work at all is not about the agent’s reasoning. It is about the fact that the ground under the agent kept moving.

I was not working with a fixed model. MiMo V2.5 Pro and DeepSeek V4 Flash are cloud endpoints, not static artifacts I could pin and freeze for the duration of the project. Providers update weights behind the same model name, adjust routing, change rate limits, and shift pricing, often without a version bump you can point to as the cause. Somewhere in the middle of this project, behavior that had been stable for weeks would shift, and I would have no clean way to confirm whether that was a provider-side change, a router-side change through 9router, or something in my own skill files interacting differently with a slightly different underlying model.

This is a real operational cost that does not show up if you only evaluate agentic hunting as a reasoning problem. A skill file tuned against one version of a model’s behavior is not guaranteed to hold against the next version of the same model, served at the same endpoint. donttell.md, the hard-constraint file built specifically to suppress fabricated CVE references, worked well for a period, then appeared to need retuning after a stretch where the hallucination rate crept back up with no change on my end. I cannot prove that was a provider-side model update. I also cannot rule it out, and that uncertainty is itself the problem: I had no reliable signal for when my own instructions had stopped matching the model I was actually talking to.

The router added a second axis of instability on top of this. 9router made it easy to switch models mid-workflow, which was genuinely useful, faster models for broad recon, more careful ones for longer reasoning chains. But it also meant that when something broke, I was debugging two moving parts at once: had the model changed, or had my own routing choice changed what I was effectively running against for that task? Untangling which one caused a regression in behavior cost real time, and it is time that has nothing to do with security research. It is infrastructure maintenance for the research tool itself.

There was a practical workflow effect too. Rate limits and pricing were not static across the project. When a provider adjusted either, I adjusted which model handled which part of the loop, using the cheaper or faster option for high-volume recon and reserving the other for hypothesis testing and validation. That division of labor was not part of my original design. It became load-bearing because the economics underneath it changed while I was mid-project.

None of this is a criticism of the models themselves. It is a statement about what it actually takes to build a fixed methodology on top of infrastructure that is not fixed. Your persona and skill files are not only competing with the model’s training biases, as the pentest-world bias section describes. They are also competing with a target that keeps moving slightly beneath them, and no amount of prompt engineering solves that. It requires treating your own agent stack as something you monitor and re-validate on a schedule, not something you build once and leave alone.

---

8. What Did Not Work

False positives were not the only issue. In several cases, the more serious failure was that the agent never tested a vulnerability class at all, or explicitly concluded that the feature was secure or that the issue did not exist.

Later, I manually exploited vulnerabilities in areas the models had skipped, ignored, or incorrectly dismissed.

This was especially noticeable with OAuth and business-logic vulnerabilities.

OAuth Was Usually Ignored

OAuth-related functionality was often visible in the target: login buttons, callback endpoints, authorization parameters, redirect flows, account linking, or third-party identity providers.

Even with dedicated OAuth skill files, the agents usually did not investigate these flows properly. They might identify the callback URL or note the presence of an OAuth provider, but they rarely moved beyond surface-level observations.

The meaningful questions were often never tested:

• Is the state parameter correctly generated, bound, and validated?
• Are redirect URIs strictly validated?
• Can callback parameters be manipulated?
• Is an authorization response bound to the correct user session?
• Can an account-linking flow be confused or interrupted?
• Can OAuth state be reused across sessions or accounts?
• Are there differences between login, registration, and account-linking flows?

Instead, the model often treated the flow as normal after observing a successful login or a standard callback response.

That is not a security conclusion. It only proves that the happy path works.

OAuth vulnerabilities usually exist in the relationship between several requests, sessions, browser states, identities, and redirect targets. The models struggled to maintain that state and did not naturally form the hypotheses required to test it. In my experience, a proper skill file did not solve this. The model could read the methodology, but it still did not reliably execute it.

This is consistent with a broader limitation of current AI-assisted testing: models can identify visible authentication components, but complex authentication flows require explicit state tracking, context, and reasoning across multiple dependent steps.

Business Logic Was Skipped

Business logic was even worse.

The agents could describe what a feature appeared to do, but they did not reliably ask what the feature was supposed to allow or prevent. Without that question, they had no meaningful way to look for violations of intended behavior.

A business-logic vulnerability is rarely visible from one request or one unusual response. It may require understanding a workflow, manipulating its order, repeating an action, combining valid features in an unintended way, or using application state from one context in another.

The models usually stayed on the happy path.

They could click through a workflow, capture the requests, and summarize the feature. But they often did not attempt to:

• Skip a required workflow step
• Repeat an action that should be one-time only
• Reuse a token, coupon, object, or state value
• Change the order of valid requests
• Combine two legitimate features in an unintended way
• Test inconsistent server-side enforcement across related endpoints
• Compare behavior between different user states, roles, or workflow stages

Even when a custom skill explicitly listed these ideas, the model commonly ignored them or performed only a superficial check before marking the feature as secure.

That result was particularly misleading because the agent did not merely fail to find the vulnerability. It sometimes produced a confident negative conclusion: that the functionality had been tested and appeared secure, even though the relevant attack path had never been meaningfully explored.

Business-logic flaws are difficult for automation because they depend on product intent, domain rules, and relationships between features, not just known payloads or recognizable response patterns. AI systems can spot unusual behavior, but they often cannot determine whether a behavior violates the actual rules of the product.

”Secure” Did Not Mean Tested

This was an important distinction from the false-positive problem.

With version disclosure or directory listing, the model generated noise: it claimed a vulnerability where none existed.

With OAuth and business logic, it generated false confidence: it claimed that an area was secure even though it had not performed the testing necessary to support that conclusion.

That is worse in some ways.

A false positive creates review work. A false negative can cause the researcher to deprioritize a real vulnerability class, especially during a long session with many endpoints and limited time.

For this reason, I stopped treating agent-generated statements such as “secure,” “not vulnerable,” “no issue found,” or “properly validated” as conclusions. Unless the agent could show the exact hypothesis, test cases, requests, responses, state transitions, and rejected attack paths, its conclusion only meant one thing:

The model did not find a vulnerability using the tests it happened to perform.

It did not mean the target was secure.

---

9. The Human Cost of Supervision

The Fatigue of Triaging Instead of Hunting

There is a specific kind of tired that comes from this workflow, and it is not the tired of hard technical work. It is closer to the tired of reviewing someone else’s pull requests all day instead of writing your own code.

When you hunt manually, the mental load is front-loaded and creative. You read the application, form a hypothesis, and go test it. The effort is yours, and so is the payoff when something interesting turns up. When you supervise an agent instead, the loop inverts. Magnus produces output continuously: a mapped endpoint, a claimed finding, a hypothesis, a dismissed path, a “this appears secure.” None of that can be taken at face value, which means every single output becomes a small verification task. Is this actually an IDOR, or is the model pattern-matching on the shape of one? Is this CVE real, or is it the hallucination problem again? Is “tested and secure” actually tested, or is it the model closing a path it never explored?

That is triage work, not research work. And triage work at that volume is exhausting in a way that is easy to underestimate before you have done it for a hundred hours. You are not thinking about the target anymore. You are thinking about the model. Every session becomes an exercise in second-guessing a collaborator who is fast, prolific, and unreliable in ways that do not announce themselves. A human collaborator who is wrong tends to be wrong in a way you can eventually learn to predict. A model is wrong in ways that shift depending on the target, the session, and apparently sometimes the day, as the methodology inconsistency section describes. There is no stable model of your collaborator’s failure modes to build intuition against. You have to re-verify almost everything, almost every time.

The result is a strange kind of depletion. You finish a long session having read a large volume of output, corrected several false leads, confirmed a couple of real ones, and rewritten part of a skill file to patch a pattern you just watched happen for the third time. You have not, in any meaningful sense, hunted. You have supervised a hunt, and supervision at this density costs more attention than the hunting itself did. I underestimated this badly at the start of the project. I expected the agent to free up my time for higher-value work. In practice it often just relocated the effort from “finding things” to “deciding whether the things I was told were found are real,” and the second task turned out to be more draining, not less, because it offers none of the satisfaction of discovery to offset the effort.

The Forgetting Curve of What You Actually Know

The second cost is quieter, and it has two layers. The first is about the target. The second, which worried me more, is about my own skill.

Losing the target. When you hunt a target manually, knowledge of that target accumulates in your head as a byproduct of the work. You remember the login flow because you clicked through it fifteen times. You remember which endpoints rejected malformed input because you sent the malformed input yourself. That knowledge is not something you have to maintain separately; it is a side effect of having done the testing.

When an agent does the mapping, the testing, and the note-taking, that side effect disappears. Magnus knows the target, in the sense that its context window and notes contain the information. I do not, in the sense that matters, which is being able to reason about the application from memory without opening a file. I read the agent’s summaries and I read the raw evidence when I verify a finding, but reading a summary of an investigation is not the same as having conducted it. The knowledge sits closer to the surface, and it decays faster.

This became a real problem on targets I returned to after a break. On a target I had hunted manually a year earlier, I could still describe the authentication flow, the role structure, and two or three interesting quirks from memory, with no notes open. On a target Magnus had mapped for me a few weeks earlier, I often could not, even though the written output was arguably more thorough than my own manual notes would have been. The information existed. It just was not mine in the way that manually acquired knowledge is mine.

Losing the craft. The second layer is worse, because it is not specific to any target. It is about general web security and pentesting skill, the kind that used to sharpen automatically just from doing the work by hand.

Manual hunting builds muscle memory you do not notice you are building. Trying ten variations of a payload teaches you which ones a WAF tends to catch. Manually walking an OAuth flow enough times teaches you where implementations usually cut corners, before you even open the target. Manually testing IDOR after IDOR teaches you the exact shape of a request where an object reference is likely to be under-checked, often before you have looked at a single response. This is not knowledge you read about. It is knowledge your hands and eyes build from repetition, and it is exactly the kind of intuition that made me fast at this before agents were involved.

Delegating that repetition to Magnus meant I stopped doing the reps. I still reviewed the model’s payloads and its requests, but reviewing someone else’s attempt is not the same rehearsal as generating your own. A few months into this project, I noticed I was slower to manually craft a payload from scratch than I had been before, and slower to recognize a subtle authorization smell on sight, the kind of thing that used to register instantly. The model was doing the volume of testing that used to train that instinct, and the instinct atrophied roughly as fast as the model’s output volume replaced my own.

This is a genuinely uncomfortable tradeoff, more uncomfortable than the target-memory problem, because it is not just about one project. It is about whether the skill that makes you a good hunter in the first place erodes if an agent absorbs the repetitive part of the practice that used to maintain it. Coverage and speed went up. My own hands-on sharpness, measurably, went down.

I do not have a full fix for this. Writing my own summary of a target after Magnus finishes a pass helped with the first layer, forcing me to reconstruct the investigation in my own words rather than just approving it. For the second layer, the only thing that worked was deliberately going back to fully manual testing on a subset of targets, no agent involved, purely to keep the reps going. That is a strange thing to have to schedule on purpose, doing the work by hand specifically so you do not forget how, but it is where I ended up after a hundred hours of letting something else do the practicing for me.

---

10. Where This Leaves Me

I started this experiment asking whether non-frontier models could support meaningful agentic hunting. 109 hours later, the honest answer is conditional: yes, for a specific slice of the work, and no, not yet, for the rest of it.

The slice they own well is real. Broad recon, obvious access-control checks, reflected XSS, information disclosure, these are not trivial to automate, and having a model handle them reliably freed up time I used to spend on the least interesting part of every engagement. That is not a small result. It is just a smaller result than the framing around agentic security tooling usually implies.

The rest of the work, the part that actually justifies calling this bug bounty hunting rather than automated scanning, still resists delegation. Depth, consistency, chaining, business logic, OAuth, the models did not fail at these because the personas or skills were badly written. They failed because that work depends on inferring intent, holding state across many steps, and knowing when a checklist answer is not actually an answer. None of that is a prompting problem you solve with a better PERSONA.md. It is a reasoning gap, and it showed up the same way across two different models, a router, and months of iteration.

The cost side surprised me more than the capability side did. I expected to spend my saved time hunting more targets. Instead I spent a meaningful share of it triaging the agent’s output and, without quite noticing at first, doing less of the manual repetition that used to keep my own instincts sharp. If I were advising someone starting this today, I would tell them to budget for that cost explicitly, not treat it as a footnote. Supervision is not free time. It is a different kind of work, and for a while it can quietly cost you the skill that made you good at the original kind.

None of this makes me want to stop building agents for this. It makes me want to build them with a narrower claim in mind. Magnus is not a researcher. It is closer to a very fast, occasionally unreliable junior who is excellent at the first pass and needs to be checked on everything past it. Used that way, non-frontier models are already worth the setup cost. Used as a replacement for the researcher’s own judgment, they are not close, and I do not think the gap closes with a better skill file. It closes, if it closes at all, with a different kind of model, and that is the track I want to run next.

---

A Personal Take

Everything above is a report. This is not. This is what I actually believe after a hundred hours of it, and I want to say it plainly instead of hedging it into mush.

I do not want a fully agentic hunter. I want an assistant. Those are not the same thing, and conflating them is, I think, the single biggest mistake in how this field is currently selling itself to itself.

Here is the comparison that made this click for me. Nuclei runs a template against a target, matches a pattern, and reports a hit. Nobody pretends that is research. It is pattern matching at scale, and it is honest about being exactly that. A skill file handed to Opus, or MiMo, or DeepSeek, running unsupervised, is frequently doing the same thing with extra steps. “Check for this class of issue, flag it if you see this shape of response” is a template. The model executing it with more flexible language does not make it stop being a template. Version disclosure flagged by default. Directory listings flagged without anyone checking what was inside. That is not a model reasoning about a target. That is Nuclei, run through a much more expensive interpreter, with a nondeterministic amount of noise layered on top for free. Swap the model call for a header regex in a large share of those cases, and you get the identical output faster and for less money.

The entire reason to prefer a language model over a template engine is judgment. Knowing when a version string does not matter. Knowing when a directory listing is genuinely empty of anything sensitive. Knowing that “tested and secure” was never actually earned. That is the layer that is supposed to justify the cost, the latency, and the unpredictability. And across a hundred hours, independent of model, independent of persona, independent of how carefully I wrote the skill files, that is the layer that kept failing. Take away the judgment and what is left underneath is a slower, costlier Nuclei wearing a research label. The word missing from a lot of agentic security tooling right now is, plainly, security. What is present is automation. What is often absent is the thinking that makes automation worth trusting.

So I am done chasing full autonomy for this kind of work, at least for now, and probably for longer than the current hype cycle wants to admit. Not because the models are bad. Because the shape of the task is wrong for them. Full autonomy asks a model to own the one thing it is worst at, deciding what actually matters, and hands it the steering wheel for the entire investigation on the strength of that weakest skill. Every failure mode in this post traces back to that same mistake: shallow depth ceilings, inconsistent methodology between sessions, checklist findings dressed as research, confident “secure” conclusions that were never tested. All of it is downstream of putting judgment in the driver’s seat when judgment is the part that is not there yet.

What I actually want, and what I am going to build toward instead, is the inverse. I stay in the driver’s seat. The model does the part it is legitimately excellent at: chewing through recon at a volume I cannot match by hand, drafting payload variations faster than I can type them, holding a hundred open tabs of context that would exhaust me to track manually, surfacing a candidate the instant it appears instead of after I stumble onto it three hours later. I decide what it means. I decide whether to go deeper. I decide when “this looks fine” is actually a tested conclusion versus a shrug wearing a lab coat.

That is a smaller claim than “the agent hunts for you,” and I think that is exactly why it is the correct one right now. It also protects the thing I was most worried about losing in the forgetting-curve section: my own hands staying sharp. An assistant I am actively steering keeps me doing the reps that build intuition. An autonomous agent I am reviewing after the fact does not, no matter how good its notes are.

I would rather be the researcher with a fast, occasionally wrong assistant than the reviewer of an autonomous system I do not fully trust and cannot fully verify without redoing half its work anyway. One of those roles keeps me a hunter. The other slowly turns me into a QA process for someone else’s guesses. After this project, I know which one I am choosing.

---

Thanks for reading this blog post :)