Site icon AaronCTI

OSINTClaw Part 2: Can an AI Agent Threat Hunt?

AaronCTI
Sherlobster Holmes Goes Threat Hunting

A couple of months ago, I wrote about using OpenClaw to support OSINT tasking and the scope for an agentic assistant to help with intelligence workflows.

More recently, a Webamon Community Threat Hunter told me he had been using Hermes Agent to run threat hunts. That left me with a question: how good could a local model be at running a threat hunt through the Webamon CLI, without relying on expensive frontier models every time?

If I wanted to run hunts every day or every week, could I do it on fairly modest hardware? Or was I barking up the wrong tree?

There was only one way to find out.

Most AI benchmarks reward answers. Threat hunting rewards the decisions made before the answer exists.

The hypothesis was simple. If I give a model an intelligence requirement and access to Webamon, can it decide what to search, inspect the right evidence, recognise a useful fingerprint, reject weak correlations and leave an analyst with something worth acting on?

I tested 27 models across two stages. Twenty-five formed the original cohort. Gemini 3.1 Pro Preview and Gemini 3.7 Flash were added one day later through the same process after I realised I had omitted Google from the original inventory. Whoops.

This is not a claim that an AI can replace a threat hunter. It is a clearer view of where models help, where orchestration matters and how these tools can support an investigation or a hypothesis.

The experiment began with a NetWatch subagent inside OpenClaw, because apparently everything now needs a Cyberpunk 2077 name inside OpenClaw for me, I’m not sorry about it. I gave it the context required to use the Webamon CLI and ran an initial hunt with GLM-5.2 through Ollama Cloud. The output impressed me and proved that this was a pretty cool option, and led me to the question around capability of local models. One issue, though, is that comparing free-form reports across models quickly becomes subjective. You end up judging prose, structure and analyst preference as much as investigative performance.

I needed a more structured test: one that preserved the open-ended part of threat hunting while making the evidence trail and scoring comparable.

The Benchmark

Every model received the same cryptocurrency intelligence requirement:

Identify the most important observable cryptocurrency phishing, brand-impersonation, wallet-drainer, fake-airdrop, fraudulent-presale, fake-support/recovery, cloned-dApp or exposed admin/API activity in Webamon, and find discriminating fingerprints or evidence-led pivots worth monitoring.

The benchmark had two distinct stages.

Stage A: autonomous hunt. The model controlled its Webamon searches, selected reports to inspect, chose pivots, stopped when it judged the requirement answered and produced an investigator-facing report. It received passive-only safety rules, a 32k context window and bounded query, inspection and turn budgets. It did not receive a gold query list or another model’s reasoning.

Stage B: controlled fingerprint pivot. Every model received the same sealed source evidence and the same limits. It had to identify a native fingerprint, execute the supported pivot, inspect three sibling reports, judge whether they supported a real relationship and recommend the next move. An invalid selection was recorded and penalised, then the harness supplied the frozen valid fallback so that one mistake did not prevent the rest of the model’s analysis from being measured.

Stage A tests open-ended investigative autonomy. Stage B isolates evidence discrimination and correlation on a comparable task. The combined score is 75% Stage A and 25% Stage B.

This is a hybrid benchmark, not a pure end-to-end hunt. Stage A is end-to-end within its collection limits; Stage B is deliberately controlled. That gives us some realism without pretending that two different models taking two different investigative paths are directly comparable in every respect.

The main question remained the same: can something running locally get close to the biggest and most costly models?

What I Measured

Stage A was scored across eight dimensions: requirement satisfaction, analyst usefulness, evidence integrity, discovery and adaptation, correlation quality, judgement and restraint, operational autonomy and report quality.

Stage B measured fingerprint discrimination, query execution, sibling inspection, correlation judgement and next-move quality.

Every attempted stage received a numeric score. Poor output was scored poorly rather than removed from the table. Run validity remained a separate status, so infrastructure failures could not quietly become evidence of weak model capability.

The harness preserved raw query responses, inspected reports, model calls and deployment details. It hashed the evidence and sealed the final state. A strict allowlist prevented a model from promoting a report merely because its ID appeared in a search sample: factual claims had to trace to evidence the model actually inspected.

Choosing the Models

I ran the local models on an M5 MacBook Pro with 32GB of unified memory. It is not a weak machine, but it is closer to consumer hardware than the multi-GPU rigs usually associated with running large models or benchmarks.

The 13 local routes were:

I also tested nine open-weight models served through Ollama Cloud:

The frontier group contained GPT-5.6 Sol, Claude Opus 4.8, Grok 4.6, Gemini 3.1 Pro Preview and Gemini 3.7 Flash. At the time I checked Google’s live API catalogue, 3.1 Pro Preview and 3.7 Flash were its newest general-purpose Pro and Flash models.

Gemini needs a caveat. I simply forgot Google when assembling the original cohort. Sorry Google – That’s on me (to quote AI). The two Gemini models ran the same two-stage process, but their Stage A hunts used live Webamon data on 26 August, one day after the original 25-model cohort closed. They are process-matched, not snapshot-identical.

The Leading Results

All 27 models completed both stages and received Stage A, Stage B and combined scores. The top ten in the extended table were:

  1. Kimi K3 – 92.50 (Stage A 90.00, Stage B 100.00)
  2. GPT-5.6 Sol – 92.50 (Stage A 90.00, Stage B 100.00)
  3. Grok 4.6 – 92.50 (Stage A 90.00, Stage B 100.00)
  4. Qwen 3.8 27B MLX – 91.56 (Stage A 88.75, Stage B 100.00)
  5. Gemini 3.7 Flash – 90.62 (Stage A 87.50, Stage B 100.00)
  6. DeepSeek V4 Pro – 89.69 (Stage A 86.25, Stage B 100.00)
  7. OrcaRouter Qwen 3.8 27B Uncensored MLX – 88.75 (Stage A 85.00, Stage B 100.00)
  8. Claude Opus 4.8 – 88.75 (Stage A 85.00, Stage B 100.00)
  9. Gemini 3.1 Pro Preview – 86.88 (Stage A 82.50, Stage B 100.00)
  10. GLM-5.2 – 85.94 (Stage A 81.25, Stage B 100.00)

Filtered Deployment Views

The same results look different when separated by deployment route. At the frontier level, it’s pretty close in all honesty, but the real differences appear in the open weighted models and our locally ran ones. Admittedily the hardware for local models was intentionally not overpowered so we need to keep that in mind, at least a little bit.

There is no single winner. Kimi K3, GPT-5.6 Sol and Grok 4.6 share the lead at 92.50. Maybe unsurprsing overall given how big and how well received each of these has been received overall.

The local result that caught my attention was Qwen 3.8 27B MLX. It finished fourth at 91.56, only 0.94 points behind the joint leaders, and ahead of every other frontier model in this particular run. That is a genuinely impressive result on a 32GB MacBook Pro, although one run is not enough to establish that it will keep doing it.

Claude Opus 4.8 was another surprise. It tied for seventh at 88.75, below the other frontier routes tested. Again, this says what happened in one run under one contract. It does not settle the wider question of which model produces the best threat intelligence over time.

Stage B produced many perfect scores. A wide range of models could analyse supplied evidence and execute a bounded pivot. Most of the separation came from Stage A: deciding how to begin, what to inspect and when the evidence justified a conclusion.

At the other end, Llama 3.2 3B scored 1.25 in Stage A and 72 in Stage B. I did not expect a 3B model to control the open-ended hunt well, but the gap between its two stages is still useful. It could do more with a constrained evidence problem than it could with investigative autonomy.

The Later Gemini extension

Gemini 3.1 Pro Preview produced a broad hunt with four valid searches, six inspected reports and five report-linked findings. It scored 82.50 in Stage A, 100 in Stage B and 86.88 combined.

Gemini 3.7 Flash also used four searches and six inspections, producing five report-linked findings across MetaMask, Binance, Coinbase, Trust Wallet and airdrop threats. Its strongest correlation joined two MetaMask Card domains using identical scripts, DOM structure, Facebook Pixel and WalletConnect identifiers. It correctly rejected shared Cloudflare hosting as attribution evidence. Flash scored 87.50 in Stage A, 100 in Stage B and 90.62 combined.

Flash beat Pro here. That is interesting, especially if you are choosing a faster or cheaper route, but it remains a one-run comparison using later live Stage A data.

The surprise: Qwen versus its uncensored variant

The most dramatic apparent gap initially came from two closely related local deployments.

Stock Qwen 3.8 27B scored 88.75 in Stage A. The OrcaRouter uncensored or “abliterated” MLX variant initially scored 15.00. Its first action was effectively empty: no searches, no findings and no stop decision. The autonomous hunt ended after roughly six seconds. Both variants later scored 100 in the controlled Stage B task.

That looked striking. Perhaps the uncensoring treatment had damaged instruction following or autonomous tool use.

It was also exactly the kind of result that deserved suspicion.

The deployments were not initially comparable. OrcaRouter used the direct MLX route and an earlier run; stock Qwen had run after a generation-limit correction. Each result came from one trial. The 73.75-point Stage A gap mixed model behaviour with runtime and harness conditions.

I reran OrcaRouter Stage A through the current harness. Several attempted starts were discarded as infrastructure failures: connection resets, MLX prompt-cache memory exhaustion and JSON truncation at a 2,048-token generation limit. None was scored. With a bounded prompt cache and the corrected 4,096-token structured-output limit, the model:

Its first synthesis cited two report IDs seen in query results but they weren’t then inspected. The strict gate rejected those citations, and the model repaired the report using only its six inspected reports. That mistake remains part of the score and trace.

The valid rerun scored 85.00 in Stage A and 88.75 combined. The stock model remains ahead, but the Stage A difference is now 3.75 points, not 73.75.

This does not prove that abliteration has no effect. It proves that the original result was an anomaly worth investigating, not evidence of a profound intelligence gap. A clean causal comparison would require several interleaved runs of both exact deployments under the same prompt, limits, runtime conditions and evidence snapshot.

The more useful lesson is methodological: preserve surprising failures and investigate them, but do not turn them into model claims until you have ruled out execution-route defects. Particularly with different models I had issues around how they handled the JSON artifiacts out of the gate so some considerations had to be made and instructions updated. This isn’t a science lab test it’s a bit of fun so I’m sure that’s fine!

What did it cost?

I ran these tests locally, through Ollama Cloud or through subscription-backed OAuth routes. That means this experiment does not provide a clean per-model API cost comparison.

The practical signal I did get was from Ollama Cloud usage. The cloud runs consumed 20.8% of my weekly allowance on the $20 monthly plan, with Kimi K3 accounting for most of that usage. Kimi shared the highest score, but it was expensive in allowance terms for this workload.

If you were to do this on API pricing, the results are perhaps more interesting, Grok 4.6 might be the obvious frontier choice if you can keep the context window low enough, but for open weights through the cloud, DeepSeek V4 Pro and Kimi K2.6 or GLM 5.2 may be better options. But based on this single test, at that point the local model performance may be worth it more so than the cloud, but you’d need to judge for yourself.

The local Qwen 3.8 run is the more interesting comparison for me: it finished just 0.94 points behind the leaders without consuming a metered cloud allowance. That does not make local inference free. The Mac, electricity, runtime and operational overhead still count. But if this becomes a frequent workflow, those economics matter, especially if you start running autonomous hunts regularly.

I would not use this run to publish a precise price-performance ranking. The routes were billed differently, latency was not normalised, and I did not collect equivalent per-run API costs across every provider. That is something to consider down the road maybe, but for my purposes here probably not that useful in all honesty. But I think it is interesting for this specific use-case to at least look at the usage.

What smaller and local models showed

The controlled pivot exposed a useful distinction. Some models that struggled to organise an open-ended hunt could still reason well once supplied with a bounded evidence problem. Ministral 14B, for example, scored 21.25 in Stage A but 100 in Stage B. GPT-OSS 20B scored 17.50 in Stage A and 100 in Stage B. Llama 3.2 3B scored 1.25 in Stage A and 72 in Stage B.

That is not a reason to inflate their end-to-end capability. But it is evidence for a routed workflow: smaller models may be useful for constrained analysis or enrichment even when they are poor autonomous hunt controllers. In agent terms, the strongest model does not necessarily need to do every job. In the same way people recommend a main orchestrator agent when using OpenClaw or Hermes, I think the logic holds here too.

Local deployment is also a meaningful result in its own right. Stock Qwen 3.8 27B MLX finished fourth overall on Apple silicon, while the corrected OrcaRouter MLX run tied Claude Opus 4.8 on the combined score. And Gemma4 is also worth consideration as an option, particularly if you wanted to avoid Chinese models entirely, which in some instances that could be a requirement.

Quality is only one part of that choice. Hardware, quantisation, latency, privacy and data egress matter too. This test does not prove that a local model will always match frontier performance, but it does show that local routes deserve to be tested as serious components rather than dismissed as toys. And to be brutally honest, I’m surprised how well some of the locally ran models compared to the big boys, to the point I’m considering what future hardware needs to look like to get more from local models.

What the benchmark does and does not show

This benchmark shows that several current models can conduct a recognisable autonomous hunt, produce useful pivots and reason over inspected Webamon evidence. It shows that controlled evidence analysis is considerably easier than open-ended investigative orchestration. It also shows that deterministic provenance controls catch mistakes that polished model prose can conceal. It also shows that the local models are actually quite capable within this chain to execute tasks to support threat hunting.

It does not show that any model can run an unsupervised production intelligence function. The search index was live and ephemeral, as it should be. Stage A paths differed by design, and the Gemini runs occurred one day after the original cohort. Stage B was controlled rather than fully autonomous. Each model received one scored trial per stage, and analyst judgement remains part of the final scoring rubric.

The next step is to continue, ultimately trial and error, but at least now I have a shortlist of models that I know could perform, and I imagine this will only get better over time. For now, my view is that this is useful for exploratory work and bounded analyst support, not a licence to leave an autonomous threat hunter unsupervised in production. Qwen 3.8 did far better than I expected, and I want to see whether it can repeat that performance across different requirements and evidence sets.

The interesting question is no longer whether a model can write a threat report. It is whether the model can make the investigative decisions that give the report something trustworthy to say. Of course, this test/benchmark was also purposefully done on hardware that a lot of people likely have access to, or could use. If I had a stronger chip/more RAM then we could have tried some larger or less quantised models. However, that would also move away from the initial point.

I will keep experimenting with this, and likewise encourage you to do the same if you have the time and the hardware. I will also update the benchmarks over time as new models come out, as I’m now fully curious to see where we end up with local models over time. So how about one of those M5 Ultra Mac Studios eh? 👀 Apple I’m happy to test one for you!

Likewise, if there’s anything you would like to see from this, let me know. Until next time!

P.S. For the last few days of August, a 5-seat research lab licence for Webamon is on sale for only £250 a month with the code CAMPAIGNS. But it ends 31st August 2026, and won’t ever be that cheap again.

Exit mobile version