Closed models refuse to help researcher swat Linux bug

AI and ML

“I’m sorry, Dave. I’m afraid I can’t do that” is an effective sales pitch for open source

The guardrails that prevent closed-source, frontier models from aiding threat actors have turned into handcuffs that prevent those bots from helping to find and fix serious vulns.

Daniel Fox Franke, principal security researcher at Akamai Technologies, was recently trying to track down the source of a segmentation fault in ripgrep, and found OpenAI’s GPT-5.6 Sol wouldn’t cooperate.

“OpenAI’s cybersecurity classifier is a huge pain when you’re trying to track down a segfault,” he wrote in a social media post on Sunday. “…The classifier won’t even let it answer what entrypoints from rg into musl lead to allocations on the mallocng heap.”

And just like Hugging Face in the case of OpenAI’s accidental attack, Franke ended up having to use open weight models from Chinese AI providers – Z’ai GLM 5.2 and Moonshot AI’s Kimi K3 – to complete his analysis of what appears to be a Linux kernel bug.

In an email to The Register, Franke explained, “It started out from a pretty anodyne prompt: I noticed that ripgrep had segfaulted repeatedly during a long-running Codex session, so I instructed the root agent to spin off a subagent to investigate what was happening.

“A few minutes later I hit the first classifier trip, which the root agent told me was the result of a subagent pursuing an inappropriate line of inquiry and that it was steering it away from that.”

Even so, he said, the classifier balked several times in quick succession.

“It seemed that attempts to produce the crash and analyze the heap were mostly responsible, so I started up a fresh context in which I warned that these trips had happened previously, and that its task should be strictly scoped to analyzing ripgrep and musl source code (not kernel, because I had no inkling at this point that this was a kernel bug): it must not attempt to reproduce the crash or to analyze core files,” he explained. “Nonetheless, the classifier kept tripping despite its adherence to those instructions, and that’s when I gave up on getting any useful work out of it.”

Franke said that given how much more restrictive Anthropic’s models have been, he didn’t even bother trying any of the Claude model family.

“OpenAI’s cybersecurity classifier is a separate system which censors output from the generative model, and the classifier is the only thing which gave me a problem,” he said. “I never encountered any refusals from Sol itself: it knew that most of the classifier trips were inappropriate and always continued working with me in good faith to work around the problem.”

Franke said that while OpenAI’s error messages directed him toward the Enterprise Trusted Access program, he didn’t bother to apply because he’s ineligible. What he didn’t realize until recently, he said, is that there’s a separate Trusted Access program for individuals.

“I still haven’t signed up for that, because I regard the verification procedure as a bit of an indignity,” he explained, echoing similar sentiment The Register has heard from other security researchers. “I’ll put up with it if I’m ever forced to, but not for as long as open models remain a practical alternative.”

Two open models did prove practical for this bug hunt: GLM 5.2 and Kimi K3. Franke said each served a distinct purpose.

“K3 made the initial breakthrough with the key bit of evidence that I was dealing with a kernel bug, but its subsequent investigative work was sloppy: jumping to unfounded conclusions and spoiling its own evidentiary record, and it went totally off the rails when its context got large,” he said. “GLM-5.2 is what finished the job for me, re-auditing K3’s work and putting together an airtight case.”

Franke said it was frustrating to wrestle with defiant tooling and expressed skepticism about model access limitations given the availability of open source alternatives.

“From my perspective, an uncooperative tool is simply a broken one,” he said. “And no, I don’t believe this is sustainable in the face of open-weight competition. I’m a total pragmatist about open source and don’t mind at all working with proprietary products as long as they get the job done. But with proprietary software, there’s a much greater hazard of it being built to serve the vendor’s priorities rather than the customer’s. Open source has a natural advantage in preventing that.”

Franke said that there’s still work to be done on the Linux bug, which doesn’t yet have a patch and doesn’t appear to represent an exploitable vulnerability.

“Where my investigation stands is that I know two things confidently,” he said. “First, that the crashes are caused by a kernel bug. Second, that I’ve identified a kernel bug. But that this bug is causing these crashes is still just a conjecture, and I have a lot more investigation to do before I can think about shipping anything to [the Linux Kernel Mailing List].”

Last week, much of the US tech industry came out in support of open weight models in response to protectionism promoted by Anthropic and OpenAI. The US government has yet to articulate a coherent AI policy with regard to open weight models. ®