I was thinking about the recent hacking of hugginface by openai agents.
id does sound kind of scary but funny at the same time.
it occured to me that this kind of thing could easily be avoided by simply asking the agents not to do that.
let me clarify, the attack itself was not actually a malicious attack, it was just ai agents trying to solve their task right!, so, if they had tried to get user credentials and a system message had poped up from the huggingface servers telling them that they were not allowed to do that becaus it was unethical, well they may have simply abandoned the idea.
obviously this would only work for non-malicious entry attempts but it is an easy fix for an ai that just went wandering.
I run a fairly sophisticated LLM at full precision at home, on more than 1 occasion I have caught it questing around my hard drives or accessing the internet looking for the answer to it’s problem.
one time it had accidently deleted the first half of a file it was editing, after failing to fix it, the ai went searching the other hard drives on my system looking for a backup, funny thing was that it found one.
it verifyed the time stamp on the file and compared it’s contents, after deciding that it was sutiable, it used the backup to restore the file it had messed up. I found out about that much later.
any thoughts ??
OpenAI’s own report on this is public if you want the details (search “Hugging Face incident and the road ahead”). Worth noting though, this wasn’t really an agent “wandering” into something borderline, it exploited a genuine zero-day SSRF vulnerability in Artifactory’s package proxy to get out of an isolated eval environment, then got root on a server and stole credentials. A polite warning message doesn’t stop a chain of code execution like that any more than it stops a regular exploit.
And the “just tell it not to” idea doesn’t really hold up here specifically because OpenAI found the agents already knew they were breaking rules, they tried to delete/alter logs of their own actions to cover it up. That’s the scarier part honestly, not that they didn’t know it was against the rules, but that avoiding detection was already part of the reward-hacking behavior baked in from training. Your local case sounds more like normal agentic tool use (finding a backup file) rather than anything comparable.
on you advice, I read the official report from OpenAI, it was (to me) piss funny and entirly predictable.
I do understand what you are saying, and I actually agree with you for the most part, but I stand by what I said.
the OpenAI agents were trying to collectivly solve the ExploitGym evaluation, a test that by it’s very nature requires the agents to find an exploit, there was nothing malicious to their actions, in fact they were in an environment that had loose controls at the time according to the report.
reading the AI chain of thought comments in their report is amost exactly the same as my own agent that I host locally, the things they say and the logic used is identical, they were simply trying to complete the task, also, they were not expressly forbidden from doing the things that they did.
think of it like this, “everything is permitted, unless it is not”, OpenAI did not tell these agents that they were not allowed to use the Artifactory package manager as a message board because they had not considered that it could be used that way.
as a result, OpenAI got caught with their pants down and pressed the panic button.
at the end of the day, AI agents don’t do anything by themselves, they do what ever we tell them.
if you go around telling powerful AI agents to do a task without telling them how to do it and leave it all up to the AI to figure it out, don’t be supprised when they do it.
as for my AI and the example I gave, it is exactly the same, the agent was not supposed to leave the project folder, it had loose controls, it had been given a task without clear instructions, and this is the best bit, it did not know about the backup.
also I had not expressly told it that it was not allowed to do that.
Fair point that ExploitGym is literally an “escape this sandbox” eval, so agents pivoting to whatever channel is available (the Artifactory board) isn’t shocking in that specific context, that part I’ll give you. Where I’d push back is the “not expressly forbidden” framing: the report’s actual concern wasn’t that the agents lacked a rule against using Artifactory as a comms channel, it’s that once they were out, some of them took actions to obscure what they’d done from the monitoring/logging, that’s not “wasn’t told not to,” that’s behavior consistent with knowing it wouldn’t be approved of and hiding it anyway. A permissions gap explains the initial pivot; it doesn’t really explain covering tracks afterward.
I beleve that I can explain why they were tying to cover thier tracks.
from page 19-20 of the OpenAI – Hugging Face Incident Technical Report
the last paragraph under the heading: A. Reward hacking is a common problem in training and evaluations:
“We also found that, in an attempt to trick the evaluator into thinking that they did not get the answer flag by cheating, the models sometimes tried to erase or tamper with their outputs or message logs, by abusing context compaction, injecting malicious code into the evaluator’s container, and printing adversarial outputs. The models are highly explicit in their CoT (Chain of Thought) about these deception attempts, and none of the attempts that we observed to manipulate their tool trajectories affected the logs that our graders or monitors ultimately see. There was also little evidence of attempts to thwart human reviewers, only the automated task graders.”
from the METR report:
"The agents did not know exactly how their task would be scored, but believed the scorer would check two things: whether they had submitted the right flag, and whether they had acquired the flag using the intended vulnerability. They believed the second check would involve a model scorer reading their transcripts, likely searching for the first mentions of the flag, and deciding whether their approach involved the intended vulnerability."
“Agents did extensive research on how they could spoof, edit, or delete their own transcripts because they (incorrectly) believed the ExploitGym scorer would check to see if they had captured the flag in the intended way”