|
OpenAI said it has disrupted a coordinated campaign aimed at extracting protected reasoning from its models and using the information to help train, reproduce or improve another model.
The company said the activity was consistent with what it described as adversarial distillation, a practice involving the systematic and unauthorised use of one model's outputs or reasoning to help develop another model. It said, We recently identified and disrupted a coordinated campaign designed to extract protected reasoning from our models. According to OpenAI, the activity began on July 1 at a low volume before increasing sharply. On July 24 and 25, the company observed 16,000 requests using a relevant extraction pattern from more than 4,000 users. Further investigation by the company identified related prompt-pattern activity across a cluster of more than 15,000 users, which OpenAI said it fully disrupted by July 28. OpenAI said the operators did not break its encryption, compromise a database or gain direct access to stored user conversations. Instead, they manipulated interactions with models so that protected reasoning could be reproduced in forms visible to the requester. Protected reasoning refers to the model's internal record for working through a task. OpenAI said extracting this information can reveal material that is withheld from the final answer and could help others reproduce the model's capabilities. The company said it also observed attempts to extract protected reasoning in novel ways. In one method, operators copied encrypted reasoning from one conversation and asked a model in another conversation to decrypt and transcribe the hidden reasoning. OpenAI said independent security researchers had also reported related cross-model and conversation-compaction vulnerabilities through responsible disclosure. The company investigated those findings and confirmed that the attack paths identified by the researchers were real. On attribution, OpenAI said it was unclear whether all the operators observed during the period originated from a single actor. However, it said a core cluster of the activity was attributed to individuals associated with Moonshot AI, the developer of Kimi. OpenAI also shared that the adversarial distillation creates safety and national security risks because extracted reasoning could be used to train another model without preserving the safeguards applied to the original model's user-facing outputs. The company also said that as models gain capabilities in dual-use areas, the risks associated with this type of activity become more significant. To disrupt the campaign, OpenAI said it used a combination of account enforcement, technical controls and coordination with partners. It banned or restricted fraudulent accounts, strengthened signup and infrastructure controls and expanded monitoring for related networks. The company also strengthened protections for hidden reasoning across users, workspaces, organisations and model families. It said it closed a pathway that could allow someone possessing another user's encrypted reasoning to replay it and recover its contents. The company said it expects adversarial distillation attempts to become more sophisticated as frontier models improve and actors seek cheaper ways to mimic their capabilities. It added that work on technical protections, detection, enforcement and threat-information sharing will continue. (ANI)
|
