Summary
Anthropic says six employees of nonprofit evaluator METR have begun working with employee-like access to internal tools and systems. METR will publish risk assessments, subject to narrow, predefined redactions.
Anthropic says it has begun giving six employees of nonprofit evaluator METR employee-like access to internal tools and systems so they can assess the company’s AI risks. METR will publish its assessments of Anthropic’s risk levels, incidents or practices; Anthropic says it cannot veto publication, though it can require narrow, predefined redactions.
The arrangement is intended to let external evaluators examine more than model outputs. According to Anthropic, the METR team has access broadly comparable to that of the company’s own risk assessment teams, and employees are encouraged to share relevant information as they would with colleagues working on alignment risk.
What METR can access and publish
Anthropic says the six evaluators have company hardware, office space and access to internal tools and systems. The team’s work is covered by enforceable agreements, and other METR employees—including researchers working with Anthropic on different projects—are kept separate from it.
Some categories are excluded from ordinary access, including customer data, attorney-client privileged information and information about model architectures. The company says architecture information may be shared when highly relevant to risk assessment, but requires explicit internal approval. Anthropic can also require redactions in the specified categories. METR may describe those redactions publicly and say whether it considers them justified, provided it does not reveal the protected information.
That publication arrangement is central to the model: an evaluator can report its conclusions without first seeking Anthropic’s approval, while the company retains a limited ability to protect sensitive information. The distinction matters because the company holds information about training, safeguards and internal processes that may be important to assessing risks but may not be appropriate for public release.
Anthropic compares the approach, with the qualification that the analogy is not exact, to resident bank examiners who work inside large banks to understand risks that outsiders may not be able to see. For AI, the company says the aim is to narrow the information gap between an evaluator and the organisation whose systems it assesses.
A three-tier approach to external evaluation
Anthropic describes three levels of access for its evaluator partnerships. At the first tier, an evaluator conducts a narrowly scoped investigation, with access set by Anthropic and publication requiring the company’s approval. At the second tier, evaluators receive more information and can publish without prior approval, while Anthropic may still require specific redactions. The METR arrangement is the third tier: employee-like access, with publication rights similar to the second tier.
The company says it has worked with METR over the past year on reviews of its Risk Reports. It also reports that SecureBio is reviewing the chemical and biological weapons chapter of Anthropic’s August Risk Report, after completing and publishing a pilot review of an earlier report. Anthropic says it will consider projects with increased access after the current review. Separately, it says it is working with Accenture and Faculty toward model evaluation and red-teaming, alignment assessments, safeguard testing and capability-building.
These partnerships are at different stages. Anthropic says the earlier engagements provide evaluators with more limited information and publication rights, with the aim of building confidence in their security, expertise and resources before expanding access. It says it hopes eventually to have several evaluators at the third tier, each publishing its own assessment of the company’s risk posture and key decisions.