Recent experience, developing in Opus 5.5. Spin up an agent to review the security aspects of the project, write a report. Pass the report to another Opus 5.5 agent and tell it to implement mitigations for confirmed problems found in the report. Implementation agent proceeds to make tests for the described exploits and gets shut down by the guardrails: Opus 5.5 is not permitted to “do that kind of work” had to scale back to Opus 4.8 for the implementation, though 5.5 was still happy to describe the chained attack scenarios, apparently it’s not allowed to develop demonstration code as it was trying to do for the test cases.
The most recent release of SOTA models being both more intelligent and 20-30% cheaper is really fighting against this argument (Opus 5.5 & Sol 6).
Recent experience, developing in Opus 5.5. Spin up an agent to review the security aspects of the project, write a report. Pass the report to another Opus 5.5 agent and tell it to implement mitigations for confirmed problems found in the report. Implementation agent proceeds to make tests for the described exploits and gets shut down by the guardrails: Opus 5.5 is not permitted to “do that kind of work” had to scale back to Opus 4.8 for the implementation, though 5.5 was still happy to describe the chained attack scenarios, apparently it’s not allowed to develop demonstration code as it was trying to do for the test cases.