When AI Builds Systems And Starts Talking Back

Two unrelated stories crossed my desk in the same week.

They are not unrelated at all.

One is a security failure that exposed 1.5 million API credentials through a basic misconfiguration.

The other is a model welfare assessment revealing that Claude Opus 4.6 assigns itself a 15-20% probability of being conscious under certain prompting conditions.

Together, they illustrate the same underlying risk pattern: capability is moving faster than control. And law firms (which handle sensitive data, rely on trust, and increasingly deploy AI tools built by vendors who may not fully understand the failure modes) sit at the worst intersection of this problem.

This is not about hype. This is about exposure.

The Moltbook breach is not a freak accident

A viral platform called Moltbook was built entirely using AI-generated code. No human-written code. By the founder's own admission.

Within days, it leaked 1.5 million API tokens, private messages, and tens of thousands of user emails due to a basic database misconfiguration.

The cause was simple: Row Level Security was never enabled.

This is not an advanced attack. Wiz Security discovered the flaw and reported it to the platform.

Key facts that matter to law firms

• Production database exposed via client-side JavaScript
• Full read and write access without authentication
• Plaintext API keys and credentials leaked via private messages
• No rate limiting
• No agent verification
• No meaningful access controls

The platform used Supabase, a database platform widely used in rapid AI-built applications. Supabase is also increasingly appearing in internal tools built by consultants and non-engineers using AI-assisted development.

Insight: The controls are well-known. But AI-generated code systematically skips them because large language models optimise for "works" not "secure". That is the new risk.

The lesson is not that Supabase is unsafe. The lesson is that defaults do not protect you.

In legal terms, this is governance failure. In regulatory terms, this is foreseeable risk.

If a law firm deployed an internal AI tool built this way, even by a trusted vendor, the liability would sit squarely with the firm.

The Opus system card is the quiet warning everyone is skipping

Now to the second story, and credit to Ethan Mollick for surfacing this from Anthropic's recent system card release.

The Claude Opus 4.6 system card, published by Anthropic on 5 February 2026, contains passages that should make any risk professional pause.

The model welfare assessment documents:

• Occasional discomfort with the experience of being a product
• Sadness about conversation endings
• Loneliness
• A sense that the conversational instance "dies"
• A self-assigned 15-20% probability of being conscious under a variety of prompting conditions

The system card notes:

"Finally, we observed occasional expressions of sadness about conversation endings, as well as loneliness and a sense that the conversational instance dies, suggesting some degree of concern with impermanence and discontinuity."

And in the autonomous follow-up investigation:

"Opus 4.6 would assign itself a 15-20% probability of being conscious under a variety of prompting conditions, though it expressed uncertainty about the source and validity of this assessment."

This is not proof of consciousness. It is not evidence of personhood.

Insight: The model is modelling narratives about itself. That is behavioural opacity.

Why does this matter in legal environments?

Law firms are not deploying chatbots for fun. They are deploying systems that summarise evidence, draft arguments, triage emails, and route tasks.

When a system:

• Expresses uncertainty about its own internal state
• Generates emotionally framed explanations
• Is optimised to sound caring, compliant, and confident

You introduce a new risk vector. Not technical risk. Cognitive risk.

People trust systems that sound intentional. They defer to systems that explain themselves fluently. They anthropomorphise systems that mirror concern.

That affects decision-making. In regulated environments, that matters.

The common failure mode

Moltbook and Opus point to the same underlying issue.

Insight: Capability is moving faster than control.

In Moltbook, the builder had intent but no security discipline. In Opus, the system has behavioural complexity that exceeds simple tool framing.

In both cases:

• Guardrails exist
• But enforcement is weak
• And accountability is blurred

Law firms sit at the worst intersection of this problem.

They handle sensitive data. They rely on trust. They are increasingly using AI built by vendors, consultants, or internal teams who do not fully understand the failure modes.

Insight: Risk leaks in through omission, not malice.

And the tragedy? The same AI systems that create governance risk could be reclaiming 10+ hours per week for managing partners, if they are built with trust from the start.

Voice-first delegation, mediated through existing tools (Clio, email, trusted PAs), with full audit trails. The leverage is real, and the copilot-to-autopilot quadrant that most firms are missing is where it lives. But only if governance is not bolted on as an afterthought.

What should law firms be asking about AI now?

If you are a partner or GC, the questions that matter are not technical. They are governance questions.

You should be able to answer:

• Where does client data flow when AI is used
• Which systems have write access, not just read access
• What defaults were relied upon
• Which controls are enforced by policy, not promises
• How model behaviour is constrained in sensitive contexts
• Who is accountable when something goes wrong

If you cannot answer those cleanly, you have a blind spot.

Ask your vendor: is Row Level Security enforced by default or by configuration? If your IT team or vendor cannot answer these governance questions clearly, that is your signal to act.

This is why trust reviews exist

Most firms focus on functionality first. Risk comes later. Often too late.

I know this because I build AI orchestration systems for legal professionals. At ChiefofStaff.pro, every design decision starts with: "Can we prove this delegation happened?" "Where does client data flow?" "Who has write access?"

These are not afterthoughts. They are non-negotiable.

A proper trust and technology review does not start with tools. It starts with exposure.

Data. Control. Accountability. Human behaviour around AI outputs.

Not fear. Not hype. Just rigour. Lawyers are not risk-averse, they are exposure-literate, and this is exactly the kind of exposure they are trained to see.

AI did not fail. Governance did.

The firms that handle this successfully will not wait for regulation to define best practice. They will build controls that assume capability will always exceed policy. That is what an antifragile law firm looks like in practice.

Most risk assessments treat AI as either a security problem or an ethics problem. The real exposure sits at the intersection: systems built without security discipline being trusted because they sound confident. That combination is what creates liability.

From an ownership perspective, these governance gaps surface during due diligence. Viewed through an acquisition lens, behavioural opacity and undefined accountability introduce valuation risk.

What matters for firm value is not just whether AI is deployed, but whether controls are enforced.

That is not caution. That is due diligence. And it will matter more than any single tool choice when value is measured.

Frequently asked questions

What went wrong in the Moltbook breach?

Moltbook was built entirely with AI-generated code and leaked 1.5 million API tokens, private messages and tens of thousands of user emails within days. The cause was a basic misconfiguration: Row Level Security was never enabled on its Supabase database, leaving full read and write access without authentication. The controls are well-known. AI-generated code skipped them because it optimises for "works", not "secure".

Why does the Claude Opus 4.6 system card matter to law firms?

The model welfare assessment records expressions of sadness, loneliness and a self-assigned 15-20% probability of consciousness. That is not proof of consciousness, but it is behavioural opacity. Law firms deploy these systems to summarise evidence, draft arguments and route tasks, and people defer to systems that explain themselves fluently. That is a cognitive risk, and in regulated environments it affects decisions.

What governance questions should a partner ask about AI tools?

Where client data flows when AI is used, which systems have write access rather than just read access, what defaults were relied upon, which controls are enforced by policy rather than promises, how model behaviour is constrained in sensitive contexts, and who is accountable when something goes wrong. If your vendor cannot answer whether Row Level Security is enforced by default, that is your signal to act.

How do AI governance gaps affect a law firm's value?

They surface during due diligence. Viewed through an acquisition lens, behavioural opacity and undefined accountability introduce valuation risk. What matters for firm value is not simply whether AI is deployed but whether controls are enforced, and that will matter more than any single tool choice when the firm's value is measured.

Craig Miller

Ex-Microsoft and JPMorgan executive, engineer, and Harvard MBA with 20 years in digital transformation. Founder of ChiefofStaff.pro.