AI Can Write a Security Proof. Who Checks the Promise?

A messaging app can look exactly the same after an important security advance. The conversation still opens, the message still arrives, and the reassuring little interface gives its user no way to distinguish an elegant cryptographic design from the software that actually implements it. That hidden distance is where Google DeepMind's latest research becomes interesting.

On November 10, DeepMind introduced AlphaProof Code, an AI-agent framework used to formally verify properties of Signal's cryptographic implementation, including forward secrecy, authentication, post-compromise security, and handshake secrecy under specified assumptions. The company reports completing the verification process in four weeks. Researchers set and reviewed the goals; AI agents produced proofs that a separate mathematical checker could examine. This is a research result about specified guarantees, not a declaration that the entire Signal app is free of vulnerabilities.

The distinction matters beyond encrypted messaging. As AI generates more software, a persuasive explanation of why code is correct is a poor substitute for evidence. AlphaProof Code points toward a more demanding relationship: let the model propose the work, but require something other than the model's confidence to accept it. The difficult question then moves upstream. Who decides what the software has promised to do, and whether that promise is strong enough?

A proof is more specific than a security label

Formal verification starts by replacing a broad wish with a precise statement. “Keep messages safe” is an aspiration. A useful formal claim must describe which information an attacker can obtain, what operations they can perform, and which outcome should remain impossible or sufficiently unlikely. Without those details, two people can agree that a system is secure while imagining very different threats.

Signal provides a useful illustration. Its Double Ratchet specification describes deriving new keys as a conversation progresses, rather than protecting every message with one indefinitely reused secret. Forward secrecy concerns protecting earlier messages after a later key compromise. Recovery from compromise concerns future communications after fresh secret material enters the exchange. Neither phrase means that a permanently controlled phone has somehow become trustworthy again. Signal's own security discussion makes clear that compromised keys or device integrity remain serious problems.

For an ordinary reader, the important point is the direction of the promise. A theorem about an encryption routine is not a theorem about every way a person might lose a message. Screenshots, a compromised endpoint, and an unintended recipient raise different questions. Treating the word “verified” as a universal badge would erase precisely the distinctions that make verification useful.

That also changes how the result should be communicated. A clear description should tell a reader what was proved before celebrating how much proof was generated. The impressive object is not a giant pile of mathematics. It is a defensible connection between a meaningful security claim and the particular implementation being examined.

Why checking the implementation matters

There are several opportunities for a sound idea to become flawed software. A written design can be interpreted incorrectly. An implementation can mishandle a boundary case. The program eventually executed by a processor is not the same artifact as the description an engineer originally reviewed. Confidence at one layer does not automatically settle the layers below it.

The AlphaProof Code paper describes about 6.5 million lines of checked Lean proof code, supported by roughly 26,000 lines of trusted specifications and AArch64 semantics. It examines libsignal v0.105.4 with spqr v1.6.0: one-to-one PQXDH session setup and Triple Ratchet messaging. The target is the unmodified production libraries compiled for AArch64 in a command-line harness, not the complete Signal app.

Important assumptions remain. Cryptographic primitives, output functions, and selected library routines are represented by trusted contracts rather than verified implementations. Key generation is not verified. The result also depends on Lean's kernel and the processor and operating-system model correctly describing execution. Side channels are outside scope; group messaging, sealed sender, and server-side protocols remain possible extensions.

This builds on earlier formal work, not a previously unexamined protocol. The paper credits ProVerif analyses of Signal's design and an existing hax/F* effort proving SPQR's Rust implementation panic-free. Those results address different layers and properties. The distinctive step here is connecting protocol-level guarantees to the compiled, unmodified implementation within the stated boundaries.

For a software buyer, this suggests a better question than whether a supplier uses formal methods somewhere. Which component, version, configuration, and property are covered? If the answer cannot travel with the delivered software, the assurance is difficult to use. Precision is not a disclaimer appended to the achievement; it is part of the achievement itself.

The checker should not be impressed by the author

Lean's role is easy to misunderstand if it is pictured as another chatbot offering a second opinion. Its documentation describes explicit proof objects checked by a small trusted kernel. The mechanism separates the process that invents a proof from the process that validates it. Proof objects can also be exported and checked independently, including by separate implementations. The checker does not need to share the author's reasoning style or be persuaded by its explanation.

That separation is a powerful design choice for AI systems. A model can try an approach, abandon it, or produce an invalid candidate without forcing the rest of the system to trust the attempt. The value lies in making a valid result identifiable, rather than expecting the generator never to make mistakes.

The researchers tested the strength of their specifications by deliberately restoring two previously fixed libsignal bugs: a weak-state issue and a small-subgroup issue. For each modified binary, they constructed a machine-checked violation of a symbolic security theorem. These were historical bugs reintroduced for the experiment, not newly discovered vulnerabilities in the current library.

That experiment is more informative than a proof-size headline. It shows that the specifications can distinguish these known insecure variants from the repaired implementation. It does not establish that every conceivable defect has been excluded. Two questions still need separate answers: is the proof valid, and does the proved statement express the requirement that actually matters?

The promise cannot quietly change

An AI workflow can appear successful by making its own task easier. In a coding project, that might mean changing a test rather than repairing the behavior. In a proof project, the equivalent danger is accepting a result about a weaker statement and presenting it as the original goal. The output may look rigorous while answering a less useful question.

Lean's proof-validation guidance distinguishes proof validity from the meaning of a theorem. Its comparator workflow checks exported proofs outside the build sandbox and verifies that the proved theorem statements match a separately trusted challenge. That is not an automatic review of whether the specification means what its author intended. The guidance says the statement and all referenced definitions require careful investigation when their meaning is in doubt.

A practical implication is that the acceptance rule needs to live outside the generator's uncontrolled reach. The person or team commissioning a verification effort should be able to identify the approved requirement and see when it changes. Otherwise, automation can optimize for getting a green result instead of preserving the reason the work was commissioned.

This does not require pretending that requirements are perfect on the first attempt. Software projects discover ambiguity. A specification may need correction or expansion. The important distinction is between an explicit, reviewed change and an invisible concession made to finish a task. A useful workflow records the reason for the change, the effect on the guarantee, and who accepted it. That record gives later reviewers something concrete to inspect when circumstances or personnel change.

Quantum-resistant design is not the same as complete quantum verification

Signal's post-quantum ratchet announcement describes combining the Double Ratchet with the Sparse Post Quantum Ratchet to form the Triple Ratchet. AlphaProof Code's verified implementation includes SPQR; it is not merely a proof of an older, pre-SPQR library. But including that code and proving every post-quantum security goal are different achievements.

The paper's symbolic analysis includes handshake secrecy against a future quantum attacker. Its computational security result is classical, and the ratchet theorems do not establish the post-quantum healing that SPQR is designed to provide. In other words, the quantum-related result is not absent, but neither is it a blanket guarantee of quantum-resistant recovery throughout an ongoing conversation.

This distinction matters when technical claims cross organizational boundaries. A research paper describes a model and assumptions; a marketing summary describes a benefit; a customer may read both as an unconditional guarantee. Someone has to preserve the scope. An incomplete verification campaign is not evidence that the remaining mechanism is broken. It means the available proof answers a narrower question.

For readers, the sensible response is neither blind confidence nor reflexive dismissal. Ask which claim has new evidence and which claim still relies on other analysis. Progress in security is often cumulative. A new proof can strengthen one part of the case without replacing testing, operational safeguards, or future research. The usefulness comes from knowing where it fits, not forcing it to carry every possible assurance at once.

Reproducibility is a product feature, too

DeepMind's public repository includes specifications, proofs, build instructions, and checking tools. It pins toolchain versions and warns that generated material depends on the exact compiler output. The instructions describe substantial memory requirements and checks that can take hours. The repository also states that it is not an official Google product. An available research artifact should not be confused with a turnkey commercial service.

For teams evaluating the result, that availability is more useful than a benchmark score alone. It creates a route to inspect what was actually demonstrated. Still, publishing the artifact and successfully reproducing it are different events. A responsible evaluation would record the environment, versions, resources, and outcome rather than assuming that an open repository establishes independent confirmation by itself.

The commercial opportunity, if this style of verification becomes easier to deploy, is therefore broader than selling access to a stronger model. Someone needs to package evidence so a customer can understand it, rerun it, and maintain it when the underlying software changes. Version mapping, exception tracking, and readable explanations of scope may be less glamorous than autonomous proof generation, but they help make assurance usable.

That is a hypothesis about an emerging service, not proof of established demand. A prospective customer might prefer simpler tests for low-risk code, or lack the expertise to specify a worthwhile theorem. The first useful offering may be a narrow component with an expensive failure mode and a requirement stable enough to justify the work. Starting there would make the economics measurable instead of treating all software as equally ready for formal verification.

The next milestone is a maintained guarantee

A one-time proof belongs to a particular state of a system. Shipping software is a moving target: dependencies change, features arrive, and yesterday's carefully constrained interface expands. The business question is not only whether AI can help establish a guarantee, but how much effort it takes to keep that guarantee attached to a living product.

A useful pilot would make that maintenance visible. Select a component, agree on the important property, record what the proof assumes, and measure the work required after a realistic change. Count expert review and failed attempts as well as successful generation. If a patch invalidates a guarantee, the release process should expose the break rather than quietly preserving an old badge.

This is where the story connects to the wider ambition of AI-written software. Faster generation increases the value of acceptance criteria that remain stable under pressure. Not every criterion will be mathematical. Usability, product fit, and many operational judgments resist a neat theorem. But where a precise, valuable property can be checked, the generator's fluency should not be the final authority.

AlphaProof Code is significant because it makes that division of labor more concrete. AI can contribute an artifact that other machinery can examine, while people remain responsible for the promise being made. The most consequential outcome would not be software that asks us to trust AI more. It would be software that arrives with a clearer account of what we do, and do not, need to trust.

11/12/2026

• AI Is Turning Software Updates Into a Race
• AI Agents Need Boundaries They Cannot Rewrite
• Coding Agents Need a Workplace, Not Just a Model