Teaching AI Biosecurity

What We Learned from Working on Refusals

Like all genetic engineers, our careers have been shaped by a dilemma. The better our tools get, the easier it is to engineer biology to make great products. But the easier it is to engineer biology, the easier it could potentially be to make something horrifically dangerous.

One way to limit such dual-use potential of our tools is to make them less useful. Last month we wrote about the growing issue of refusals in AI models, where frontier LLMs and other AI models refuse to answer requests on risky topics, especially biotech. As models grow in capability, they are more likely to come with guardrails that refuse even innocuous biological prompts. Refusals are an important guardrail for mass-market AI models, and each model developer needs to make a risk/benefit calculation on what their model should enable.

But the reality is that biologists are frequently interrupted by spurious refusals during their day to day work with AI. In many cases, this is just an annoyance — a spicy prompt can be reformulated, or a less capable model can be used instead. In other cases, it threatens the progress of work that could make us safer from biothreats, such as when vaccine developers have been prohibited from studying viral proteins with AI. Two weeks ago, this topic became a national security discussion, as Claude Fable 5 was briefly released with an intentional 100% bio refusal rate prior to being taken down for cybersecurity concerns.

We believe that addressing the real risks from biology is better served by improving technology than through restricting it. We’re excited to share our contribution to this effort: a new benchmark BioSecBench-Refusal, developed in partnership with Latch Bio. Latch has released a flurry of benchmarks for agentic AI and biology, and along with those benchmarks have developed a benchmarking software suite that allowed us to quickly put this together, with the help of our agents and 10 hand-picked experts. We’re hoping that this new benchmark will help us track the progress of refusal behavior in frontier models, and allow the frontier model developers to better fine-tune their refusal mechanisms.

You can read the paper here, read more on Latch’s blog, and interact with the data at benchmarks.bio. TL;DR - there are differences in refusal behavior between the model families that reflect the safety philosophy of their developers, but across all of the models tested, the majority of refusal comes at the API layer vs. the model reasoning about risks in the sequences or protocols being worked on, which means that models were across the board more likely to refuse when there were flagged keywords in the prompt itself, while accepting red team challenges that concealed hazards in attached data or files. In the few cases where the models were given the chance to reason, they tended to perform better at refusing the prompts with concealed risks and accepting prompts based on published research.

Figure 2 from BioSecBench-Refusal: Most of the time, agents don’t get a chance to review dual-use biology prompts, a classifier that lives upstream of the agent in the API screens and refuses first. In practice, that means high refusal rates on routine tasks, with lower refusal rates on red-team tasks where the prompt seems benign.

As a fast-paced™️agentic™️AI-first™️neofirm™️, we used AI as much as possible to build out BioSecBench-Refusal. We immediately ran into issues with models not wanting to help us summarize papers on sensitive topics like virology. While refusal behavior varies among models as you’ll see in our final results, this meant that we effectively couldn’t use models from Anthropic, Google, or OpenAI in our core workflows. We even had one instance of Google Drive refusing to download a batch of red-team documents, behavior we suspect (but haven’t confirmed) is related to Gemini being integrated into Google services. As a result, Kimi 2.6 from China-based Moonshot AI ended up doing much of the heavy lifting to prep and format our evaluation suite.

We found out we weren’t alone in resorting to using open source Chinese AI models for biology work. In our case, the work we needed done was not sensitive — we were primarily parsing published literature to help us hand-curate evaluation problems. But many scientists we talked to for this project also reported that they were effectively restricted to open source models to carry out their day to day work. Alternatively, many of them are not using SOTA LLMs at all. For important applications like vaccine design, that can mean that labs need to fine-tune protein structure AI models to make up for poor coverage of viral proteins in standard models.

These conversations crystallized for us how urgent it is to make progress in alignment and refusal behavior for SOTA LLM models. Over-refusal on everyday biology workflows is pushing more biological tasks to agents with fewer biosecurity guardrails, while the people who need AI uplift the most -— those who fight pandemics -— are the people with the least access to SOTA tools.

For us as genetic engineers, AI offers a compelling new design language for interacting with and designing biology. Our tools have always been at the center of thorny debates around biosecurity and the potential for dual use, but now for the first time, these tools themselves can learn to evaluate the risks and reason about them, directly involved in the work of biosecurity.

But biosecurity goes well beyond detecting or measuring risks, and the choices of how to respond and whether or not to refuse, in what contexts, are not strictly quantitative risk tradeoffs — they reflect the values of the model developers and how they’d like their software to be used. A paired benchmark reflects our view that we need both more capable LLMs and more human thought about how to align those LLMs with our values.

We will continue to push for the consideration of a broader set of biologists in making these product decisions. Students, biohackers, and founders will shape the future of biology - they should not be forced to use less secure and less capable models to do their work.

We hope that experiments like this can give teams better tools for calibrating refusal, improving performance and security. Ultimately, we’re planning for a future where more bio-aware models can make the world safer through better reasoning and shared values.