This post covers a shortlist of AI security problems with scope for existential harm but few experts. The goal of these is to encourage more people to work on them, especially those with relevant expertise.
If you have a relevant security or ML background, I encourage you to work on these problems! If you’re earlier in your career or from a different field, but interested anyway, I encourage you to investigate these problems and find compelling directions that fit your skill set.
Each section will summarize the threat model, state of play, and proposed research directions – it’s a non-comprehensive summary to help you get oriented before going deeper.
For more analysis on how I chose these problems, read here.
In this article:
Securing AI from exfiltration
Protecting model weights and algorithms from theft
What we want to avoid:
A terrorist actor gains access to a powerful model without guardrails or safety training, and uses it to create a bioweapon
A well-resourced state or malicious actor exfiltrates a model capable of recursive self-improvement, gives it access to compute without appropriate safety or control measures, and uses it to create a superintelligence that it loses control over
A malicious state actor gains access to a frontier model and deploys it in the military, for surveillance, and for economic purposes to establish stable totalitarianism
A rogue AI system self-exfiltrates and proliferates, leading to catastrophic loss of control
What’s happening right now:
Securing AI Model Weights lays out what’s necessary to prevent the theft of model weights by a malicious human actor, with graded security levels corresponding to resilience against low-resource, low-skill attackers (SL1) to expert nation state attackers (SL5).
SL5 Task Force has been researching how to turn SL5 into an adoptable standard for AI companies, and Intelligence Security Lab is developing highly secure clusters. AI lab security teams are pursuing increased security but trying to avoid slowing down development. Some leading models from outside the U.S. are completely open-weight and unprotected.
Relevant backgrounds:
Experience building SL5 environments
Experience developing security standards
Experience conducting security audits for high-risk environments
Experience with securing AI infrastructure (datacenters and AI hardware)
Possible directions:
Mapping out threat actors, kill chains, and their likelihoods at various levels of security and model capability
Adapting national security paradigms to a frontier lab’s development requirements
Developing or commissioning R&D, for example on projects similar to those scoped in SL5 Novel Recommendations - Security Framework
Thinking through the adoption plan for higher security standards
Concrete project ideas:
Map out kill chains for self-exfiltration under SL3-4 controls
Design new network security protocols for inference servers to prevent exfiltration
Securing AI integrity
Protecting models from tampering via data poisoning or other attacks
What we want to avoid:
A lab insider threat with legitimate access to the training pipeline adds a password-based backdoor that they can trigger once the model is widely deployed in critical systems.
A state or terrorist actor creates public internet data to poison the model, causing it to, e.g. write vulnerable code when in a certain nation, or render the model useless when it sees a password. This backdoor allows them to gain a meaningful upper hand in military conflict, possibly leading to stable totalitarianism via military force.
A government actor requires an AI lab in their nation to insert a backdoor in their model, the model gets widely deployed in military and other critical systems, and the backdoor is used maliciously.
What’s happening right now:
Academic research on creating and detecting backdoors is popular, and encouraged by the IARPA project TrojAI. Labs research this risk and develop training data filters to try to avoid data poisoning, but consuming public data on Twitter was enough to create a guardrails-removing backdoor.
On the policy side, the Institute for Progress published a post on sleeper agents, and the Institute for AI Policy and Strategy released a paper on model integrity recently, but this issue generally has not reached high salience in Washington. Tom Davidson has done futurist macrostrategy on how secret loyalties in AI could be used to enable a coup.
Relevant backgrounds and knowledge:
Experience securing high-stakes development environments (e.g. at a government contractor or government cloud service provider)
Experience with AI training pipelines (e.g. working at or contracting for a large AI training project)
Experience with backdoor research
Experience with insider threat mitigation
Possible directions:
Work with ML researchers to identify the most dangerous types of backdoors possible today, and forecast when more dangerous secret loyalties might be possible in the future
Research different threat actors and their kill chains
Security research into how to prevent these kill chains and implement mitigations
Weighing security risks and concentration of power risks for any particular mitigation
A game plan for how to socialize this risk to appropriate stakeholders, establish standards for preventing this risk, and motivate labs to meet the standard
Concrete project ideas:
Red-team frontier lab data filtering methods for pre-training and post-training
Define forensic signatures of post-training data poisoning, based on current data management practices
Disconnected model safety
Preventing models that are not being served by AI labs from being used for catastrophic purposes
What we want to prevent:
A terrorist actor uses a stolen or open-source model after removing guardrails, e.g. to create a bioweapon
A state actor steals a non-deployed frontier model to supercharge its state bioweapons or cyber offense program
A model that has been backdoored or tampered with is used at scale to build military force and enact a coup, against the model’s original use policies
What’s happening right now:
Today, all checks to prevent misuse happen when labs serve a model via API, which allows them to enforce authorization and apply live safety classifiers. Once a model has become open source or exfiltrated, there are no oversight mechanisms besides whatever alignment has been trained into the model itself. There are currently many examples of overriding this safety training (via jailbreaking or fine-tuning).
The FlexHEG (flexible hardware-enabled guarantees) project is one approach to this problem — using on-chip mechanisms to provide oversight even when the model is disconnected. FlexHEG is quite early stage, broad, and may require meaningful chip redesign.
There are also machine learning-based approaches directly on the model, like tamper-resistance, jailbreaking resistance, and potentially password-locked models (this paper explores the mechanism for other purposes but demonstrates feasibility).
The layers between the model and the hardware are also intervention points: safeguards could be deployed by cloud providers or by various infrastructure providers. This might not apply to local compute, but could cover large-scale compute use cases.
Relevant backgrounds and knowledge:
Experience with ML research on relevant misuse prevention
Experience with datacenter supply chains and orchestration
Experience with chip or server hardware, or firmware
Some possible approaches:
Deploying confidential safety guardrails on chips or via datacenters, that can prevent extreme misuse without surveillance
Requiring KYC for training or inference processes over a certain compute threshold
Storing or training high-capability models such that they will break if a user is not authorized to do inference or fine-tuning on them
Research the feasibility of tamper-resistant guardrails or password-locking models
Concrete project ideas:
Design a retrofittable hardware or firmware intervention that can attest to a safety classifier being run at inference time
Estimate the amount of compute needed for catastrophic activities (CBRN, cyberattack)
Secure compute verification
Verifying that large-scale compute workloads are compliant with law or agreements
What we want to prevent:
We agree about the laws and policies AI should follow, including for recursive self-improvement, but are unable to verify whether these policies are being followed, leading to catastrophic misuse or loss of control
Major AI labs and world governments want to sign an international treaty to slow down or limit future AI development, but are unable to verify that the other side is not defecting, and therefore continue to race to develop unsafe AI
What’s happening right now:
Verifying International Agreements on AI lays out potential strategies for trying to enable an international agreement similar to a nuclear non-proliferation agreement. These strategies require a large amount of technical hardware mechanisms and governing bodies that do not currently exist. Some small research groups at FLI, RAND, and in private industry are working to advance the technical R&D to verify, at the hardware level, whether compute is being used for its claimed tasks.
For use in an international treaty or against a malicious, highly resourced actor, this technology must be extremely secure in an adversarial environment.
Relevant backgrounds:
Experience with datacenter and hardware security
Experience with AI workloads at a low level
Experience with intelligence work and nation-state security
Possible approaches:
R&D on network taps, power sensors, and on-chip mechanisms that could possibly implement this technology
Red-team the security of this technology and develop a roadmap for getting to SL5
Develop a DARPA-style plan to commission experts to develop sub-systems
Assess the barriers for datacenters worldwide to adopt this technology and develop plans to accelerate adoption
Assess the supply chain risks of building this technology and develop plans for secure production
Engage with experts on nuclear non-proliferation and think about the inspection organizations that need to be in place for an agreement to be verified
Learn about how the intelligence community and international governance bodies would interact with this technology
Addressing the dark compute problem: how can we ensure that the majority of compute takes part in the verification agreement?
Concrete project ideas:
Quantify dark compute: how expensive would it be to hide large-scale compute, and how could AI help with both evasion and detection?
Develop a prover for confirming that a declared workload complies with an agreement (eg, only inference, training only under a specific threshold, no automated AI R&D), etc.
Protocols for using compromised AI
Also called AI control – how to use untrusted AI to do useful work
What we want to avoid
A backdoored AI doing the majority of AI R&D at a major lab is propagating its backdoor to the next version of itself. After the model reaches a high enough level of capability and deployment, the backdoor is triggered by its author, enabling stable totalitarianism
A secretly scheming AI is doing the majority of AI R&D at major labs, and sabotaging safety and alignment processes for the next version of itself, leading to loss of control when the model reaches something like ASI
What’s happening right now
Redwood Research has done work on this threat model and potential protocols, as part of their work on AI Control. Redwood identifies rogue deployments as one of the key cruxes in AI catastrophe. Another Redwood post identifies the risk of using AI to rewrite AI infrastructure for automated R&D as a major threat model.
Relevant backgrounds:
Experience as a security architect in a complicated development environment
Experience as a security protocol designer
Experience as an offensive security researcher
Experience with formal verification
Possible approaches:
Understand the state of AI Control research and where security expertise can add value
Think about adopting security paradigms for the novel type of insider threats posed by AI “employees”: internal actors that are numerous and tireless, but also have extremely limited access to the outside world and can be directly modified.
Think about the highest-risk parts of AI R&D for this threat model and how a combination of human and weak trusted AI workers can protect against highly capable untrusted AI taking dangerous actions
Concrete project ideas:
Develop a prioritized list of AI infrastructure to be formally verified, and write security specs for these systems designed for AI R&D by untrusted AI
Develop a kill chain for an AI to accomplish a rogue deployment, and use this to write transcripts of suspicious behavior that monitors can be trained on
Detecting and responding to rogue deployments
What to do if we fail to contain transformative AI and it gets deployed (or deploys itself) without oversight
What we’re trying to prevent:
Building off the previous problem statement, which would ideally prevent a rogue deployment, what would we do in the case of failure?
This might be a rogue deployment within an AI company, orchestrated by a rogue AI or human insider threat, leading to loss of control
It also could be a rogue deployment outside of an AI lab, either at another company or some other diffusely controlled compute. This deployment could be orchestrated by a self-exfiltrated model or a malicious actor who has stolen the model, leading to stable totalitarianism or catastrophic misuse
What’s happening here:
Evaluating Select Global Technical Options for Countering a Rogue AI by Michael Vermeer suggests that most promising interventions for a rogue deployment have large collateral damage.
Relevant backgrounds:
Experience in offensive security
Experience with strategic research on global networks (internet, telecom, IoT), likely in a national security context
Experience with global threat tracking and emergency response
Possible approaches:
How would we identify a rogue deployment while it was happening? Think about unhackable signals of computation (overlap with secure verification work) or incidents that could be attributed to a rogue deployment
Think about tensions between monitoring and concentration of power — are there privacy-preserving ways to monitor for rogue deployments? (overlap with authorization solutions for misuse)
Think about how to set compute thresholds for monitoring, and how a malicious actor might try to circumvent them
What’s the minimum compute amount and connection a malicious actor would need to do existential harm?
What options are there to shut down a rogue deployment?
Concrete project ideas:
Evaluate whether a “model fingerprinting” system for network detection could be resilient to fine-tuning or other tampering
Draft an incident response playbook modeled on NIST incident response