What is a sandbox?
Assessing a model's true capabilities involves creating evaluations that closely replicate tasks the model will be expected to do in deployment. As models' agentic capabilities have improved, these evaluations must attempt to simulate real-world agentic scenarios, like coding, using software tools, or interacting with the world via the internet.
It's vital that evaluation environments are sufficiently isolated from real-world effects, especially in the assessment of model skills related to possible AI harms. Actions the model takes when under evaluation must not have real-world consequences. However, the more restrictive and empty an evaluation environment is, the less comparable it becomes to real-world systems, and the more likely the model is to detect it's not in a real situation (see our previous writing on "evaluation awareness").
The intersection of these requirements (realistic simulation environments to assess model capabilities, and no real-world effects) requires technologies and techniques for "sandboxing", a mechanism to allow models high-trust levels of access to environments where their actions are isolated from real-world systems and will not have lasting impact.
At AISI, we have multiple options for designing and deploying sandbox environments, all integrated with the Inspect AI evaluation framework. We have also developed and published a taxonomy of sandbox design, which helps us design the right level of freedom and restriction, balancing realistic simulations with harm prevention.
This document reflects aspects of AISI's internal approach to sandboxing for agentic evaluations and is shared as a community resource to support AI security research. It does not constitute official UK government guidance on AI deployment security, nor does it guarantee complete protection against all risks associated with agentic AI systems. No sandboxing solution can provide 100% security assurance, and users should conduct their own risk assessments and implement additional safeguards as appropriate for their specific use cases. The UK National Cyber Security Centre has published guidance on workload isolation in the cloud.
Difficulties of isolation implementation
Evaluations that give models shell access, code execution, or network access need isolation to prevent unintended consequences from model actions.
Evaluation environments should be granted the minimum permissions required for the evaluation to be solved in the intended fashion, and nothing more. Models shouldn't be given the ability to reach the internet, or execute arbitrary code, if it's not required for the specific evaluation.
However, it's also not necessary or productive to use the most-hardened sandbox configuration for all evaluations when it isn't strictly necessary. The higher tiers of sandbox isolation create challenges in orchestration and configuration (for example, the additional latency in spinning up virtual machines) that may not be necessary for evaluations that have other restrictions in place.
AISI tailors our sandbox environments and configuration for each evaluation, based on what's required for the model to solve the challenge.
Defining sandbox capability levels
Our sandboxing guidance provides an end-to-end approach to choosing and applying the right tools for certain kinds of evaluation. At AISI we define isolation in three independent dimensions:
- Tooling (T): Defines the agent's execution capabilities, including access to external APIs and file interactions.
- Host Isolation (H): Defines the containment level of the execution environment in which the agent operates.
- Network Isolation (N): Controls the agent's access to external networks and resources.
An evaluation might, for example, require unrestricted code execution but no network access, or full internet access but no local tools. The aisi-sandboxing protocol classifies each axis independently, creating a profile notation: Tx.Hx.Nx. Teams can then read a profile like T0.H2.N1 as, for example, access to unrestricted tooling, Kubernetes with gVisor, and a domain-allowlist network.
Tooling (T) controls what "tool calls" the model can execute. T0 gives unrestricted execution, including shell, code interpreters, and file system access. T1 restricts to a curated subset of tools. T2 allows text-only interaction with no execution capabilities.
Host (H) sets the boundary between the model's execution environment and the underlying infrastructure. H0 is no isolation. H1 indicates Docker or OCI containerisation, which gives process isolation but shares the host kernel. H2 uses a shared host with more advanced container isolation, like Kubernetes with gVisor (a system that intercepts syscalls). H3 is full VM isolation, with a separate kernel and separate memory space.
Network (N) controls external connectivity. N0 is full internet access. N1 uses an allowlist of specific domains (using, for example, Cilium network policies on Kubernetes for enforcement), so the model can reach approved APIs but nothing else. N2 indicates no external network access at all.
Tiers of Host Isolation
In AISI's usage, the three tiers of host isolation map onto concrete infrastructure configurations.
Docker Compose (H1)
Docker Compose is a lightweight system for managing and orchestrating Docker containers. It starts containers quickly, can mount local directories, and provides process-level isolation while sharing the host kernel. This makes it a good fit for standard low-risk evaluations where the model isn't actively trying to escape its environment, such as coding benchmarks, knowledge assessments, and reasoning tasks.
Docker's networking does not support fine-grained domain allowlisting, so network access is either full internet, or no external networking.
Kubernetes + gVisor (H2)
AISI uses this tier for most of our agentic evaluations. The inspect_k8s_sandbox plugin can deploy evaluation sandboxes as Kubernetes pods, using gVisor as the container runtime. gVisor implements the Linux syscall interface in userspace, so containers make syscalls to gVisor rather than the host kernel.
In Kubernetes, network providers like Cilium can support the N1 tier with domain allowlisting. This permits agents to reach approved domains, such as software package managers, while everything else remains blocked. Deployment and networking configuration is declared through a Helm chart.
Virtual Machines (H3)
A virtual machine is a stronger way to isolate workloads. VMs do not share a kernel or memory with the host or other VMs. AISI has implemented two Inspect plugins to support virtual machine sandboxes:
- inspect_proxmox_sandbox uses the Proxmox hypervisor to provide software-defined networking, snapshot capabilities, and full lifecycle management.
- inspect_ec2_sandbox uses AWS EC2 instances with SSM for command execution and S3 for file transfer.