The combination of running AI agents which can install and run anything with supply chain attacks / the Hackening6 has made me much more paranoid about my personal machine than I used to be. I’ve been slowly improving this, and finally moved all of my AI agents into a VM today. I’m sharing this to show how far you can take this, even if you have requirements like GPU access or the ability to run (something close enough to) Docker.

This is a follow-up / longer version of a recent shortform7.

Diagram of a workstation (host). Outside it are two boxes: "My devices (Tailscale)" and "Internet". Inside the host, an agent VM (KVM, 20 vCPU, 64 GB), whose QEMU process runs unprivileged and AppArmor-confined. It contains: agents and their web UI, as a user without root; rootless Podman for their containers; their credentials; a separate admin account agents can't reach; its own Tailscale node serving the web UIs; and a phone passed through over USB. Its firewall allows the internet and one port to the GPU box. Next to it, a GPU box: its own no-login account running rootless Podman, containing a gVisor sandbox with SSH reachable only from this machine, a GPU job queue and jobs, a storage key scoped to one bucket, and nvproxy to the GPU driver. Its firewall allows the internet only. Along the bottom: my account (desktop and browser, not in the docker or libvirt groups, sudo needs a password); text-to-speech (Docker, GPU worker in gVisor, no network); and the GPU (8 GB, shared with the desktop). Arrows: my devices to the VM, labeled "web UIs over Tailscale"; the VM and the GPU box each to the internet; the VM to the GPU box through an SSH proxy; the VM to text-to-speech; the GPU box and text-to-speech to the GPU. Footer: Secure Boot, Livepatch, unused kernel modules blocked, firewall inbound only over Tailscale.

Goals

I run Claude Code in a custom harness8 which currently disables all permission prompts. I’m not really that worried about Claude doing something dangerous on purpose, but I worry about:

  • Claude breaking something by accident (mistyping a command that turns into rm -rf /).
  • Claude installing malware (most likely via a supply chain attack).

My biggest worry is kernel exploits giving attackers access to my user account, allowing them to install a keylogger and steal my password manager vault1.

Claude needs the ability to run Docker for testing, and shared access to my GPU for ML work.

Non-Goals

To get its work done, Claude needs access to many API keys (GitHub, OpenRouter, etc.). These keys are already as limited as they can be, and I limit the amount of API credits each key can access anyway because Claude doesn’t understand the value of money.

Agent VM

I moved Claude agents to an unprivileged user account inside of an unprivileged VM. Any user that can run Docker can easily get root access (and root access within the VM makes sandbox escape much easier) so agents use rootless Podman9 instead.

GPU drivers allow direct access to hardware, and assigning the GPU to a VM would prevent my computer from being used by humans, so the VM can’t access the GPU directly.

Clawed Abode web interface: a chat session titled "Server security hardening review", showing one of Claude's replies and a message box.

I access the UI for my agent harness from multiple devices over HTTPS via Tailscale10. Agents can serve files by putting them in a specific folder, and I can access files directly using VS Code for the Web11.

GPU Container

For GPU usage, agents use my GPU queueing tool12 over SSH into an unprivileged container using gVisor13 + nvproxy. This has an allowlist for Nvidia driver calls, which reduces the attack surface14 for driver exploits (although imperfectly).

The gpuc web dashboard. One host, "local", is reached over SSH at root@192.168.122.1, with its dispatcher seen 1 second ago and 0 of 1 GPUs free. Its RTX 3060 Ti 8 GB is busy, using 2404 MiB at 45% utilization. One job is running, normalizer-cleanups-listen, 11 seconds in, with Logs, Estimate, Limit, Preempt and Cancel buttons. Five finished jobs are listed from a day ago: four succeeded and pythia-smoke2 failed with exit 1.

Monitoring the GPU container via Tailscale.

Production TTS Containers

I run a Text-to-Speech service for my RSS reader15 on this machine, since everything commercially available is either weirdly slow2 or ridiculously expensive. I decided to run Kokoro-82M16 locally, but didn’t want to expose a service containing C code3 to the internet. I had Claude rewrite the C code in Rust17, and (just in case) heavily sandbox the code with GPU access:

Diagram of optimized-kokoro's deployment. At the top, a "Clients" box: Lion Reader, other apps, the demo page. An arrow labeled "text, HTTPS" runs from the clients down to the server, and one labeled "24 kHz PCM" runs back up. Inside a host box are two containers side by side. The server (Rust), a gVisor sandbox with 3 GB and 256 pids, handles HTTP, API keys and rate limits; text cleanup and sentence splitting; text to phonemes in all 9 languages, with espeak-ng and misaki ported to Rust; and a priority queue that streams audio back. It has no unsafe code of its own. It reads config and API keys (as secrets) and UniDic, and writes nothing. Its network is a bridge with a port published on 127.0.0.1:8880, with Tailscale or a Cloudflare tunnel in front. The worker (PyTorch), a gVisor sandbox with 12 GB and 4 CPUs, runs Kokoro-82M in fp16 with torch.compile, batches queued requests, and re-validates every message. It never sees text or HTTP, only arrays of phoneme IDs. It reads model weights and a compile cache, and writes a 2 GB /tmp for the Triton cache. Its network is none, highlighted. It reaches the GPU through nvproxy (a CDI device). Below them, a shared volume: a 1 MB tmpfs owned by nobody, mode 0700, holding only worker.sock. Arrows labeled "phoneme IDs, audio" run from the server into the volume and from the volume to the worker. A separate arrow runs from the worker to a GPU box labeled "host NVIDIA driver." Footer: host runs rootful Docker with runtime runsc --directfs=false --host-uds=all; every container runs as user nobody with a read-only root, no capabilities and no-new-privileges.

The only port opened to the internet (via Cloudflare Tunnel184) is written entirely in Rust and has no privileges or GPU access. It talks to a separate container with GPU access (through gVisor + nvproxy, like my other GPU container) over a very simple binary protocol where the only inputs are tokenized phonemes (a list of numbers).

I’m still unsure if this piece is sufficiently paranoid, but I really like having fast TTS and don’t want to spend $200+/mo for a cloud GPU (or to send my users’ data to other people’s gaming machines19).

Gotchas

Segfaults in the agent VM

This may have something to do with faulty hardware on my machine5, but switching from host-passthrough to host-model in libvirt fixed it for me.

File descriptor leak in gVisor20 with directfs=false

I was using this to avoid running the sandbox as root. Moving the GPU container to rootless Podman with directfs worked instead.

Other Security Hardening

  • Kernel updates are applied without restarting via Ubuntu Pro21 / Livepatch22 (free for up to 5 machines, although the agent VM counts for one).
  • Secure Boot is enabled, using Canonical’s signed driver modules.
  • Several unused kernel modules are disabled, especially network protocols that I don’t use (sctp, rds, tipc and n_hdlc).
  • I removed my own account from the docker group to limit malware’s impact on my own account, although a keylogger could still get past this.
  • A firewall blocks all inbound requests except via Tailscale, and my Tailscale policy is fairly locked-down now: My personal devices can access each other via advertised services and (key-only) SSH; the agent VM can only access the TTS service.
  • Package managers proxy installs through Socket Firewall23 (free), which warns or blocks known malware in packages.

Next Steps

The biggest remaining security improvement for me is probably running every application in a separate VM24, or at least running my browser in a VM (although I already use the Snap version of Firefox25, which does some sandboxing26).

I’m also slightly worried about Canonical or another vendor I depend on getting hacked and putting malware in the official repos. I haven’t figured out how to mitigate this yet.

Summary

If you’re sufficiently paranoid, it’s possible to have AI agents running on your machine while sandboxing them to a pretty acceptable level, even if they need GPU access and tools like Docker.

  1. I intentionally don’t keep 2FA keys in my password manager vault, so this wouldn’t be entirely catastrophic, but most of my accounts don’t use 2FA and it would still be very annoying to change hundreds of passwords. ↩

  2. OpenRouter claims that Kokoro-82M’s latency is <300 ms27, but in practice I was seeing several seconds, especially for longer inputs. My local machine using a low-end previous generation GPU responds in ~50 ms and processes at 200× realtime. ↩

  3. espeak-ng28 CVEs that haven’t been fixed in a released version: CVE-2023-4999029, CVE-2023-4999130, CVE-2023-4999231, CVE-2023-4999332. ↩

  4. I chose Cloudflare Tunnel over Tailscale Funnel because it adds additional protections in front of your service, including an API key handled by Cloudflare before hitting a second API key handled by my service. ↩

  5. My CPU is known to be flaky in some situations. ↩