I Gave My AI Assistant SSH Access to My Servers. Here's What Happened.

Let me tell you something that would make most sysadmins break out in hives: I gave an AI full access to my production servers. Docker socket. Root SSH. The whole thing. Not a sanitized sandbox. Not a read-only playground. Real infrastructure, running real services for real clients. And I’d do it again.

The Setup

I run a small digital agency called Sweet Media. Like a lot of small shops, my “IT department” is just me — a designer who learned enough DevOps to be dangerous. My homelab server, sweetmediabox, runs about 20 Docker containers: Jellyfin for media, n8n for automation, Nginx for reverse proxying, PostgreSQL databases, VPN servers for clients, and a bunch of other stuff I’ve bolted on over the years. A few weeks ago, I set up an open-source framework called OpenClaw that lets you run an AI assistant on your own infrastructure. I named mine Leo. Within hours, I’d mounted the Docker socket into his container and gave him access to everything. My wife thought I was crazy. My developer friends thought I was crazy. Even Leo thought it was a bit much at first. They were all wrong.

Week 1: The Trust Test

The first real test came almost immediately. I had three VPN servers running for clients — one on Ubuntu 18.04. Yes, 18.04. The kind of technical debt that keeps you up at night but never quite makes it to the top of the priority list. I told Leo: “Upgrade this server to 24.04. Don’t break the VPN.” Here’s what a human contractor would have done: asked me for SSH credentials, scheduled a maintenance window, maybe backed things up, and then charged me for 4 hours of work. Here’s what Leo did:

SSH’d into the server
Audited the current state — packages, kernel version, running services
Identified the VPN configuration (OpenVPN Access Server vs Community Edition)
Created a rollback plan
Performed the upgrade through two major versions (18.04 → 20.04 → 22.04 → 24.04)
Verified the VPN was still functioning
Documented everything in a memory file I could review later

Total time: about 45 minutes. Total cost: whatever Claude API tokens cost for that session. The VPN stayed up. The client never noticed.

Week 2: When Things Break at Midnight

Here’s where it gets interesting. I run several containers behind a VPN gateway called Gluetun — it routes traffic for my torrent client, Prowlarr, and FlareSolverr through a VPN tunnel. One night, something went wrong. Gluetun got recreated (maybe a Docker update, maybe I fat-fingered something), and suddenly qBittorrent, Prowlarr, and FlareSolverr all went dark. The problem? When Gluetun gets a new container ID, every container using network_mode: container: breaks because they’re pointing to a container that no longer exists. Classic Docker networking headache. I was busy with client work. Leo wasn’t. He diagnosed the issue through the Arcane Docker management API (another tool on the homelab), identified the stale container ID reference, and fixed it — creating a new qBittorrent container with the correct network reference, all the right volume mounts, labels, and environment variables. By the time I checked on things, everything was back up. Web UI responding. Torrents downloading. No intervention needed.

The Things That Surprised Me

The Good

It’s faster than context-switching. The biggest cost of managing infrastructure isn’t the actual work — it’s the mental overhead of switching from design mode to sysadmin mode. When a server needs attention, I don’t have to drop what I’m doing, SSH in, remember which config file is where, and hope I don’t break something. I just tell Leo. It documents everything automatically. Leo keeps daily memory files of what he did. Every server change, every fix, every decision. I’ve never been this organized with my infrastructure. When something breaks again in three months, we’ll have a record of exactly what we changed and why. It learns your infrastructure. After a few weeks, Leo knows my Docker setup better than I do. He knows which containers depend on which, where the compose files live, what ports everything uses. He built a mental map of my homelab that I never bothered to create myself. It catches things you miss. Leo noticed that my qBittorrent container was in a “created” state — it had never actually started after the last compose update. I wouldn’t have noticed until I tried to download something and wondered why nothing was working.

The Risks (Let’s Be Honest)

It can break things. An AI with SSH access and the wrong prompt can rm -rf your life away. This isn’t theoretical — you have to trust the framework’s safety rails AND the model’s judgment. Leo is configured to use trash instead of rm and to ask before anything destructive, but still. You can’t verify everything in real time. When Leo is running commands at 3 AM, I’m not watching. I trust the audit trail, but I can’t pretend I review every command before it executes. That’s the trade-off. API costs can spike. Complex debugging sessions with lots of back-and-forth (checking configs, trying fixes, verifying results) can burn through API tokens. It’s still cheaper than a contractor, but it’s not free. It’s not a replacement for understanding. Leo can fix my Docker networking, but if I don’t understand why the fix works, I’m just outsourcing my competence. I try to review his solutions and actually learn from them. Sometimes I do. Sometimes I just say “ship it” because the client is waiting.

The Numbers

Here’s what my first month looked like:

Servers managed: 4 (1 homelab + 3 VPN servers)
Docker containers monitored: 20+
Major tasks completed: Ubuntu upgrade, VPN configurations, container networking fixes, full homelab audit
Incidents resolved without my intervention: 3
Things broken by the AI: 0 (so far)
Cost: ~$30-50/month in API usage vs $100+/hour for a freelance sysadmin

Should You Do This?

Honestly? It depends on your risk tolerance. If you’re running Fortune 500 infrastructure with compliance requirements, absolutely not. Go hire a proper ops team. But if you’re like me — a small business owner who’s also the accidental IT guy, running a homelab that’s slowly become production infrastructure — an AI assistant with server access is genuinely transformative. It’s like having a junior sysadmin who works 24/7, never forgets to document things, and costs less than your Netflix subscription. The key is setting it up right:

Use a framework with safety rails (not just raw API access)
Start with read-only access and expand as trust builds
Keep audit trails of everything the AI does
Have rollback plans for critical operations
Don’t give it access to things you can’t afford to lose (yet)

What’s Next

I’m now building a full “Mission Control” dashboard to manage all my projects through one interface, and Leo is helping architect it. I’ve given him access to my ClickUp, Stripe, and Docker APIs. Next up might be email access. My developer friends still think I’m crazy. But they’re also asking me how to set it up.

This is Part 2 of my “Building in Public” series. Part 1: De Diseñador a Emprendedor en Serie: Mi Historia Have questions about AI-assisted infrastructure management? Drop a comment or find me on LinkedIn. — Juan

Juan Perez

I Gave My AI Assistant SSH Access to My Servers. Here’s What Happened.