Self-hosting an open source LLM: the security trade-offs nobody mentions

Self-hosting an open source model is having a real moment. Llama, Mistral, Qwen and others have closed much of the gap with proprietary models, tools like Ollama and vLLM make deployment genuinely accessible, and the pitch is appealing. Your data never leaves your infrastructure, no per token costs, no dependence on a third party provider. Most of what gets written about this covers the setup, the hardware, and the cost breakeven point. Almost nothing covers the security trade-offs you are actually taking on when you make this move, and they are real.

This article is for the founder or engineer weighing self-hosting for genuine security or data residency reasons, not just cost. It covers what self-hosting actually solves, and the new risks it introduces that using an API from OpenAI, Anthropic, or Google does not carry.

What self-hosting genuinely solves

Start with the honest case for it, because it is real. If you have data that legally cannot leave your infrastructure, certain healthcare data, defence related work, some financial services use cases, self-hosting is not a nice to have, it is often the only path available to you. Your data never reaches a third party's servers, which sidesteps an entire category of questions about provider agreements, data residency, and what a provider does with your inputs.

There are genuine secondary benefits too. Costs become predictable rather than usage based, latency can be lower since you remove the round trip to a remote API, and you have full control to customise or fine tune the model on your own data. For the right use case, these are not marketing claims, they are real advantages.

The trade-off nobody frames properly

Here is the reframe that matters. Using a frontier model through an API means the provider is responsible for securing the serving infrastructure, the model weights, and the layer between your request and the model. The moment you self-host, all of that responsibility transfers to you. You are no longer just choosing a model. You are taking on the job of running a piece of AI infrastructure securely, and that infrastructure has its own attack surface that most teams have never had to think about before.

This is the trade-off that gets lost in the cost and setup guides. Self-hosting does not remove your security work, it relocates it, and it adds new categories of risk that a managed API simply does not expose you to.

The specific risks that come with self-hosting

• The serving layer itself can be a weak point. Tools like vLLM and Ollama are built and optimised for performance, not hardened as security products. An unauthenticated or poorly secured API endpoint on your serving layer can become an open door into your infrastructure, bypassing whatever security you have built around the rest of your product.
• Shared GPU environments can leak data between requests. If you run multiple workloads or serve multiple tenants on shared GPU infrastructure, there is a genuine risk of one request's data leaking into another through shared memory, sometimes called the KV cache, where fragments of previous conversations can potentially be exposed to a different session sharing the same resources.
• Your model weights become an asset that can be stolen. If you have fine tuned a model on your own data, those weights represent real intellectual property and potentially encode information about the data you trained on. A compromised serving layer can expose those weights to theft, which is a risk that simply does not exist when you call a hosted API.
• Elevated privileges create a path to your wider infrastructure. Serving engines often need elevated access to communicate with GPU hardware. If that layer is compromised, an attacker can potentially move from the AI component into the underlying host or the rest of your infrastructure, a form of lateral movement that a well isolated managed API does not expose you to.
• You inherit the patching and update burden. When a vulnerability is found in a serving framework, you are responsible for knowing about it and applying the fix. With a managed API, the provider handles this. Self-hosted, an unpatched serving engine is your unpatched serving engine, indefinitely, until someone on your team does something about it.

None of this means self-hosting is unsafe by definition. It means the safety is now entirely a function of how well you secure it, and that is a meaningfully different job than integrating an API.

What securing a self-hosted deployment actually requires

If you decide self-hosting is right for your situation, here is what genuinely needs to be in place, beyond just getting the model running.

• Authentication and network isolation on every serving endpoint. The API your serving engine exposes should never be reachable without authentication, and it should sit behind a proper reverse proxy or within a private network, not directly exposed to the internet.
• Least privilege for the serving process itself. The account and permissions the serving engine runs under should be as constrained as possible, so a compromise of the AI layer does not hand an attacker a path to the rest of your systems.
• Genuine isolation if you serve multiple tenants or workloads. If more than one customer or workload shares your infrastructure, you need real isolation between them, not just an assumption that requests will not interfere with each other.
• Monitoring for unusual behaviour at the infrastructure level, not just the application level. Standard code scanning and container checks are necessary but not sufficient. You also need visibility into what the serving engine itself is doing, since a compromised serving layer can behave normally from the application's perspective while quietly doing something else underneath.
• A real patching process. Track the serving framework and model runtime you use, and have an actual process for applying security updates promptly rather than running whatever version was installed on day one.
• Protection for the weights themselves, particularly if you have fine tuned on proprietary or customer derived data, since the weights can encode information about that data and represent real intellectual property.

This is genuine, ongoing security work, comparable in seriousness to securing any other piece of production infrastructure. It is not a weekend project that ends once the model responds to your first request.

The honest decision framework

Self-hosting is worth the overhead in a narrower set of situations than the enthusiasm around it suggests. It makes sense when you have a genuine data residency or regulatory requirement that a third party API cannot satisfy, when your usage volume is high enough that the economics clearly favour it, or when you need deep customisation an API cannot offer. In those cases, the security work described above is simply part of the cost of doing it properly, and it is worth paying.

It is often the wrong call when the primary motivation is cost alone, especially at low to moderate usage, because the security and operational burden can easily outweigh the savings for a small team without dedicated infrastructure expertise. It is also often unnecessary when your actual driver is data protection concerns that a proper Data Processing Agreement and the right provider settings, such as zero data retention, would already resolve without you taking on infrastructure you now have to secure yourself.

Many teams land on a hybrid approach, using a self-hosted model specifically for the sensitive workload that requires it, while continuing to use a managed API for everything else. That is often the most sensible position, rather than treating it as an all or nothing decision.

The honest takeaway

Self-hosting an open source LLM can be the right and sometimes the only responsible choice, particularly where data residency genuinely requires it. But it is not a security upgrade by default, and treating it as one is the mistake. It trades one set of risks, the ones that live with a third party provider, for a different set that now lives entirely with you, including risks in the serving infrastructure itself that most teams have never had to secure before.

If you are considering it, go in with your eyes open about what you are taking on, and make sure the security work around your serving layer gets the same attention you would give any other piece of infrastructure handling sensitive data. The model running locally does not mean the risk has gone. It means the risk has moved, and it is now yours to manage.