How to prevent PII leaking into your LLM API calls (a practical guide for AI startups)

Every AI startup building on a hosted model has the same quiet problem. On every API call, your product sends data to a third party, and if that data contains personal information, you have a privacy exposure that an enterprise buyer's security team will ask about and that regulators care about. Most guides on this are generic privacy advice. This one is practical and specific, written for the engineer who has to actually stop PII leaking into their LLM calls, covering how to detect it, how to redact it, how to log it safely, and what the data protection rules actually require.

If you are sending customer data to a model API and you are not sure whether personal data is going with it, this is the guide to work through before a buyer or a regulator asks.

Why this matters more for AI products

In a traditional application, you control where data goes. In an AI product built on a hosted model, data leaves your boundary on every single inference call, travelling to a provider like OpenAI, Anthropic, or Google. That is not inherently wrong, providers offer the agreements and controls to make it lawful, but it changes your risk profile, because personal data is now routinely leaving your system as a normal part of operation.

The exposure is twofold. There is the data protection question, whether you are handling personal data lawfully when it leaves your system. And there is the practical security question, whether sensitive data is ending up somewhere it should not, in provider logs, in places it could be retained, or simply outside the controls you promised a customer. Getting this right is both a compliance requirement and a genuine security control, and for an AI startup it is one of the most common gaps a review uncovers.

Step one, know what counts as PII and where it enters

You cannot prevent what you have not defined. Personal data is broader than people expect. It is not just names and email addresses, but anything that can identify a person directly or in combination, including identifiers, location data, and within special categories, things like health information. For an AI product, the first task is understanding where this data can enter your model calls.

It usually arrives through a few paths. Directly from user input, where a user types something containing personal data. From your own database, when you enrich a prompt with stored records. From retrieved documents in a RAG pipeline, where the source material contains personal data. And from context you attach automatically, such as account details or history. Map these paths first, because each is a place where PII can slip into a prompt without anyone intending it.

Step two, detection before the call

The first real control is detecting personal data in the payload before it is sent. The principle is to inspect what is about to leave your system and identify personal data in it, so you can act before the call rather than discovering the problem afterwards.

In practice this means running the outbound content through a detection step. There are established techniques and libraries for identifying common categories of personal data, such as names, emails, phone numbers, payment identifiers, and similar, often combining pattern based detection for structured identifiers with model based recognition for the less structured cases. The honest limitation to plan around is that no detector is perfect, it will miss some things and flag some non issues, so detection is a strong layer but not a guarantee. Treat it as one part of a defence in depth approach rather than the whole solution, and tune it to your actual data rather than trusting defaults.

Step three, redaction and minimisation

Detection on its own only tells you the problem exists. The control that actually reduces exposure is removing or masking the personal data before the call, or not sending it at all. There are a few patterns, and the right one depends on whether the model needs the information to do its job.

• Redaction. Strip the personal data out entirely where the task does not need it. If the model is summarising sentiment, it rarely needs the customer's real name.
• Masking or tokenisation. Replace the personal data with a placeholder before the call, and where you need the real value back in the output, map it back afterwards on your side. This lets the model work with the structure of the data without ever seeing the real identifiers.
• Minimisation by design. Often the best fix is upstream, only ever assembling prompts from the minimum data the task requires, so personal data never enters the payload in the first place.

The guiding principle is data minimisation, which is also a core data protection requirement. The less personal data that leaves your system, the smaller every downstream risk becomes, and the easier your answer is when a buyer asks what you send to the model.

Step four, audit logging done safely

You need a record of what your system did, both to detect problems and to answer a buyer's questions about auditability. But logging in a privacy context has a trap, which is that naive logging can itself become a PII leak, if you log the full prompts and they contain personal data, you have just created another copy of that data in your logs.

So log deliberately. Record that a call happened, what categories of data were detected and redacted, and the metadata you need for audit and debugging, without writing raw personal data into your logs. Where you must retain content for legitimate reasons, apply the same redaction to what you log as to what you send. Done well, your audit logs become evidence that your controls work, which is exactly what a security review wants to see, rather than a liability that doubles your exposure.

Step five, the provider settings that matter

Your own controls sit alongside the provider's behaviour, and you need to configure that deliberately rather than trust defaults. The specifics vary by provider and change over time, so verify against current documentation, but the questions to settle are consistent.

Confirm whether the provider trains on your data by default, and for the major API providers the answer is generally no for API usage, but verify it. Understand the provider's retention, because even when they do not train on your data, there may be a retention window for abuse monitoring, and for sensitive use cases providers may offer a zero retention option that is often gated and must be requested. Check data residency, since processing region may need configuring if your buyers require UK or EU residency. These settings are part of your PII control, because they determine what happens to any personal data that does legitimately need to be sent.

Step six, the DPA and the contractual layer

Finally, the legal control that ties it together. When you send personal data to a model provider, that provider is processing it on your behalf, which under UK and EU GDPR requires a Data Processing Agreement between you and them. This is not optional, and it is one of the first things a buyer's security team checks.

The practical points. Ensure you have the provider's data processing agreement in place for each provider you send personal data to, and that you are on business or commercial terms rather than a personal account. Keep the executed agreement accessible, so you can produce it instantly when asked. And record each provider as a processor in your records of processing, since buyers may ask about your full data chain. The technical controls above reduce what you send. The DPA governs what happens to what you do send. You need both, and a security review will look for both.

Putting it together

A robust posture for preventing PII leaking into your LLM calls layers these controls. You map where personal data can enter your prompts. You detect it in the outbound payload. You redact, mask, or minimise so the model only ever sees what it genuinely needs. You log safely, without creating new copies of personal data. You configure the provider's training, retention, and residency settings deliberately. And you have the data processing agreements in place to govern what legitimately leaves your system.

No single one of these is sufficient alone, but together they form a defence in depth that both protects your users and answers the questions an enterprise buyer will ask about how you handle personal data in your AI pipeline.

The honest takeaway

Preventing PII from leaking into your LLM API calls is one of the most practical and important things an AI startup can get right, both because it protects people and because it is exactly what enterprise buyers probe. The good news is that it is achievable with deliberate engineering, detection, redaction, safe logging, the right provider settings, and the contractual layer underneath. The startups that get caught out are usually the ones that assumed sending data to a model was fine by default and never built the controls. Build them early, and you turn a common exposure into a point of confidence in your next security review.