Prompt injection defense, part 2: controls that limit what injected text can do

Enforce limits in code, not in the prompt

A system prompt that says “never refund more than 50 EUR” is a request. A tool function that rejects 51 is a control. Every rule that matters belongs in the tool implementation, which checks arguments against the authenticated session, not against anything the model claims.

def issue_refund(session, order_id: str, eur: Decimal) -> dict:
    order = orders.get(order_id)
    if order is None or order.customer_id != session.ticket.customer_id:
        raise ToolDenied("order is not on this ticket")
    if eur <= 0 or eur > min(Decimal("50"), order.total):
        raise ToolDenied("amount outside refund policy")
    return pending_actions.create(kind="refund", order_id=order_id, eur=eur,
                                  needs_confirmation_by=session.agent_id)

Human confirmation for side effects

The refund function above does not refund; it creates a pending action that the support agent confirms in the UI, with the amount and order shown in plain text outside the model’s output. The email tool works the same way and only sends to the ticket’s customer address, taken from the session. An injected “email this to someone else” has nowhere to go.

Output handling

Treat model output like any untrusted HTML source. Render Markdown with raw HTML disabled, and do not auto-load images from arbitrary hosts, because an image URL can carry private data in its query string to an outside server the moment it renders. Links are shown with their full destination.

const md = markdownIt({ html: false, linkify: true })
md.renderer.rules.image = (tokens, idx) => {
  const src = tokens[idx].attrGet('src') || ''
  return ALLOWED_IMAGE_HOSTS.has(new URL(src, location.href).host)
    ? `<img src="${md.utils.escapeHtml(src)}" alt="">`
    : `<span class="blocked-image">[image from external host not loaded]</span>`
}

Worked design choice: separating reading from acting

For the invoice-summarising feature, the team uses two calls. The first model reads the untrusted invoice and returns a fixed JSON schema: vendor, amount, due date, and a free-text summary. Code validates the schema. The second step, which can suggest actions, receives only the validated fields, never the raw invoice text. An instruction hidden in the invoice can at worst distort the summary, which the user reads; it cannot reach the step that proposes payments.

Content marking, such as wrapping retrieved text in tags and telling the model it is data, lowers the success rate of injections and is worth keeping. It is a mitigation layered on these controls, not a replacement for them.

{"vendor": "Northwind Supplies", "amount": "1240.00", "currency": "EUR",
 "due": "2026-11-30", "summary": "Invoice 4471 for October supplies."}