What you need first
A local inference server already answering on loopback — Ollama on 11434, llama.cpp on 8080, vLLM on 8000 — a domain pointing at the host, and a certificate. Nginx is doing three jobs here: terminating TLS, holding connections open long enough for generation, and being the only thing on the public interface. The backend stays bound to 127.0.0.1; if it is listening on 0.0.0.0 the proxy is decoration.
# The backend must NOT be publicly bound. Check before you proxy:
ss -tlnp | grep -E "11434|8080|8000"
# Expect 127.0.0.1:11434, not 0.0.0.0:11434
sudo certbot --nginx -d llm.example.comThe two settings that break streaming
This is the part people get wrong, and the symptom is confusing: the API "works" in curl with a non-streaming request and appears to hang with a streaming one. Nginx buffers proxied responses by default, so server-sent events arrive in one lump at the end instead of token by token. And the default read timeout is 60 seconds — long enough for a short reply and not for a long generation, which then looks like the model crashed.
server {
listen 443 ssl;
http2 on;
server_name llm.example.com;
ssl_certificate /etc/letsencrypt/live/llm.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/llm.example.com/privkey.pem;
location /v1/ {
proxy_pass http://127.0.0.1:11434;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
# Stream tokens instead of buffering the whole reply:
proxy_buffering off;
proxy_cache off;
# A long generation is not a hung connection:
proxy_read_timeout 600s;
proxy_send_timeout 600s;
}
}Put something in front of it
An open endpoint on the public internet is a GPU anyone can spend. Nginx can do the cheap half of that — a shared secret and a rate limit — and it should, because the backends mostly have no authentication of their own. `limit_req` with a burst absorbs a normal client’s bursty behaviour while still stopping a script.
limit_req_zone $binary_remote_addr zone=llm:10m rate=10r/m;
map $http_authorization $api_ok {
default 0;
"Bearer YOUR_LONG_RANDOM_TOKEN" 1;
}
location /v1/ {
if ($api_ok = 0) { return 401; }
limit_req zone=llm burst=5 nodelay;
# …proxy settings from above…
}Check it actually worked
Three checks, because three different things can be wrong. Test that the endpoint answers over TLS, that a streaming request actually streams rather than arriving all at once, and — most importantly — that the backend is not reachable directly from outside.
# 1. Does it answer at all?
curl -H "Authorization: Bearer YOUR_LONG_RANDOM_TOKEN" \
https://llm.example.com/v1/models
# 2. Does it stream? Tokens should appear progressively, not in one burst.
curl -N -H "Authorization: Bearer YOUR_LONG_RANDOM_TOKEN" \
-H "Content-Type: application/json" \
-d '{"model":"llama3.1:8b","messages":[{"role":"user","content":"count to 20"}],"stream":true}' \
https://llm.example.com/v1/chat/completions
# 3. From ANOTHER machine — this must fail:
curl --max-time 5 http://llm.example.com:11434/api/tagsWhat the numbers should look like
The proxy adds no measurable latency to generation — the time is spent reading weights on the GPU, not forwarding bytes. What changes is time-to-first-token as perceived by the client: with `proxy_buffering on` it equals the whole generation, and with it off it is milliseconds. If throughput through the proxy differs from throughput measured locally by more than noise, you are looking at buffering rather than at the network.
When it does not work
Streaming requests deliver everything at once at the end: `proxy_buffering off` is missing from that location block. A 504 partway through a long generation: `proxy_read_timeout` is still the 60-second default. 502 on every request: the backend is not listening where `proxy_pass` points, or it bound to a different interface — check with `ss -tlnp` rather than assuming. Clients get 401 with a correct token: header matching is exact, so a trailing space or a different capitalisation of `Bearer` fails. And if step 3 above succeeds from another machine, stop and fix that first: a proxy in front of a publicly-bound backend protects nothing.
Common questions
Why does streaming stop working behind Nginx?
Nginx buffers proxied responses by default, so server-sent events are held and delivered in one piece at the end of the generation. Set `proxy_buffering off` (and `proxy_cache off`) in the location block that proxies the API. The request still succeeds, which is why this is usually diagnosed as a client bug first.
Why do long generations return 504?
The default `proxy_read_timeout` is 60 seconds, and a long answer on a local model can easily exceed that — Nginx closes the connection and reports a gateway timeout while the model is still working. Raise `proxy_read_timeout` and `proxy_send_timeout` to something matching your longest realistic reply.
Is a reverse proxy enough to secure a local LLM API?
Only if the backend is not reachable without it. Ollama, llama.cpp and vLLM ship with no authentication, so the proxy must be the only route in — bind the backend to `127.0.0.1` and verify from another machine that its port refuses connections. Adding a token check and a `limit_req` rate limit at the proxy covers the cheap half of the problem; leaving the backend on `0.0.0.0` means none of it counts.
What this guide uses
Did this actually run?
Copying a command is not the same as it working, so this is the only place the site asks. Nothing is collected beyond the answer itself.
Related guides
Run Llama 3.1 8B on a €20/month VPS
A complete guide to running a private LLM API on a budget Linux VPS using llama.cpp server mode.
Multi-Model API Server on RTX 4090 with vLLM
Serve multiple AWQ-quantized models with vLLM's continuous batching for production-grade throughput.
TabbyAPI: ExLlamaV2 with a Web UI
Wrap ExLlamaV2 in TabbyAPI for a polished OpenAI-compatible server with streaming and model hot-swap.