If you don't run your own LLM proxy, how do you know whats going on?
Points and comments are a snapshot, not live.
Running your own LLM proxy is essential for observability, control, and multi-provider flexibility.
The author argues that relying on LLM providers' analytics panels offers meaningless token usage statistics. Running your own proxy gives you logs of which key called what model, virtual API keys to avoid littering a single key, and the ability to pivot providers quickly. The author uses a proxy at home for fun and at work for proper instrumentation. They emphasize modifying the proxy for custom needs like MCP endpoints and better insight into cache misses.
What commenters are saying
Commenters largely agree on the value of self-hosted LLM proxies for observability and security. Several recommend LiteLLM and GoModel for this purpose, citing features like virtual keys, guardrails, and dashboards. A debate emerges about the cost of API keys versus subscriptions, with some arguing subscriptions are subsidized loss leaders that may not last, while others counter that switching providers is easy and preparing for hypothetical price increases is unnecessary. One commenter notes prompt injection guardrails cost only 5-20 cents per day.