Local LLM Inference & Discord Bot

Implementation

I integrated Ollama with an asynchronous Python bot and SQLite-backed history retrieval. The bot streams responses and retrieves indexed context from the channel receiving the reply. It includes model controls and permission checks.

Results

Time-to-first-token and generation-speed instrumentation make it possible to inspect local inference performance. The project exposes latency and model metrics alongside controls for loading, unloading, and selecting models.

Observations

Model choice, available memory, and context length all affect responsiveness. Instrumentation is more useful than assuming that a local model will feel fast enough in conversation.

Limitations

This is a hobby project for private servers with informed participant consent. Response quality and throughput depend on the model and hardware. Consent is an operator requirement; the bot does not track it automatically.

Code & reports

• Implementation & limitations →