blog.rust.gay

Running local LLMs with Ollama

2026-08-31 · ai local

Ollama is a local runtime for downloading and running open language models on your own machine. It exposes a command-line interface and an HTTP server, which makes it useful for quick experiments, offline-friendly workflows, and applications that need a local model endpoint without building an inference stack from scratch. The trade-off is the usual one for local inference: model size, quantization, context length, and available CPU/GPU memory determine what is practical.

This post focuses on the local workflow. It does not cover production multi-replica serving, scheduler tuning, or throughput-oriented deployment patterns; those belong in a separate inference-serving discussion.

Install, pull, and run a model

The basic local loop is simple: install Ollama, pull or run a model, and talk to it through the CLI or API. The Ollama quickstart shows ollama run gemma4:e2b as a starting point: Ollama downloads the model and starts an interactive chat on the local machine. In the chat, /bye exits the session.

For application use, the local server listens on http://localhost:11434 by default. The Ollama API documentation distinguishes direct cloud access from the local server: the native Ollama API is available at https://ollama.com/api for cloud requests and http://localhost:11434/api for local requests. Local requests do not require an API key. Cloud requests do, and creating or deleting models requires a local Ollama server.

A minimal local chat request looks like this:

curl http://localhost:11434/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma4:e2b",
    "messages": [{"role": "user", "content": "Say hello in one sentence."}],
    "stream": false
  }'

The response contains the assistant message in message.content. Ollama also documents OpenAI-compatible and Anthropic-compatible endpoints on the same local server (/v1 and the Anthropic client base URL), which can reduce integration work when an existing tool already speaks one of those APIs. Compatibility is not identical to the full upstream APIs, so the supported features should be checked against Ollama's compatibility documentation before relying on them.

Choose models with hardware limits in mind

Ollama's model library is the discovery layer, but the important decision is whether the selected model and quantization fit the machine you actually have. A small model can be responsive on a laptop CPU or integrated GPU; a larger model may require substantial VRAM, aggressive quantization, or a cloud route. The right question is not "can Ollama run this model?" in the abstract, but "can this machine run this model at an acceptable context length and latency?"

Ollama has added performance and compatibility work over time. Its June 5, 2026 blog post, Improved performance and model support with GGUF, announced Ollama 0.30 with improved GGUF model compatibility through llama.cpp and said Vulkan is enabled by default to extend GPU acceleration to a wider range of hardware, including AMD and Intel devices. The same post described creating a model from a local GGUF file with a Modelfile containing FROM ./my-model.Q4_K_M.gguf, followed by ollama create -f Modelfile my-model and ollama run my-model. Those details are useful when you need to import a GGUF model or evaluate hardware support, but they should still be validated against the version you run.

Use Modelfiles when defaults are not enough

A Modelfile is Ollama's declarative way to define or customize a local model. It can set a base model, system prompt, parameters, template behavior, and other model configuration. This is the right place for repeatable local behavior: a coding assistant with a stable system instruction, a model imported from GGUF, or a variant with specific runtime options.

The practical rule is to keep model identity explicit. If a workflow depends on a particular prompt, context setting, or imported file, encode it in a Modelfile and give the resulting model a clear name. Ad hoc CLI flags are fine for exploration; checked-in or documented Modelfiles are better when the behavior needs to be reproduced by teammates or future you.

Capabilities, caveats, and privacy boundaries

Ollama's documentation and release history show support for features that matter in local application development, including streaming responses, structured outputs, tool calling, embeddings, vision, and thinking-capable models. These capabilities depend on the selected model and API path, not only on the Ollama runtime. A model that runs locally is not automatically good at tool use, JSON-constrained output, or image understanding.

Local execution changes the privacy boundary, but it does not remove every boundary. Prompts and model outputs can stay on the machine when you use the local server, which is useful for sensitive drafts, internal code, or disconnected environments. If you switch to Ollama's hosted cloud API, requests go to https://ollama.com and require authentication. Treat local and cloud modes as different data paths even when the client code looks similar.

Operationally, verify three things before trusting a local setup: the model fits memory at the context length you need; the API compatibility layer supports the feature your client expects; and the output quality is acceptable for the task. "Runs locally" is a deployment property. Correctness, latency, and safety still need evaluation.

Conclusion

Ollama is a practical entry point for local LLM work because it combines model management, a local HTTP API, and model customization in one tool. Start with a small model, confirm local API behavior against the official docs, and use a Modelfile when the configuration needs to be repeatable. If a project grows into multi-user serving or strict latency targets, move the design conversation to an inference-serving stack rather than stretching a laptop workflow past its limits.

Sources

← back to posts