Inference Backends

ollama-classifier supports four inference backends, all behind a single unified LLMClassifier. Create a backend and wrap it in a classifier:

from ollama_classifier import LLMClassifier
from ollama_classifier.backends import OllamaBackend  # or VLLMBackend, etc.

backend = OllamaBackend(model="llama3.2")
classifier = LLMClassifier(backend)

All backends communicate with their inference engine via HTTP. OllamaBackend uses the native Ollama Python SDK; VLLMBackend, SGLangBackend, and LlamaCppBackend communicate via the OpenAI-compatible chat completions API.

Each backend translates the high-level constrain_labels parameter to its native constraint mechanism so the classifier code stays identical regardless of engine.

Ollama 

Backend wrapping the Ollama runtime (≥v0.12) via the official Python SDK.

Important

OllamaBackend requires Ollama runtime v0.12 or later for logprobs support. The ollama Python SDK is an optional dependency — install it with pip install "ollama-classifier[ollama]".

Constraint mechanism: JSON Schema enum via the format parameter. The model generates {"label": "<chosen_label>"}; structural JSON tokens ({, "label", :, ", }) are filtered during trie reconstruction. tokenize() uses context-dependent tokenization so the label is tokenized within the JSON prefix it appears in, ensuring the trie matches the actual response tokens.

Because Ollama wraps labels in JSON, its supports_bare_label_constraint property is False.

from ollama_classifier import LLMClassifier
from ollama_classifier.backends import OllamaBackend

# Local (defaults to http://localhost:11434)
backend = OllamaBackend(model="llama3.2")

# Remote
backend = OllamaBackend(model="llama3.2", host="http://remote-host:11434")

classifier = LLMClassifier(backend)

vLLM 

High-throughput serving engine for LLMs. Supports guided decoding and logprobs out of the box.

Constraint mechanism: ``guided_choice`` — constrains the model to generate exactly one of the provided label strings, as bare label text (no JSON wrapper). Logprobs are pre-mask (raw model logits before guided-decoding masking). supports_bare_label_constraint is True.

Local server:

python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.2-3B-Instruct \
    --host 0.0.0.0 --port 8000

Connect:

from ollama_classifier.backends import VLLMBackend

backend = VLLMBackend(
    model="meta-llama/Llama-3.2-3B-Instruct",
    base_url="http://localhost:8000/v1",
)

Remote server:

backend = VLLMBackend(
    model="your-model",
    base_url="https://your-vllm-server.com/v1",
    api_key="your-api-key",  # if auth is required
)

SGLang 

Fast serving system for large language models with efficient radix attention.

Constraint mechanism: regex — builds a regex from the escaped labels (("label1|label2|...")) so the model generates bare label text with no JSON wrapper. Logprobs are pre-mask (raw model logits before regex masking). supports_bare_label_constraint is True.

Local server:

python -m sglang.launch_server \
    --model-path meta-llama/Llama-3.2-3B-Instruct \
    --host 0.0.0.0 --port 30000

Connect:

from ollama_classifier.backends import SGLangBackend

backend = SGLangBackend(
    model="meta-llama/Llama-3.2-3B-Instruct",
    base_url="http://localhost:30000/v1",
)

llama.cpp 

Lightweight inference via llama-server. Ideal for CPU or mixed CPU/GPU environments.

Constraint mechanism: GBNF grammar — builds a grammar rule that allows exactly one of the provided labels:

root ::= "label1" | "label2" | "label3"

This generates bare label text with no JSON wrapper, so logprob reconstruction is clean. llama-server accepts a non-standard grammar field on the /v1/chat/completions endpoint (response_format with JSON schema is buggy in llama.cpp). supports_bare_label_constraint is True.

Local server:

./llama-server -m model.gguf --host 0.0.0.0 --port 8080 -c 4096

Connect:

from ollama_classifier.backends import LlamaCppBackend

backend = LlamaCppBackend(
    model="model",
    base_url="http://localhost:8080/v1",
)

Note

Logprobs and grammar constraints require llama.cpp to be compiled with the appropriate flags (e.g. LLAMA_SUPPORT_LOGPROBS).

Constraint Mechanisms Summary 

Each backend translates the high-level constrain_labels parameter to its native constraint mechanism:

`supports_bare_label_constraint`

Each backend exposes a supports_bare_label_constraint property that tells the classifier how to tokenize labels for trie construction:

``True`` (vLLM, SGLang, llama.cpp): chat() generates bare label text with no wrapper. Labels are tokenized standalone (context=None).
``False`` (Ollama): chat() wraps labels in JSON. Labels are tokenized within the JSON prefix (context='{"label": "') so the trie matches the actual response tokens.

This is handled automatically by LLMClassifier — you do not need to inspect it directly.

Backend Configuration 

All backends share common configuration options:

Parameter	Default	Description
`model`	(required)	Model identifier
`base_url`	Engine-specific	Base URL of the inference server
`api_key`	`"not-needed"`	API key for authentication (not used by Ollama)
`timeout`	`120.0`	Request timeout in seconds
`max_tokens` \| `256`		Maximum tokens to generate
`extra_body` \| `{}`		Extra parameters merged into every request body

OllamaBackend additionally accepts a host parameter (defaults to http://localhost:11434) and accepts pre-initialized sync_client / async_client instances for testing or connection reuse.

Switching Backends 

The LLMClassifier exposes the same API regardless of which backend you use, making it trivial to switch between engines:

from ollama_classifier.backends import (
    OllamaBackend, VLLMBackend, SGLangBackend, LlamaCppBackend,
)
from ollama_classifier import LLMClassifier

# Switch just by changing the backend
backends = [
    OllamaBackend(model="llama3.2"),
    VLLMBackend(model="my-model", base_url="http://localhost:8000/v1"),
    SGLangBackend(model="my-model", base_url="http://localhost:30000/v1"),
    LlamaCppBackend(model="my-model", base_url="http://localhost:8080/v1"),
]

for backend in backends:
    classifier = LLMClassifier(backend)
    result = classifier.classify(
        text="Hello world!",
        choices=["a", "b", "c"],
    )
    print(f"{backend.__class__.__name__}: {result.prediction}")

Inference Backends

Ollama

vLLM

SGLang

llama.cpp

Constraint Mechanisms Summary

supports_bare_label_constraint

Backend Configuration

Switching Backends

Ollama 

vLLM 

SGLang 

llama.cpp 

Constraint Mechanisms Summary 

`supports_bare_label_constraint`

Backend Configuration 

Switching Backends 