API Reference

This section provides the complete API reference for ollama-classifier.

LLMClassifier

The single, unified, backend-agnostic classifier. Accepts any LLMBackend instance and exposes two scoring methods: generate() (adaptive constrained generation) and classify() (exact multi-call completion scoring).

class ollama_classifier.classifier.LLMClassifier(backend: LLMBackend, *, max_workers: int = 4)[source]

Bases: object

Backend-agnostic text classifier with two confidence scoring methods.

generate()[source]: Adaptive constrained generation with divergence-aware confidence. Budget-controlled via max_calls.

classify()[source]: Multi-call completion scoring with geometric-mean normalization. Always exact.

Parameters:

backend – An LLMBackend instance (e.g., OllamaBackend, VLLMBackend, SGLangBackend, or LlamaCppBackend).
max_workers – Thread pool size for sync batch operations.

__init__(backend: LLMBackend, *, max_workers: int = 4)[source]

generate(text: str, choices: List[str] | Dict[str, str], system_prompt: str | None = None, *, max_calls: int | None = 1) → ClassificationResult[source]

Adaptive constrained classification with divergence-aware confidence.

Makes 1 to max_calls constrained API calls. Each call walks a prefix trie of label tokens. After each call, labels are scored up to their divergence point from the winning path. Unresolved clusters trigger supplementary calls (recursive cluster resolution).

With max_calls=1: single call, partial scoring (fast, approximate). With max_calls=None: resolves everything recursively (exact).

Parameters:

text – Text to classify.
choices – Labels as list or {label: description} dict.
system_prompt – Optional custom system prompt.
max_calls – Maximum number of API calls. None = unlimited.

Returns:

ClassificationResult with method="adaptive_generate".

classify(text: str, choices: List[str] | Dict[str, str], system_prompt: str | None = None) → ClassificationResult[source]

Multi-call classification with geometric-mean completion scoring.

For each label, scores the label as a completion of the prompt and extracts per-token logprobs WITHOUT generation. Applies geometric-mean normalization to eliminate token-count bias. This is the gold-standard confidence method.

Makes N API calls for N choices (parallelizable via async).

Parameters:

text – Text to classify.
choices – Labels as list or {label: description} dict.
system_prompt – Optional custom system prompt.

Returns:

ClassificationResult with method="multi_call", approximate=False.

batch_generate(texts: List[str], choices: List[str] | Dict[str, str], system_prompt: str | None = None, *, max_calls: int | None = 1) → List[ClassificationResult][source]: Batch adaptive generation (parallelized via ThreadPoolExecutor).

batch_classify(texts: List[str], choices: List[str] | Dict[str, str], system_prompt: str | None = None) → List[ClassificationResult][source]: Batch multi-call classification (parallelized via ThreadPoolExecutor).

async agenerate(text: str, choices: List[str] | Dict[str, str], system_prompt: str | None = None, *, max_calls: int | None = 1) → ClassificationResult[source]: Async adaptive generation with divergence-aware confidence.

async aclassify(text: str, choices: List[str] | Dict[str, str], system_prompt: str | None = None) → ClassificationResult[source]: Async multi-call classification (labels scored concurrently).

async abatch_generate(texts: List[str], choices: List[str] | Dict[str, str], system_prompt: str | None = None, *, max_calls: int | None = 1) → List[ClassificationResult][source]: Async batch adaptive generation.

async abatch_classify(texts: List[str], choices: List[str] | Dict[str, str], system_prompt: str | None = None) → List[ClassificationResult][source]: Async batch multi-call classification.

ClassificationResult

A Pydantic model returned by both generate() and classify().

class ollama_classifier.types.ClassificationResult(*, prediction: str, confidence: float, probabilities: ~typing.Dict[str, float], method: str = 'multi_call', approximate: bool = False, coverage: ~typing.Dict[str, float] = <factory>, n_calls: int = 1, raw_response: ~typing.Dict[str, ~typing.Any] = <factory>)[source]

Bases: BaseModel

Result of a classification operation.

prediction

The predicted class label.

Type:: str

confidence

Confidence score for the prediction (0.0 to 1.0).

Type:: float

probabilities

Probability distribution over all choices.

Type:: Dict[str, float]

method

Scoring method used: "adaptive_generate" or "multi_call".

Type:: str

approximate

True if any label has partial coverage (unresolved tokens). Only relevant for adaptive_generate; always False for multi_call.

Type:: bool

coverage

Per-label fraction of tokens scored (0.0 to 1.0). 1.0 = fully resolved. Only present for adaptive_generate.

Type:: Dict[str, float]

n_calls

Number of API calls made.

Type:: int

raw_response

Raw data for debugging.

Type:: Dict[str, Any]

prediction: str

confidence: float

probabilities: Dict[str, float]

method: str

approximate: bool

coverage: Dict[str, float]

n_calls: int

raw_response: Dict[str, Any]

model_config = {}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

ChoicesType

ollama_classifier.types.ChoicesType: alias of List[str] | Dict[str, str]

Backends

Inference engine backends for ollama-classifier.

Each backend implements the LLMBackend protocol and communicates with its respective inference engine. OllamaBackend uses the native Ollama SDK; VLLMBackend, SGLangBackend, and LlamaCppBackend communicate via the OpenAI-compatible API.

Each backend translates the high-level constrain_labels parameter to its native constraint mechanism:

Ollama: JSON Schema enum via format
vLLM: guided_choice
SGLang: regex
llama.cpp: GBNF grammar

class ollama_classifier.backends.LLMBackend(model: str, base_url: str = '', *, api_key: str | None = None, timeout: float = 120.0, max_tokens: int = 256, extra_body: Dict[str, Any] | None = None)[source]

Bases: ABC

Abstract base class for LLM inference backends.

All backends communicate with their respective inference engine via HTTP (OpenAI-compatible API). Each backend translates constrain_labels to its native constraint mechanism and implements score() for completion scoring and tokenize() for context-dependent tokenization.

__init__(model: str, base_url: str = '', *, api_key: str | None = None, timeout: float = 120.0, max_tokens: int = 256, extra_body: Dict[str, Any] | None = None)[source]

Initialize the backend.

Parameters:

model – The model identifier to use.
base_url – Base URL of the inference server.
api_key – Optional API key. Defaults to "not-needed".
timeout – Request timeout in seconds.
max_tokens – Maximum tokens to generate.
extra_body – Extra parameters merged into every request body.

property model: str

property base_url: str

abstract property supports_bare_label_constraint: bool

True if chat() generates bare label text (no JSON wrapper).

vLLM/SGLang/llama.cpp: True. Ollama: False (uses JSON enum wrapper).

abstractmethod chat(messages: List[ChatMessage], *, temperature: float = 0.0, constrain_labels: List[str] | None = None, logprobs: bool = False, top_logprobs: int = 5) → ChatResponse[source]

Perform a synchronous constrained chat completion.

Each backend translates constrain_labels to its native constraint mechanism (guided_choice, regex, JSON enum, or GBNF grammar).

Parameters:

messages – List of chat messages.
temperature – Sampling temperature.
constrain_labels – Optional list of valid label strings. If provided, output is constrained to exactly one of these.
logprobs – Whether to return log probabilities.
top_logprobs – Number of top log probabilities per token.

Returns:

ChatResponse with content and optional logprobs.

abstractmethod score(messages: List[ChatMessage], completion: str) → ScoringResponse[source]

Score a completion by computing per-token logprobs of the completion text given the message context.

No generation occurs — the completion is provided and the backend returns the model’s logprob for each token of the completion.

Parameters:

messages – The chat messages forming the context.
completion – The completion text to score.

Returns:

ScoringResponse with per-token logprobs.

abstractmethod tokenize(text: str, *, context: str | None = None) → List[Token][source]

Tokenize text.

If context is provided, tokenizes context + text and returns only the tokens corresponding to text (context-dependent tokenization). This is critical for backends that wrap labels in JSON (Ollama), where the tokenization of a label inside JSON differs from standalone tokenization.

Parameters:

text – The text to tokenize.
context – Optional prefix to include for context-dependent tokenization.

Returns:

List of Token objects.

abstractmethod async achat(messages: List[ChatMessage], *, temperature: float = 0.0, constrain_labels: List[str] | None = None, logprobs: bool = False, top_logprobs: int = 5) → ChatResponse[source]: Async constrained chat completion.

abstractmethod async ascore(messages: List[ChatMessage], completion: str) → ScoringResponse[source]: Async completion scoring.

abstractmethod async atokenize(text: str, *, context: str | None = None) → List[Token][source]: Async tokenization.

class ollama_classifier.backends.ChatMessage(role: str, content: str)[source]

Bases: object

A single chat message.

role: str

content: str

class ollama_classifier.backends.ChatResponse(content: str, label: str = '', logprobs: List[TokenLogprob] | None = None, raw: Dict[str, ~typing.Any]=<factory>)[source]

Bases: object

Response from a constrained generation call.

content: str

label: str = ''

logprobs: List[TokenLogprob] | None = None

raw: Dict[str, Any]

class ollama_classifier.backends.ScoringResponse(completion: str, logprobs: List[TokenLogprob] = <factory>, raw: Dict[str, ~typing.Any]=<factory>)[source]

Bases: object

Response from a completion-scoring call (for classify()).

Contains per-token logprobs for the COMPLETION text only, not the prompt.

completion: str

logprobs: List[TokenLogprob]

raw: Dict[str, Any]

class ollama_classifier.backends.TokenLogprob(token: str, token_id: int = -1, logprob: float = 0.0, top_logprobs: dict[str, float]=<factory>)[source]

Bases: object

A single token with its log probability and alternatives.

top_logprobs follows the OpenAI format: a list of dicts, each with "token", "logprob", and optionally "bytes" keys.

token: str

token_id: int = -1

logprob: float = 0.0

top_logprobs: dict[str, float]

class ollama_classifier.backends.Token(text: str, id: int)[source]

Bases: object

A tokenized unit.

text: str

id: int

class ollama_classifier.backends.OllamaBackend(model: str, *, host: str | None = None, sync_client: Any = None, async_client: Any = None, timeout: float = 120.0, max_tokens: int = 256, extra_body: Dict[str, Any] | None = None)[source]

Bases: LLMBackend

Backend for the Ollama runtime (≥v0.12) via the official Python SDK.

Ollama provides a local LLM runtime with an OpenAI-compatible API and a native API. This backend uses the native API via the ollama Python SDK. JSON schema constraints and logprobs are supported as of v0.12.

Note

Ollama’s constraint mechanism is JSON Schema enum, which wraps the label in JSON structural tokens. The tokenize() method supports context-dependent tokenization so the label is tokenized within the JSON prefix it appears in, ensuring the trie matches the response tokens.

__init__(model: str, *, host: str | None = None, sync_client: Any = None, async_client: Any = None, timeout: float = 120.0, max_tokens: int = 256, extra_body: Dict[str, Any] | None = None)[source]

Initialize the Ollama backend.

Parameters:

model – Model name (e.g., "llama3.2").
host – Ollama server URL. Defaults to http://localhost:11434.
sync_client – Pre-initialized ollama.Client (sync). Created lazily if None.
async_client – Pre-initialized ollama.AsyncClient. Created lazily if None.
timeout – Request timeout in seconds.
max_tokens – Maximum tokens to generate.
extra_body – Extra parameters merged into every request options.

property supports_bare_label_constraint: bool: False — Ollama uses JSON enum wrapper.

chat(messages: List[ChatMessage], *, temperature: float = 0.0, constrain_labels: List[str] | None = None, logprobs: bool = False, top_logprobs: int = 5) → ChatResponse[source]: Perform a synchronous constrained chat completion via Ollama.

score(messages: List[ChatMessage], completion: str) → ScoringResponse[source]

Score a completion using Ollama’s generate endpoint with suffix.

Uses client.generate(prompt=..., suffix=completion) to compute the per-token logprobs of the completion given the prompt context. No generation occurs (num_predict=0).

tokenize(text: str, *, context: str | None = None) → List[Token][source]

Tokenize text using Ollama’s tokenize API.

If context is provided, tokenizes context + text and returns only the tokens corresponding to text. This ensures context-dependent tokenization matches the actual response tokens (critical for JSON-wrapped labels).

async achat(messages: List[ChatMessage], *, temperature: float = 0.0, constrain_labels: List[str] | None = None, logprobs: bool = False, top_logprobs: int = 5) → ChatResponse[source]: Async constrained chat completion via Ollama.

async ascore(messages: List[ChatMessage], completion: str) → ScoringResponse[source]: Async completion scoring via Ollama.

async atokenize(text: str, *, context: str | None = None) → List[Token][source]: Async tokenization via Ollama.

class ollama_classifier.backends.VLLMBackend(model: str, base_url: str = 'http://localhost:8000/v1', *, api_key: str | None = None, timeout: float = 120.0, max_tokens: int = 256, extra_body: Dict[str, Any] | None = None)[source]

Bases: LLMBackend

Backend for vLLM inference server.

vLLM provides a high-throughput serving engine with an OpenAI-compatible API endpoint. It supports guided decoding (guided_choice) and logprob return out of the box. Logprobs are pre-mask (raw model logits before guided decoding masking).

property supports_bare_label_constraint: bool: True — vLLM guided_choice generates bare label text.

chat(messages: List[ChatMessage], *, temperature: float = 0.0, constrain_labels: List[str] | None = None, logprobs: bool = False, top_logprobs: int = 5) → ChatResponse[source]: Synchronous constrained chat completion via vLLM.

score(messages: List[ChatMessage], completion: str) → ScoringResponse[source]

Score a completion using vLLM’s completions endpoint.

Uses /v1/completions with prompt_logprobs to compute the per-token logprobs of the completion given the prompt context.

tokenize(text: str, *, context: str | None = None) → List[Token][source]

Tokenize text using vLLM’s tokenize endpoint.

vLLM provides a /tokenize endpoint (non-standard but supported in recent versions).

async achat(messages: List[ChatMessage], *, temperature: float = 0.0, constrain_labels: List[str] | None = None, logprobs: bool = False, top_logprobs: int = 5) → ChatResponse[source]: Async constrained chat completion via vLLM.

async ascore(messages: List[ChatMessage], completion: str) → ScoringResponse[source]: Async completion scoring via vLLM.

async atokenize(text: str, *, context: str | None = None) → List[Token][source]: Async tokenization via vLLM.

class ollama_classifier.backends.SGLangBackend(model: str, base_url: str = 'http://localhost:30000/v1', *, api_key: str | None = None, timeout: float = 120.0, max_tokens: int = 256, extra_body: Dict[str, Any] | None = None)[source]

Bases: LLMBackend

Backend for SGLang inference server.

SGLang is a fast serving system for large language models with an OpenAI-compatible API. It supports regex-guided decoding and logprobs. Logprobs are pre-mask (raw model logits before regex masking).

property supports_bare_label_constraint: bool: True — SGLang regex constraint generates bare label text.

chat(messages: List[ChatMessage], *, temperature: float = 0.0, constrain_labels: List[str] | None = None, logprobs: bool = False, top_logprobs: int = 5) → ChatResponse[source]: Synchronous constrained chat completion via SGLang.

score(messages: List[ChatMessage], completion: str) → ScoringResponse[source]: Score a completion using SGLang’s completions endpoint.

tokenize(text: str, *, context: str | None = None) → List[Token][source]: Tokenize text using SGLang’s tokenize endpoint.

async achat(messages: List[ChatMessage], *, temperature: float = 0.0, constrain_labels: List[str] | None = None, logprobs: bool = False, top_logprobs: int = 5) → ChatResponse[source]: Async constrained chat completion via SGLang.

async ascore(messages: List[ChatMessage], completion: str) → ScoringResponse[source]: Async completion scoring via SGLang.

async atokenize(text: str, *, context: str | None = None) → List[Token][source]: Async tokenization via SGLang.

class ollama_classifier.backends.LlamaCppBackend(model: str, base_url: str = 'http://localhost:8080/v1', *, api_key: str | None = None, timeout: float = 120.0, max_tokens: int = 256, extra_body: Dict[str, Any] | None = None)[source]

Bases: LLMBackend

Backend for llama.cpp server (llama-server).

llama.cpp provides a lightweight inference server with an OpenAI-compatible API. GBNF grammar constraints and logprobs are supported when compiled with the appropriate flags. Logprobs are pre-mask (model’s raw distribution before grammar masking).

Note

response_format with JSON schema on /v1/chat/completions is buggy in llama.cpp (GitHub issues #11988, #11847). This backend uses the non-standard grammar field with GBNF instead, which works reliably for bare label generation.

property supports_bare_label_constraint: bool: True — GBNF grammar generates bare label text.

chat(messages: List[ChatMessage], *, temperature: float = 0.0, constrain_labels: List[str] | None = None, logprobs: bool = False, top_logprobs: int = 5) → ChatResponse[source]: Synchronous constrained chat completion via llama.cpp server.

score(messages: List[ChatMessage], completion: str) → ScoringResponse[source]

Score a completion using llama.cpp’s completions endpoint with suffix.

Uses /v1/completions with suffix to compute the per-token logprobs of the completion given the prompt context (fill-in-the-middle style scoring).

tokenize(text: str, *, context: str | None = None) → List[Token][source]

Tokenize text using llama.cpp’s tokenize endpoint.

llama-server provides a /tokenize endpoint.

async achat(messages: List[ChatMessage], *, temperature: float = 0.0, constrain_labels: List[str] | None = None, logprobs: bool = False, top_logprobs: int = 5) → ChatResponse[source]: Async constrained chat completion via llama.cpp server.

async ascore(messages: List[ChatMessage], completion: str) → ScoringResponse[source]: Async completion scoring via llama.cpp server.

async atokenize(text: str, *, context: str | None = None) → List[Token][source]: Async tokenization via llama.cpp server.

class ollama_classifier.backends.base.LLMBackend(model: str, base_url: str = '', *, api_key: str | None = None, timeout: float = 120.0, max_tokens: int = 256, extra_body: Dict[str, Any] | None = None)[source]

Bases: ABC

Abstract base class for LLM inference backends.

All backends communicate with their respective inference engine via HTTP (OpenAI-compatible API). Each backend translates constrain_labels to its native constraint mechanism and implements score() for completion scoring and tokenize() for context-dependent tokenization.

__init__(model: str, base_url: str = '', *, api_key: str | None = None, timeout: float = 120.0, max_tokens: int = 256, extra_body: Dict[str, Any] | None = None)[source]

Initialize the backend.

Parameters:

model – The model identifier to use.
base_url – Base URL of the inference server.
api_key – Optional API key. Defaults to "not-needed".
timeout – Request timeout in seconds.
max_tokens – Maximum tokens to generate.
extra_body – Extra parameters merged into every request body.

property model: str

property base_url: str

abstract property supports_bare_label_constraint: bool

True if chat() generates bare label text (no JSON wrapper).

vLLM/SGLang/llama.cpp: True. Ollama: False (uses JSON enum wrapper).

abstractmethod chat(messages: List[ChatMessage], *, temperature: float = 0.0, constrain_labels: List[str] | None = None, logprobs: bool = False, top_logprobs: int = 5) → ChatResponse[source]

Perform a synchronous constrained chat completion.

Each backend translates constrain_labels to its native constraint mechanism (guided_choice, regex, JSON enum, or GBNF grammar).

Parameters:

messages – List of chat messages.
temperature – Sampling temperature.
constrain_labels – Optional list of valid label strings. If provided, output is constrained to exactly one of these.
logprobs – Whether to return log probabilities.
top_logprobs – Number of top log probabilities per token.

Returns:

ChatResponse with content and optional logprobs.

abstractmethod score(messages: List[ChatMessage], completion: str) → ScoringResponse[source]

Score a completion by computing per-token logprobs of the completion text given the message context.

No generation occurs — the completion is provided and the backend returns the model’s logprob for each token of the completion.

Parameters:

messages – The chat messages forming the context.
completion – The completion text to score.

Returns:

ScoringResponse with per-token logprobs.

abstractmethod tokenize(text: str, *, context: str | None = None) → List[Token][source]

Tokenize text.

If context is provided, tokenizes context + text and returns only the tokens corresponding to text (context-dependent tokenization). This is critical for backends that wrap labels in JSON (Ollama), where the tokenization of a label inside JSON differs from standalone tokenization.

Parameters:

text – The text to tokenize.
context – Optional prefix to include for context-dependent tokenization.

Returns:

List of Token objects.

abstractmethod async achat(messages: List[ChatMessage], *, temperature: float = 0.0, constrain_labels: List[str] | None = None, logprobs: bool = False, top_logprobs: int = 5) → ChatResponse[source]: Async constrained chat completion.

abstractmethod async ascore(messages: List[ChatMessage], completion: str) → ScoringResponse[source]: Async completion scoring.

abstractmethod async atokenize(text: str, *, context: str | None = None) → List[Token][source]: Async tokenization.

class ollama_classifier.backends.ollama.OllamaBackend(model: str, *, host: str | None = None, sync_client: Any = None, async_client: Any = None, timeout: float = 120.0, max_tokens: int = 256, extra_body: Dict[str, Any] | None = None)[source]

Bases: LLMBackend

Backend for the Ollama runtime (≥v0.12) via the official Python SDK.

Ollama provides a local LLM runtime with an OpenAI-compatible API and a native API. This backend uses the native API via the ollama Python SDK. JSON schema constraints and logprobs are supported as of v0.12.

Note

Ollama’s constraint mechanism is JSON Schema enum, which wraps the label in JSON structural tokens. The tokenize() method supports context-dependent tokenization so the label is tokenized within the JSON prefix it appears in, ensuring the trie matches the response tokens.

__init__(model: str, *, host: str | None = None, sync_client: Any = None, async_client: Any = None, timeout: float = 120.0, max_tokens: int = 256, extra_body: Dict[str, Any] | None = None)[source]

Initialize the Ollama backend.

Parameters:

model – Model name (e.g., "llama3.2").
host – Ollama server URL. Defaults to http://localhost:11434.
sync_client – Pre-initialized ollama.Client (sync). Created lazily if None.
async_client – Pre-initialized ollama.AsyncClient. Created lazily if None.
timeout – Request timeout in seconds.
max_tokens – Maximum tokens to generate.
extra_body – Extra parameters merged into every request options.

property supports_bare_label_constraint: bool: False — Ollama uses JSON enum wrapper.

chat(messages: List[ChatMessage], *, temperature: float = 0.0, constrain_labels: List[str] | None = None, logprobs: bool = False, top_logprobs: int = 5) → ChatResponse[source]: Perform a synchronous constrained chat completion via Ollama.

score(messages: List[ChatMessage], completion: str) → ScoringResponse[source]

Score a completion using Ollama’s generate endpoint with suffix.

Uses client.generate(prompt=..., suffix=completion) to compute the per-token logprobs of the completion given the prompt context. No generation occurs (num_predict=0).

tokenize(text: str, *, context: str | None = None) → List[Token][source]

Tokenize text using Ollama’s tokenize API.

If context is provided, tokenizes context + text and returns only the tokens corresponding to text. This ensures context-dependent tokenization matches the actual response tokens (critical for JSON-wrapped labels).

async achat(messages: List[ChatMessage], *, temperature: float = 0.0, constrain_labels: List[str] | None = None, logprobs: bool = False, top_logprobs: int = 5) → ChatResponse[source]: Async constrained chat completion via Ollama.

async ascore(messages: List[ChatMessage], completion: str) → ScoringResponse[source]: Async completion scoring via Ollama.

async atokenize(text: str, *, context: str | None = None) → List[Token][source]: Async tokenization via Ollama.

class ollama_classifier.backends.vllm.VLLMBackend(model: str, base_url: str = 'http://localhost:8000/v1', *, api_key: str | None = None, timeout: float = 120.0, max_tokens: int = 256, extra_body: Dict[str, Any] | None = None)[source]

Bases: LLMBackend

Backend for vLLM inference server.

vLLM provides a high-throughput serving engine with an OpenAI-compatible API endpoint. It supports guided decoding (guided_choice) and logprob return out of the box. Logprobs are pre-mask (raw model logits before guided decoding masking).

__init__(model: str, base_url: str = 'http://localhost:8000/v1', *, api_key: str | None = None, timeout: float = 120.0, max_tokens: int = 256, extra_body: Dict[str, Any] | None = None)[source]

Initialize the backend.

Parameters:

model – The model identifier to use.
base_url – Base URL of the inference server.
api_key – Optional API key. Defaults to "not-needed".
timeout – Request timeout in seconds.
max_tokens – Maximum tokens to generate.
extra_body – Extra parameters merged into every request body.

property supports_bare_label_constraint: bool: True — vLLM guided_choice generates bare label text.

chat(messages: List[ChatMessage], *, temperature: float = 0.0, constrain_labels: List[str] | None = None, logprobs: bool = False, top_logprobs: int = 5) → ChatResponse[source]: Synchronous constrained chat completion via vLLM.

score(messages: List[ChatMessage], completion: str) → ScoringResponse[source]

Score a completion using vLLM’s completions endpoint.

Uses /v1/completions with prompt_logprobs to compute the per-token logprobs of the completion given the prompt context.

tokenize(text: str, *, context: str | None = None) → List[Token][source]

Tokenize text using vLLM’s tokenize endpoint.

vLLM provides a /tokenize endpoint (non-standard but supported in recent versions).

async achat(messages: List[ChatMessage], *, temperature: float = 0.0, constrain_labels: List[str] | None = None, logprobs: bool = False, top_logprobs: int = 5) → ChatResponse[source]: Async constrained chat completion via vLLM.

async ascore(messages: List[ChatMessage], completion: str) → ScoringResponse[source]: Async completion scoring via vLLM.

async atokenize(text: str, *, context: str | None = None) → List[Token][source]: Async tokenization via vLLM.

class ollama_classifier.backends.sglang.SGLangBackend(model: str, base_url: str = 'http://localhost:30000/v1', *, api_key: str | None = None, timeout: float = 120.0, max_tokens: int = 256, extra_body: Dict[str, Any] | None = None)[source]

Bases: LLMBackend

Backend for SGLang inference server.

SGLang is a fast serving system for large language models with an OpenAI-compatible API. It supports regex-guided decoding and logprobs. Logprobs are pre-mask (raw model logits before regex masking).

__init__(model: str, base_url: str = 'http://localhost:30000/v1', *, api_key: str | None = None, timeout: float = 120.0, max_tokens: int = 256, extra_body: Dict[str, Any] | None = None)[source]

Initialize the backend.

Parameters:

model – The model identifier to use.
base_url – Base URL of the inference server.
api_key – Optional API key. Defaults to "not-needed".
timeout – Request timeout in seconds.
max_tokens – Maximum tokens to generate.
extra_body – Extra parameters merged into every request body.

property supports_bare_label_constraint: bool: True — SGLang regex constraint generates bare label text.

chat(messages: List[ChatMessage], *, temperature: float = 0.0, constrain_labels: List[str] | None = None, logprobs: bool = False, top_logprobs: int = 5) → ChatResponse[source]: Synchronous constrained chat completion via SGLang.

score(messages: List[ChatMessage], completion: str) → ScoringResponse[source]: Score a completion using SGLang’s completions endpoint.

tokenize(text: str, *, context: str | None = None) → List[Token][source]: Tokenize text using SGLang’s tokenize endpoint.

async achat(messages: List[ChatMessage], *, temperature: float = 0.0, constrain_labels: List[str] | None = None, logprobs: bool = False, top_logprobs: int = 5) → ChatResponse[source]: Async constrained chat completion via SGLang.

async ascore(messages: List[ChatMessage], completion: str) → ScoringResponse[source]: Async completion scoring via SGLang.

async atokenize(text: str, *, context: str | None = None) → List[Token][source]: Async tokenization via SGLang.

class ollama_classifier.backends.llamacpp.LlamaCppBackend(model: str, base_url: str = 'http://localhost:8080/v1', *, api_key: str | None = None, timeout: float = 120.0, max_tokens: int = 256, extra_body: Dict[str, Any] | None = None)[source]

Bases: LLMBackend

Backend for llama.cpp server (llama-server).

llama.cpp provides a lightweight inference server with an OpenAI-compatible API. GBNF grammar constraints and logprobs are supported when compiled with the appropriate flags. Logprobs are pre-mask (model’s raw distribution before grammar masking).

Note

response_format with JSON schema on /v1/chat/completions is buggy in llama.cpp (GitHub issues #11988, #11847). This backend uses the non-standard grammar field with GBNF instead, which works reliably for bare label generation.

__init__(model: str, base_url: str = 'http://localhost:8080/v1', *, api_key: str | None = None, timeout: float = 120.0, max_tokens: int = 256, extra_body: Dict[str, Any] | None = None)[source]

Initialize the backend.

Parameters:

model – The model identifier to use.
base_url – Base URL of the inference server.
api_key – Optional API key. Defaults to "not-needed".
timeout – Request timeout in seconds.
max_tokens – Maximum tokens to generate.
extra_body – Extra parameters merged into every request body.

property supports_bare_label_constraint: bool: True — GBNF grammar generates bare label text.

chat(messages: List[ChatMessage], *, temperature: float = 0.0, constrain_labels: List[str] | None = None, logprobs: bool = False, top_logprobs: int = 5) → ChatResponse[source]: Synchronous constrained chat completion via llama.cpp server.

score(messages: List[ChatMessage], completion: str) → ScoringResponse[source]

Score a completion using llama.cpp’s completions endpoint with suffix.

Uses /v1/completions with suffix to compute the per-token logprobs of the completion given the prompt context (fill-in-the-middle style scoring).

tokenize(text: str, *, context: str | None = None) → List[Token][source]

Tokenize text using llama.cpp’s tokenize endpoint.

llama-server provides a /tokenize endpoint.

async achat(messages: List[ChatMessage], *, temperature: float = 0.0, constrain_labels: List[str] | None = None, logprobs: bool = False, top_logprobs: int = 5) → ChatResponse[source]: Async constrained chat completion via llama.cpp server.

async ascore(messages: List[ChatMessage], completion: str) → ScoringResponse[source]: Async completion scoring via llama.cpp server.

async atokenize(text: str, *, context: str | None = None) → List[Token][source]: Async tokenization via llama.cpp server.

Method Summary

LLMClassifier exposes the following methods. All return a ClassificationResult.

Choosing a Scoring Method

Use Case	Recommended Method
Speed is critical, approximate OK	`generate(max_calls=1)`
Adaptive accuracy within a budget	`generate(max_calls=K)`
Fully resolved, unlimited calls	`generate(max_calls=None)`
Gold-standard confidence (exact)	`classify`
Batch processing	`batch_classify` / `batch_generate`
Concurrent processing	Async variants (`aclassify`, `agenerate`)