mixle.task.vlm module

Provider-agnostic image-conditioned scoring with explicit exactness boundaries.

Sibling of mixle.task.llm, extended with an image. A VLM is anything with next_logprobs(image, prefix) -> [(token, log_prob), ...]. A generic string-token callback is useful for diagnostics and fixed candidate ranking, but it is not automatically an exact enumeration oracle.

Scope, deliberately: this targets an open-weight vision-language model served behind an OpenAI-compatible /v1/chat/completions endpoint that returns real per-token logprobs (vLLM, TGI, and similar self-hosted stacks serving e.g. LLaVA / Qwen-VL / …). These endpoints generally expose only a truncated top-logprob list and continue from concatenated token text. That cannot certify complete probability order because omitted tokens and server re-tokenization are unbounded. OpenAICompatVLM therefore requires explicit allow_truncated=True before returning such a callback.

Exact best-first enumeration is available through exact_token_scorer_for(), which requires tokenizer IDs and a complete log-probability vector for every vocabulary item at every prefix.

  • Diagnostic truncated decoding from an OpenAI-compatible endpoint:

    vlm = OpenAICompatVLM("http://localhost:8000/v1", "llava-onevision")
decode = vlm.next_logprobs_for(
    image,
    prompt="Describe this image in one sentence.",
    allow_truncated=True,
)
for tokens, log_prob in best_first_decode(decode, eos="<|eot_id|>", max_len=40, max_results=5):
    print("".join(tokens), log_prob)   # the 5 highest-probability captions, best first

  • Ranking a fixed candidate set by the model’s own teacher-forced probability, via mixle.enumeration.top_k_scored():

    score = score_fn_for(decode, eos="<|eot_id|>")
top_k_scored([("cat",), ("dog",), ("bird",)], score, k=3)

Teacher-forced candidate scoring includes the termination token and costs one call per candidate token plus one for EOS. Against the OpenAI-compatible bridge it remains a truncated diagnostic, not an exact probability, because the endpoint does not bind caller-visible strings to stable tokenizer IDs.

class VLM(*args, **kwargs)[source]

Bases: Protocol

Anything that can score an image-conditioned next-token continuation.

class TokenizerBoundVLM(*args, **kwargs)[source]

Bases: Protocol

Exact VLM contract: stable token IDs plus complete vocabulary support.

class CallableVLM(fn)[source]

Bases: object

Wrap a plain fn(image, prefix) -> [(token, log_prob), ...] as a VLM – local models and tests.

Parameters:

fn (Callable[[Any, tuple[str, ...]], Iterable[tuple[str, float]]])

next_logprobs_for(image)[source]

Bind an image without claiming complete support or stable tokenization.

Parameters:

image (Any)

Return type:

Callable[[tuple[str, …]], Iterable[tuple[str, float]]]

class OpenAICompatVLM(base_url, model, *, api_key=None, top_logprobs=20, timeout=60.0, continue_key='continue_final_message', continue_value=True, extra_body=None)[source]

Bases: object

A VLM backed by an OpenAI-compatible /v1/chat/completions endpoint that returns real per-token logprobs for an open-weight vision-language model (a vLLM- or TGI-served LLaVA / Qwen-VL / … deployment). See the module docstring for why this deliberately does not target proprietary hosted vision APIs.

Continuing a partial completion (every next_logprobs call after the first token of a decode) needs the server to prefill the given prefix rather than start generation fresh; this uses vLLM’s continue_final_message extension by default (append the prefix as a partial assistant message, set that flag). Pass continue_key/continue_value to target a server with a different convention.

Parameters:

  • base_url (str)

  • model (str)

  • api_key (str | None)

  • top_logprobs (int)

  • timeout (float)

  • continue_key (str)

  • continue_value (Any)

  • extra_body (dict[str, Any] | None)

next_logprobs(image, prefix, *, prompt, system=None)[source]

One image-conditioned next-token distribution given the tokens generated so far (prefix).

Parameters:
Return type:

list[tuple[str, float]]

next_logprobs_for(image, prompt, *, system=None, allow_truncated=False)[source]

Bind a truncated string-token diagnostic callback.

allow_truncated=True is an explicit acknowledgement that this callback cannot certify exact enumeration or exact candidate scores.

Parameters:

  • image (Any)

  • prompt (str)

  • system (str | None)

  • allow_truncated (bool)

Return type:

Callable[[tuple[str, …]], Iterable[tuple[str, float]]]

exact_token_scorer_for(model, image)[source]

Bind a tokenizer-ID, full-vocabulary VLM for exact enumeration.

Every callback result is checked for complete support, normalized probability mass, valid token IDs, and stable vocabulary size. The returned EOS token ID must be passed to the enumeration engine.

Parameters:

  • model (TokenizerBoundVLM)

  • image (Any)

Return type:

tuple[Callable[[tuple[int, …]], list[tuple[int, float]]], int]

score_candidate(next_logprobs_fn, candidate_tokens, *, eos)[source]

Score a fixed candidate plus its required termination token.

Walks one token at a time, reading off the ACTUAL log-probability of the candidate’s own next token at each step, then adds P(EOS | candidate). If a step’s returned continuations do not include the candidate’s token (e.g. it fell outside top_logprobs), returns -inf rather than silently dropping or padding the score. The empty candidate is therefore scored by its EOS probability, never assigned log probability zero by construction.

Parameters:
Return type:

float

score_fn_for(next_logprobs_fn, *, eos)[source]

Bind a next_logprobs function into the score(candidate) -> float shape mixle.enumeration.top_k_scored() expects directly, for ranking a fixed candidate set.

Parameters:
Return type:

Callable[[Sequence[Any]], float]