mixle.engines.formats module

Numeric format codecs across mixle’s precision spectrum: low-bit float, fixed-point, and codebook/VQ – the compression + “fixed precision” end, each with a provable error bound so a precision-allocation pass can pick the smallest format that preserves accuracy.

A NumericFormat is a lossy codec: quantize maps a float64 array to a compact representation, dequantize maps it back, and max_abs_error / max_rel_error bound the round trip. The bound is the “logic” a caller uses to spend minimal bits while keeping error under a target (see min_float_mantissa_bits()). Fixed-point and codebook codecs store an actually smaller array (real compression, vectorized in numpy) with a hard range/index limit enforced at construction or quantize time; the float codec instead rounds each value’s MANTISSA to an n-bit band’s precision with an unbounded exponent (no overflow/underflow/subnormal handling – see FloatFormat) to measure that band’s rounding accuracy at any magnitude (true sub-byte bit-packing, and a real range-limited fpN encoding, are the Cython/C tail).

class NumericFormat

Bases: object

Base codec: quantize / dequantize plus round-trip error bounds and a storage bit count.

quantize(x)

Encode values into the format’s storage representation.

Parameters:

x (Any)

Return type:

Any

dequantize(q)

Decode stored values back to float64.

Parameters:

q (Any)

Return type:

ndarray

round_trip(x)

Quantize and immediately dequantize values.

Parameters:

x (Any)

Return type:

ndarray

measured_max_abs_error(x)

Empirical max absolute round-trip error over x (codecs also expose analytic bounds).

Parameters:

x (Any)

Return type:

float

compression_ratio()

Stored bits per value relative to float64.

Return type:

float

class FloatFormat(mantissa_bits, exp_bits=11)

Bases: NumericFormat

Round-to-nearest MANTISSA rounding at mantissa_bits, nominally styled after an IEEE-ish fpN split (see fp()).

quantize rounds each value’s significand to mantissa_bits explicit bits (round-to-nearest); the exponent passes through UNCHANGED, with no range limit. exp_bits therefore only feeds name / bits_per_value (the advertised bit budget an allocation pass compares against) – it is NOT enforced: there is no overflow/underflow, no saturation or flush-to-zero, no subnormals, and no NaN/Inf bit-pattern encoding. This is a deliberate scope choice, not an oversight: an unbounded exponent is what makes max_rel_error a genuine bound for any finite magnitude (see that property’s docstring), so a caller can read the round trip as a pure mantissa-rounding measurement independent of range.

A consequence: this codec does NOT faithfully emulate a real bounded fpN storage format. A real 8-bit float cannot represent 1e100 at all, but FloatFormat.fp(8).round_trip([1e100]) rounds 1e100’s mantissa to 3 bits and returns a value that is still ~1e100 in magnitude – by design, per the above, not a bug. Do not use fp(8)/fp(16)/etc. where actual overflow/underflow/saturation behavior matters; they measure mantissa-rounding error only.

Parameters:

• mantissa_bits (int)

• exp_bits (int)

classmethod fp(total_bits)

Build a format whose mantissa_bits/exp_bits follow an IEEE-like split of total_bits (n = 1..1024+). Only the resulting mantissa_bits affects quantize’s numeric behavior – exp_bits is bit-budget bookkeeping only (see the class docstring); this does NOT build a range-limited fpN codec.

Parameters:

total_bits (int)

Return type:

FloatFormat

property max_rel_error: float

Return the worst-case relative rounding error – a universal bound, true for every finite input regardless of magnitude (the exponent is never range-limited; see the class docstring).

quantize(x)

Round values to this mantissa precision; the exponent passes through exactly, unclamped.

Parameters:

x (Any)

Return type:

ndarray

dequantize(q)

Decode quantized floating values to float64.

Parameters:

q (Any)

Return type:

ndarray

class FixedPointFormat(frac_bits, int_bits=31)

Bases: NumericFormat

Fixed-point: store round(x * 2**frac_bits) as an integer; real compression + a hard error bound.

int_bits sets the representable magnitude [-2**int_bits, 2**int_bits); out-of-range clamps. in_range_max_abs_error == 2**-(frac_bits+1) bounds round-to-nearest only for values inside that range. The universal max_abs_error is infinite because saturation error is unbounded. Storage is a plain int32/int64 – there is no arbitrary-width packed storage – so the declared total width 1 + int_bits + frac_bits cannot exceed 64 bits; wider requests raise at construction instead of silently overflowing the int64 cast (and falsifying max_abs_error).

Parameters:

• frac_bits (int)

• int_bits (int)

property max_abs_error: float

Return the universal error bound; saturation makes it unbounded.

property in_range_max_abs_error: float

Return the half-step error bound for inputs inside the representable range.

quantize(x)

Encode values as clipped scaled integers.

Parameters:

x (Any)

Return type:

ndarray

dequantize(q)

Decode scaled integers back to float64 values.

Parameters:

q (Any)

Return type:

ndarray

class CodebookFormat(codebook)

Bases: NumericFormat

Scalar vector-quantization: store an index into a learned codebook; log2(K) bits per value.

The genuine pure-numpy compression codec – quantize gathers the nearest code (an unsigned index array), dequantize gathers the code values back. Fit the codebook to data with fit().

The codebook must be finite and nonempty (validated at construction, ValueError otherwise). Duplicate entries are allowed: a repeated code costs a wasted, unreachable index but does not break nearest-code assignment or decoding (both entries decode to the same, correct value), and fit()’s Lloyd iteration can legitimately converge two cluster means onto the same value on degenerate/skewed data – rejecting that outcome would be pure loss, not a correctness fix.

Parameters:

codebook (Any)

classmethod fit(data, n_codes, iters=25, seed=0)

Learn n_codes codes by 1-D k-means (Lloyd) on data; codes are the cluster means.

Raises ValueError if n_codes or iters is not a positive integer.

Parameters:

• data (Any)

• n_codes (int)

• iters (int)

• seed (int)

Return type:

CodebookFormat

quantize(x)

Map values to nearest codebook indices.

Parameters:

x (Any)

Return type:

ndarray

dequantize(q)

Map codebook indices back to representative values.

Parameters:

q (Any)

Return type:

ndarray

compress(x)

Quantize x and pack the indices to bytes: returns (packed_uint8, count).

For K <= 256 codes the indices are sub-byte or byte-exact and pack_bits() bit-packs them (e.g. 16 codes -> 4-bit indices -> 2 values/byte -> 16x vs float64). Larger codebooks need 16- or 32-bit indices, which are not sub-byte-packable – those are stored byte-aligned (little-endian), still returned as a flat uint8 array.

Parameters:

x (Any)

Return type:

tuple[ndarray, int]

decompress(packed, count)

Inverse of compress(): unpack indices and gather the codebook back to float64.

Parameters:

• packed (Any)

• count (int)

Return type:

ndarray

min_float_mantissa_bits(target_rel_error)

Smallest mantissa-bit count whose round-to-nearest relative error meets target_rel_error.

The error-tracing primitive: given a tolerated relative error, return the minimal float precision that preserves it – i.e. spend the fewest bits the accuracy budget allows.

Parameters:

target_rel_error (float)

Return type:

int