Data Layer¶

mixle.data is the bridge between external data and the encoder contract used by distributions. It is optional: plain Python lists still work. Use the data layer when you need typed schemas, lazy sources, reproducible hashes, structure-aware partitioning, or reusable encoded payloads.

Core Objects¶

DataSource: A lazy, typed, structured source of records. A source exposes records() and encode(...) and carries optional schema and sample-structure metadata.
Schema: An ordered set of named fields with logical types such as Real, Count, Categorical, Text, Vector, Optional, and Nested.
SampleStructure: The exchangeability assumption for a dataset: IID, EXCHANGEABLE, SEQUENTIAL, or a partially exchangeable grouping.
exchangeability_check: A permutation-based diagnostic for whether numeric row order carries trend or regime-shift information.
dataset_hash and model_hash: Stable identifiers used by provenance, registries, drift checks, and reproducibility workflows.

Schema Validation¶

Schemas coerce records into the Python values expected by encoders and report problems early.

from mixle.data import Field, Real, Schema, Text, check_dataset

schema = Schema(
    (
        Field("country", Text()),
        Field("age", Real()),
        Field("spend", Real()),
    )
)

rows = [
    {"country": "US", "age": "41", "spend": 12.5},
    {"country": "CA", "age": 39, "spend": 8.0},
]

conformed = schema.conform(rows)

Use schema validation at the boundary of a pipeline. The fitted model can still receive ordinary Python records after they are conformed. Once a model exists, check_dataset(model, data) derives the model’s own schema (via Schema.for_model) and checks both coercion and support in one pass:

report = check_dataset(model, rows)

Do not let schema coercion hide data quality problems. Record coercion errors, missing fields, dropped records, and unit conversions when the conformed data feeds a release or production artifact.

Sources¶

as_source wraps in-memory data. open_source constructs a lazy source for supported external formats when the relevant optional extra is installed.

from mixle.data import EXCHANGEABLE, as_source

source = as_source(rows, structure=EXCHANGEABLE, schema=schema)
for record in source.records():
    print(record)

The public source system includes adapters for pandas, Arrow, SQL, Mongo, Hadoop, Spark, text, and graph data. Heavy integrations are imported lazily so the base install stays small.

External sources should record connection kind and query or file identity without embedding secrets. A source wrapper is not a data-governance boundary; the caller still owns credential handling and data-classification review.

Sample Structure¶

Sample structure tells mixle how records may be partitioned or interpreted.

Structure	Use when
`IID`	independent and identically distributed samples
`EXCHANGEABLE`	order does not matter, but exact iid assumptions are not asserted
`SEQUENTIAL`	record order is meaningful, as in time series or event streams
`partially_exchangeable(by=...)`	samples are exchangeable within groups, such as users or sessions

Structure-aware partitioning matters for streaming, distributed fitting, and validation. It prevents sequence or group boundaries from being split accidentally.

Use sample structure when splitting data. Random row splits can leak information across users, sessions, or time windows when the correct structure is sequential or partially exchangeable.

Exchangeability Diagnostics¶

Many high-level verbs assume rows can be pooled into one model or sampled as “more rows like these.” exchangeability_check tests that assumption for numeric scalar or tuple/list fields by looking for order trends and first-half versus second-half shifts.

from mixle.data import exchangeability_check

report = exchangeability_check(values, alpha=0.01, seed=0)
print(report.label)
print(report.as_dict())

The report label is one of:

exchangeable: The probes ran and found no order signal at the tested level.
trend: Values co-move with row position; fit a temporal or sequential model instead of pooling silently.
shift: The early and late halves differ in location; treat the rows as a regime change unless the split is deliberate.
inconclusive: No probe could run at all: fewer than 20 rows, no numeric field in the record shape, or every numeric field non-finite. This is deliberately not exchangeable – the assumption was never tested, rather than tested and upheld. report.exchangeable is False here for the same reason, so a caller reading only the boolean cannot mistake “we could not check” for “we checked and found nothing”.

exchangeable is failure to reject, not certification. The probes look for a monotone trend and for a between-halves location shift; data can violate exchangeability in ways neither probe examines, and a real violation can go undetected for want of power. report.multiplicity records the correction the verdict was decided under (method, family size, and alpha), without which the tested level is not interpretable, and each field record carries both the raw and the adjusted p-value (trend_p_raw/trend_p, shift_p_raw/shift_p).

mixle.inference.create and mixle.inference.synthesize run this check when applicable and store the verdict in provenance. It is a warning signal, not an automatic refusal, because some applications intentionally pool after domain review.

When a warning is overridden, record the reason. The override is part of the modeling assumption and should be visible in provenance.

Encoded Data¶

Most users do not need to call encoders directly. When repeated fits should reuse the same preprocessing boundary, save encoded data:

from mixle.data import load_encoded, save_encoded

encoder = model.dist_to_encoder()
encoded = encoder.seq_encode(rows)
save_encoded(encoded, "encoded.mixle", encoder=encoder)
encoded_again = load_encoded("encoded.mixle", encoder=encoder, trusted=True)

The encoded payload is the same kind of data consumed by optimize internally.

trusted=True is required and is not a formality. The body is pickle, so loading it executes whatever the file contains, and the stored integrity digest is computed from and stored inside that same file – it detects truncation and header tampering, not a file replaced wholesale. Pass it only for a path you control.

Encoded payloads should be versioned with the encoder or fitted model that created them. Reusing encoded data after a schema or encoder change can produce valid-looking arrays with the wrong meaning.

Hashes and Provenance¶

dataset_hash and model_hash provide durable identifiers:

from mixle.data import dataset_hash, model_hash

data_id = dataset_hash(rows)
model_id = model_hash(model)

Production helpers use these values in model headers, registries, drift reports, and lineage checks.

Hashes identify byte-level or canonicalized content; they do not prove that the data were appropriate for the model. Keep validation reports and data classification alongside hashes.

DataFrame and Spark Helpers¶

Optional adapters keep tabular and distributed data close to the same record shape used by ordinary lists.

from mixle.data import dataframe_records

rows = list(dataframe_records(df, fields=["country", "age", "spend"]))

Spark helpers include sampling functions for RDD-backed workflows. Use them after the local model shape works on an in-memory sample.

For distributed sources, record sampling policy and partition boundaries. A small local sample is useful for model shape, but it is not evidence that the distributed source has the same schema everywhere.

Result Egress¶

The adapters above move data into mixle. Three result types move data back out, into a pandas.DataFrame or a Parquet file, for the callers who need to hand a result to ordinary data-science tooling instead of consuming it in Python:

from mixle.inference import posterior, calibration_report

# posterior parameter draws -> one row per draw
theta = posterior(model, data, over="params")
theta.to_dataframe(1000).describe()
theta.to_parquet("theta.parquet", 1000)

# a calibration report's PIT histogram -> one row per bin
calibration_report(model, holdout).to_dataframe()

# an HMM's per-position state posterior -> one row per sequence position
model.latent_posterior(sequence).to_dataframe()

to_dataframe/to_parquet are methods on ParameterPosterior, CalibrationReport, and MarkovChainLatentPosterior – not a general sample(n)-to-DataFrame facility across every distribution family yet. Both need the pandas extra (pip install mixle[pandas]); to_parquet additionally needs a Parquet engine (pip install mixle[arrow] for pyarrow, or fastparquet).

Graph Data¶

Graph adapters expose graph observations to graph distributions without making graphs part of the scalar record path. GraphDataEncoder and GraphObservation are loaded lazily to avoid import cycles with graph families.

Practical Workflow¶

Start with a plain list of representative records.
Fit the model locally and confirm the estimator shape.
Add a Schema to make field coercion explicit.
Wrap the data in a DataSource when you need lazy loading, external data, sample structure, or partitioning.
Add hashes and encoded-data persistence once the workflow becomes repeatable or production-facing.

Release Evidence¶

For data-layer workflows, preserve:

schema and one representative conformed record;
validation report with coercion and missing-field behavior;
sample-structure declaration and split policy;
exchangeability diagnostics or override rationale;
source identity without credentials;
dataset hash and data classification; and
encoded-data versioning when payloads are persisted.

API Map¶

Import	Purpose
`DataSource`, `MaterializedSource`, `LazySource`, `as_source`	source abstraction and in-memory/lazy wrappers
`Schema`, `Field`, `Real`, `Count`, `Categorical`, `Text`	typed field schemas
`FieldType`, `Boolean`, `Timestamp`	additional schema types and the base field-type protocol
`IID`, `EXCHANGEABLE`, `SEQUENTIAL`, `partially_exchangeable`	sample-structure declarations
`exchangeability_check`, `ExchangeabilityReport`	row-order diagnostics used by creation and synthesis provenance
`check_dataset`, `DataReport`	validation and diagnostics
`dataset_hash`, `model_hash`	reproducibility identifiers
`save_encoded`, `load_encoded`	persist encoded payloads
`open_source`, `source_kinds`	external data source discovery and construction
`seq_encode_dataframe`, `sample_rdd`, `sample_seq_as_rdd`, `take_sample`	tabular and Spark/RDD sampling or encoding helpers