mixle.data.validate module

Check that a dataset conforms to the spec a model expects, before fitting.

check_dataset(model, data) derives the model’s schema (mixle.data.schema.Schema.for_model()) and, over a sample of records, verifies each record (a) coerces to the schema’s logical types and (b) lies in the model’s support (finite log-density). It returns a report (and can raise on failure) so an invalid feed – wrong column types, out-of-support values, malformed records – is caught up front rather than surfacing later as a non-finite likelihood or a cryptic error deep in EM.

class DataReport(ok, n_checked, schema, issues=<factory>)[source]

Bases: object

Result of checking sample records against a model-derived schema and support.

Parameters:
check_dataset(model, data, *, sample=1000, check_support=True, raise_on_error=False)[source]

Validate data against the schema/support model expects (over the first sample records).

Records both type-coercion failures (wrong shape/dtype for a field) and, when check_support is True, support violations (a value the model assigns probability 0 -> -inf log-density). With raise_on_error the first batch of issues is raised as a ValueError.

sample must be a positive integer. A DataReport is meant to certify that the sampled records were actually inspected, so sample=0 used to check nothing and still return DataReport(ok=True, n_checked=0) – a clean bill of health for data nobody looked at – and a negative sample used to fail deep inside itertools.islice with an error that never mentions sample at all. Both are now rejected clearly, immediately, before any work starts.

Parameters:
Return type:

DataReport