mixle.utils.automatic.profiling module

Data profiling and model recommendation for automatically-typed data.

Profiles sequences of observations to recommend per-field leaf estimators, measure unconditional pairwise dependency hints, and assemble the composite estimator via DatumNode. Estimator builders are imported from .factories.

class MarginalFieldProfile(path, role, count, missing_count, missing_fraction, observed_count, kind, recommendation, bits_per_obs=None, entropy_bits=None, cardinality=None, unique_fraction=None, effective_cardinality=None, is_constant=False, top_mass=None, numeric_mean=None, numeric_var=None, integer_min=None, integer_max=None, integer_density=None, model_scores_bits=<factory>, model_score_gap_bits=None, validation_scores_bits=<factory>, validation_recommendation=None, validation_score_gap_bits=None, validation_count=0, validation_notes=<factory>, gof_ks=None, gof_pvalue=None, notes=<factory>)

Bases: object

Marginal evidence for one detected scalar field or structural feature.

Parameters:

• path (tuple[Any, ...])

• role (str)

• count (int)

• missing_count (int)

• missing_fraction (float)

• observed_count (int)

• kind (str)

• recommendation (str)

• bits_per_obs (float | None)

• entropy_bits (float | None)

• cardinality (int | None)

• unique_fraction (float | None)

• effective_cardinality (float | None)

• is_constant (bool)

• top_mass (float | None)

• numeric_mean (float | None)

• numeric_var (float | None)

• integer_min (int | None)

• integer_max (int | None)

• integer_density (float | None)

• model_scores_bits (dict[str, float])

• model_score_gap_bits (float | None)

• validation_scores_bits (dict[str, float])

• validation_recommendation (str | None)

• validation_score_gap_bits (float | None)

• validation_count (int)

• validation_notes (list[str])

• gof_ks (float | None)

• gof_pvalue (float | None)

• notes (list[str])

robust_recommendation()

The model choice to actually trust, combining the in-sample (BIC) pick with the held-out validation check rather than leaving their disagreement as a note for a human to notice.

Held-out generalization is stronger evidence than an in-sample penalized-likelihood score: a flexible family (a 3-parameter shape family, a 2-component mixture) can win BIC by fitting noise/outliers in the training data that do not repeat in held-out data, which is exactly the failure mode BIC’s asymptotic parameter-count penalty does not always catch at small-to-moderate n. So: agree with recommendation whenever there is no held-out evidence, or the two already agree; defer to validation_recommendation only when it disagrees by a DECISIVE margin (the same AMBIGUOUS_SCORE_GAP_BITS threshold already used to flag a close call elsewhere in this module) – a marginal, ambiguous validation preference is not enough evidence to overturn BIC.

Return type:

str

model_weights()

Return Schwarz (BIC) model weights over the scored candidates, summing to 1.

From per-observation code lengths L_i (model_scores_bits) and the observed count n, BIC_i = 2 * n * ln2 * L_i so the Schwarz weight is w_i proportional to exp(-0.5 * (BIC_i - min BIC)) = exp(-n * ln2 * (L_i - min L)) – an approximate posterior probability over the candidate models. Empty when nothing was scored.

Return type:

dict[str, float]

summary()

Return a JSON-serializable summary of the marginal field evidence.

Return type:

dict[str, Any]

class PairwiseDependencyHint(left, right, mi_bits, adjusted_mi_bits, bic_gain_bits, normalized_mi, left_entropy_bits, right_entropy_bits, joint_count, method, p_value=None, notes=<factory>)

Bases: object

Unconditional pairwise dependency hint measured from encoded values.

Parameters:

• left (tuple[Any, ...])

• right (tuple[Any, ...])

• mi_bits (float)

• adjusted_mi_bits (float)

• bic_gain_bits (float)

• normalized_mi (float)

• left_entropy_bits (float)

• right_entropy_bits (float)

• joint_count (int)

• method (str)

• p_value (float | None)

• notes (list[str])

summary()

Return a JSON-serializable summary of the dependency hint.

Return type:

dict[str, Any]

class StructureProfile(estimator, fields, pairwise_hints, warnings, sampled_rows, total_rows, dependency_tree_edges=<factory>, dependency_residual_edges=<factory>, dependency_redundancy_ratio=0.0, encoded_pairwise_fields=0, pairwise_fields_available=0, pairwise_pairs_available=0, pairwise_pairs_checked=0, pairwise_pair_strategy='none')

Bases: object

Structure-analysis result returned by analyze_structure.

Parameters:

• estimator (ParameterEstimator)

• fields (list[MarginalFieldProfile])

• pairwise_hints (list[PairwiseDependencyHint])

• warnings (list[str])

• sampled_rows (int)

• total_rows (int)

• dependency_tree_edges (list[PairwiseDependencyHint])

• dependency_residual_edges (list[PairwiseDependencyHint])

• dependency_redundancy_ratio (float)

• encoded_pairwise_fields (int)

• pairwise_fields_available (int)

• pairwise_pairs_available (int)

• pairwise_pairs_checked (int)

• pairwise_pair_strategy (str)

recommend()

Return the estimator selected by the structure-analysis pass.

Return type:

ParameterEstimator

summary()

Return a JSON-serializable summary of the structure profile.

Return type:

dict[str, Any]

explain()

Render human-readable explanation lines for field and dependency choices.

Return type:

list[str]

format_path(path)

Format a tuple path as a compact JSONPath-like field reference.

Parameters:

path (tuple[Any, ...])

Return type:

str

analyze_structure(data, pairwise=True, max_pairwise_fields=32, max_pairwise_pairs=512, max_cardinality=128, num_bins=8, sample_size=5000, validate_marginals=True, validation_fraction=0.25, max_validation_rows=1000, validation_min_count=30, validation_seed=17, mi_threshold_bits=0.05, bic_gain_threshold_bits=0.0, pairwise_permutations=0, permutation_alpha=0.05, dependency_tree=True, rng=None, pseudo_count=1.0, emp_suff_stat=True, use_bstats=False)

Profile data and return marginal recommendations plus pairwise hints.

Integer marginals are compared by BIC-style average code length. Pairwise hints report plug-in MI, finite-sample adjusted MI, and BIC edge gain. Pairwise hints are deliberately unconditional and encoded through low-overhead empirical/quantile codes. They are useful evidence, not proof of topology: latent classes or states can explain the same bit gains. Marginal validation is a bounded deterministic train/validation split over scalar fields, meant as a low-overhead predictive sanity check on the BIC choice.

Parameters:

• pairwise (bool)

• max_pairwise_fields (int)

• max_pairwise_pairs (int)

• max_cardinality (int)

• num_bins (int)

• sample_size (int | None)

• validate_marginals (bool)

• validation_fraction (float)

• max_validation_rows (int)

• validation_min_count (int)

• validation_seed (int)

• mi_threshold_bits (float)

• bic_gain_threshold_bits (float)

• pairwise_permutations (int)

• permutation_alpha (float)

• dependency_tree (bool)

• rng (RandomState | None)

• pseudo_count (float | None)

• emp_suff_stat (bool)

• use_bstats (bool)

Return type:

StructureProfile

class DatumNode(parent=None, data=None)

Bases: object

Accumulates type/structure evidence for one slot of the data.

Tuples are treated as fixed-arity records (positional children). Lists, arrays, and other sized iterables are positional only if every observation has the same length (vector semantics); otherwise they are variable-length sequences of a merged element type with a length model. Sets map to a Bernoulli set model. Dicts map to keyed independent records in stats mode.

add_data(x)

Add an iterable of observations to the node profile.

add_datum(x)

Add one observation and update scalar/container child profiles.

copy()

Return a deep copy of this profiling node and its children.

merge(x)

Merge another compatible profiling node into this node.

get_estimator(pseudo_count=1.0, emp_suff_stat=True, use_bstats=False)

Infer and return an estimator for the profiled observations.

Parameters:

• pseudo_count (float | None)

• emp_suff_stat (bool)

• use_bstats (bool)

normalize_input(data, *, rdd_cap=200000)

Coerce a profiler input to a list of records, accepting more than a bare Python list.

Recognized inputs (each yields the same record stream the profiler/encoder consume): * a mixle DataSource (typed/structured) -> its records(); * a pandas DataFrame (duck-typed via columns/itertuples; pandas is never imported) ->

one record per row across its columns (scalar for a single column, tuple otherwise);

• a Spark RDD -> the first rdd_cap rows (profiling works on a bounded sample);

• anything else (a list / sequence) is returned unchanged.

Parameters:

rdd_cap (int)

get_estimator(data, pseudo_count=1.0, emp_suff_stat=True, use_bstats=False)

Profile data and return the automatically selected estimator.

Parameters:

• pseudo_count (float | None)

• emp_suff_stat (bool)

• use_bstats (bool)

get_prototype(data, *, seed=None, p=0.1, pseudo_count=1.0, emp_suff_stat=True, use_bstats=False)

Infer a model structure and return an initialized prototype distribution.

Where get_estimator() returns the estimator used for fitting, this returns a concrete unfitted distribution whose tree mirrors the detected families. Use it when the inferred model shape should be inspected, customized, or passed to optimize(data, prototype) as a prototype.

proto = get_prototype(records) # see the inferred composite structure model = optimize(records, proto) # fit it (or pass proto to fit(…))

seed makes the randomized initialization reproducible; p is the per-observation keep-probability of the vectorized initializer. Remaining arguments mirror get_estimator().

Parameters:

• seed (int | None)

• p (float)

• pseudo_count (float | None)

• emp_suff_stat (bool)

• use_bstats (bool)