mixle.utils.automatic.profiling module

Data profiling and model recommendation for automatically-typed data.

Profiles sequences of observations to recommend per-field leaf estimators, measure unconditional pairwise dependency hints, and assemble the composite estimator via DatumNode. Estimator builders are imported from .factories.

class MarginalFieldProfile(path, role, count, missing_count, missing_fraction, observed_count, kind, recommendation, bits_per_obs=None, entropy_bits=None, cardinality=None, unique_fraction=None, effective_cardinality=None, is_constant=False, top_mass=None, numeric_mean=None, numeric_var=None, integer_min=None, integer_max=None, integer_density=None, model_scores_bits=<factory>, model_score_gap_bits=None, validation_scores_bits=<factory>, validation_recommendation=None, validation_score_gap_bits=None, validation_count=0, validation_notes=<factory>, gof_ks=None, gof_pvalue=None, notes=<factory>)[source]

Bases: object

Marginal evidence for one detected scalar field or structural feature.

Parameters:

  • path (tuple[Any, ...])

  • role (str)

  • count (int)

  • missing_count (int)

  • missing_fraction (float)

  • observed_count (int)

  • kind (str)

  • recommendation (str)

  • bits_per_obs (float | None)

  • entropy_bits (float | None)

  • cardinality (int | None)

  • unique_fraction (float | None)

  • effective_cardinality (float | None)

  • is_constant (bool)

  • top_mass (float | None)

  • numeric_mean (float | None)

  • numeric_var (float | None)

  • integer_min (int | None)

  • integer_max (int | None)

  • integer_density (float | None)

  • model_scores_bits (dict[str, float])

  • model_score_gap_bits (float | None)

  • validation_scores_bits (dict[str, float])

  • validation_recommendation (str | None)

  • validation_score_gap_bits (float | None)

  • validation_count (int)

  • validation_notes (list[str])

  • gof_ks (float | None)

  • gof_pvalue (float | None)

  • notes (list[str])

robust_recommendation()[source]

The model choice to actually trust, combining the in-sample (BIC) pick with the held-out validation check rather than leaving their disagreement as a note for a human to notice.

Held-out generalization is stronger evidence than an in-sample penalized-likelihood score: a flexible family (a 3-parameter shape family, a 2-component mixture) can win BIC by fitting noise/outliers in the training data that do not repeat in held-out data, which is exactly the failure mode BIC’s asymptotic parameter-count penalty does not always catch at small-to-moderate n. So: agree with recommendation whenever there is no held-out evidence, or the two already agree; defer to validation_recommendation only when it disagrees by a DECISIVE margin (the same AMBIGUOUS_SCORE_GAP_BITS threshold already used to flag a close call elsewhere in this module) – a marginal, ambiguous validation preference is not enough evidence to overturn BIC.

Return type:

str

model_weights()[source]

Return Schwarz (BIC) model weights over the scored candidates, summing to 1.

From per-observation code lengths L_i (model_scores_bits) and the observed count n, BIC_i = 2 * n * ln2 * L_i so the Schwarz weight is w_i proportional to exp(-0.5 * (BIC_i - min BIC)) = exp(-n * ln2 * (L_i - min L)) – an approximate posterior probability over the candidate models. Empty when nothing was scored.

Return type:

dict[str, float]

summary()[source]

Return a JSON-serializable summary of the marginal field evidence.

Return type:

dict[str, Any]

class MarginalSelectionDecision(path, training_recommendation, validation_recommendation, validation_score_gap_bits, selected_recommendation, validation_overrode)[source]

Bases: object

Immutable evidence and final family choice for one marginal field.

Parameters:

  • path (tuple[Any, ...])

  • training_recommendation (str)

  • validation_recommendation (str | None)

  • validation_score_gap_bits (float | None)

  • selected_recommendation (str)

  • validation_overrode (bool)

class AutomaticSelectionResult(estimator, decisions)[source]

Bases: object

The estimator returned to callers together with every decision that produced it.

Parameters:

  • estimator (ParameterEstimator)

  • decisions (tuple[MarginalSelectionDecision, ...])

class ValueArraySamplingReceipt(input_weight, input_cardinality, output_count, output_cardinality, cap, approximated, method, max_cdf_error_bound)[source]

Bases: object

Evidence describing a bounded expansion of weighted numeric values.

Parameters:

  • input_weight (float)

  • input_cardinality (int)

  • output_count (int)

  • output_cardinality (int)

  • cap (int)

  • approximated (bool)

  • method (str)

  • max_cdf_error_bound (float)

class PairwiseDependencyHint(left, right, mi_bits, adjusted_mi_bits, bic_gain_bits, normalized_mi, left_entropy_bits, right_entropy_bits, joint_count, method, p_value=None, notes=<factory>)[source]

Bases: object

Unconditional pairwise dependency hint measured from encoded values.

Parameters:
summary()[source]

Return a JSON-serializable summary of the dependency hint.

Return type:

dict[str, Any]

class StructureProfile(estimator, fields, pairwise_hints, warnings, sampled_rows, total_rows, dependency_tree_edges=<factory>, dependency_residual_edges=<factory>, dependency_redundancy_ratio=0.0, encoded_pairwise_fields=0, pairwise_fields_available=0, pairwise_pairs_available=0, pairwise_pairs_checked=0, pairwise_pair_strategy='none', selection=None)[source]

Bases: object

Structure-analysis result returned by analyze_structure.

Parameters:

  • estimator (ParameterEstimator)

  • fields (list[MarginalFieldProfile])

  • pairwise_hints (list[PairwiseDependencyHint])

  • warnings (list[str])

  • sampled_rows (int)

  • total_rows (int)

  • dependency_tree_edges (list[PairwiseDependencyHint])

  • dependency_residual_edges (list[PairwiseDependencyHint])

  • dependency_redundancy_ratio (float)

  • encoded_pairwise_fields (int)

  • pairwise_fields_available (int)

  • pairwise_pairs_available (int)

  • pairwise_pairs_checked (int)

  • pairwise_pair_strategy (str)

  • selection (AutomaticSelectionResult | None)

recommend()[source]

Return the estimator selected by the structure-analysis pass.

Return type:

ParameterEstimator

summary()[source]

Return a JSON-serializable summary of the structure profile.

Return type:

dict[str, Any]

explain()[source]

Render human-readable explanation lines for field and dependency choices.

Return type:

list[str]

format_path(path)[source]

Format a tuple path as a compact JSONPath-like field reference.

Parameters:

path (tuple[Any, ...])

Return type:

str

analyze_structure(data, pairwise=True, max_pairwise_fields=32, max_pairwise_pairs=512, max_cardinality=128, num_bins=8, sample_size=5000, validate_marginals=True, validation_fraction=0.25, max_validation_rows=1000, validation_min_count=30, validation_seed=17, mi_threshold_bits=0.05, bic_gain_threshold_bits=0.0, pairwise_permutations=0, permutation_alpha=0.05, dependency_tree=True, rng=None, pseudo_count=1.0, emp_suff_stat=True, use_bstats=False, modality=None)[source]

Profile data and return marginal recommendations plus pairwise hints.

Integer marginals are compared by BIC-style average code length. Pairwise hints report plug-in MI, finite-sample adjusted MI, and BIC edge gain. Pairwise hints are deliberately unconditional and encoded through low-overhead empirical/quantile codes. They are useful evidence, not proof of topology: latent classes or states can explain the same bit gains. Marginal validation is a bounded deterministic train/validation split over scalar fields, meant as a low-overhead predictive sanity check on the BIC choice.

Parameters:

  • pairwise (bool)

  • max_pairwise_fields (int)

  • max_pairwise_pairs (int)

  • max_cardinality (int)

  • num_bins (int)

  • sample_size (int | None)

  • validate_marginals (bool)

  • validation_fraction (float)

  • max_validation_rows (int)

  • validation_min_count (int)

  • validation_seed (int)

  • mi_threshold_bits (float)

  • bic_gain_threshold_bits (float)

  • pairwise_permutations (int)

  • permutation_alpha (float)

  • dependency_tree (bool)

  • rng (RandomState | None)

  • pseudo_count (float | None)

  • emp_suff_stat (bool)

  • use_bstats (bool)

  • modality (str | None)

Return type:

StructureProfile

class DatumNode(parent=None, data=None)[source]

Bases: object

Accumulates type/structure evidence for one slot of the data.

Tuples are treated as fixed-arity records (positional children). Lists, arrays, and other sized iterables are positional only if every observation has the same length (vector semantics); otherwise they are variable-length sequences of a merged element type with a length model. Sets map to a Bernoulli set model. Dicts map to keyed independent records in stats mode.

add_data(x)[source]

Add an iterable of observations to the node profile.

add_datum(x)[source]

Add one observation and update scalar/container child profiles.

copy()[source]

Return a deep copy of this profiling node and its children.

merge(x)[source]

Merge another compatible profiling node into this node.

get_estimator(pseudo_count=1.0, emp_suff_stat=True, use_bstats=False, *, recommendations=None, path=(), modality=None, unimodal_leaves=False)[source]

Infer and return an estimator for the profiled observations.

unimodal_leaves keeps a per-field 2-component Gaussian mixture out of the candidate set, for a caller building the components of an outer mixture (see _numeric_candidate_bics()).

What happens to values that are not ordinary numbers. mixle.inference.optimize(), the entry point most callers reach this through, summarizes this policy in its own docstring (see its “Identifier-like columns” and “Missing values” sections) and points back here for the full detail, written out below.

  • None and nan are the two spellings of ABSENT and mean one model: the leaf is wrapped in an OptionalDistribution whose missingness rate is fit from the data, so the model stays a normalized law and still samples. (They must agree: a pandas float Series stores None as nan behind the caller’s back.)

  • +inf and -inf are values rather than absences, and each sign present gets its own fitted-rate wrapper – an atom at the sentinel beside the base family, total mass one. Through 0.7.x these used the transparent (est_prob=False) wrapper instead, which scored the sentinel at log_density == 0.0, probability one, on top of an unscaled base: total mass 2.0, and every such row free in the objective.

  • A scalar column carrying BOTH numbers and strings is modeled as a typed dispatch mixture rather than retyped wholesale as a categorical over the observed values – see get_typed_mixture_estimator().

  • A scalar type this profiler does not recognize (datetime, Decimal, …), an ambiguous bool/number mix, and an identifier-like column are frozen: the empirical categorical over the observed values, held fixed by IgnoredDistribution. A value not seen at profiling time scores -inf there, the same finite-support behavior every automatically fitted categorical has.

Parameters:
normalize_input(data, *, rdd_cap=200000)[source]

Coerce a profiler input to a list of records, accepting more than a bare Python list.

Recognized inputs (each yields the same record stream the profiler/encoder consume):

  • a mixle DataSource (typed/structured) -> its records();

  • a pandas DataFrame (duck-typed via columns/itertuples; pandas is never imported) -> one record per row across its columns (scalar for a single column, tuple otherwise);

  • a Spark RDD -> the first rdd_cap rows (profiling works on a bounded sample);

  • a one-shot iterator (generator, map/filter/zip, a file object) -> materialized to a list, because the profiler iterates the records more than once (schema detection, then fitting) and a one-shot iterator would be exhausted after the first pass;

  • anything else (a list / sequence) is returned unchanged.

Parameters:

rdd_cap (int)

class RaggedRowDiagnosis(modal_width, modal_count, row_count, row_index, row_width, malformed)[source]

Bases: object

What the arity evidence says about rows that do not all carry the same number of fields.

malformed is the verdict: True when one arity is so dominant that the odd rows are a defect in a table (a lost or duplicated delimiter, a blank line), False when the majority is real but not overwhelming, which is the genuinely ambiguous case that gets fitted AND disclosed.

Parameters:

  • modal_width (int)

  • modal_count (int)

  • row_count (int)

  • row_index (int)

  • row_width (int)

  • malformed (bool)

property modal_share: float

Fraction of the rows carrying the dominant arity – the evidence the verdict rests on.

contract_error()[source]

The refusal for a malformed table, naming the offending row and the arity it must have.

Return type:

ContractError

note()[source]

One line naming the arity evidence, the reading taken, and the remedy for the other one.

Return type:

str

diagnose_ragged_rows(rows)[source]

Weigh the arity evidence for rows of differing width, or None when there is nothing to say.

None covers three distinct cases, all of which must stay silent: rows that are not positional containers, rows that all share one arity, and rows whose arities are spread widely enough that no single one is a majority – the fingerprint of real variable-length data.

Parameters:

rows (Sequence[Any])

Return type:

RaggedRowDiagnosis | None

get_estimator(data, pseudo_count=1.0, emp_suff_stat=True, use_bstats=False, *, modality=None, ragged='auto', unimodal_leaves=False)[source]

Profile data and return the automatically selected estimator.

unimodal_leaves=True keeps a per-field 2-component Gaussian mixture out of the candidate set. Pass it when the result will be a COMPONENT of an outer mixture: the outer mixture is what models multimodality, and a mixture inside a component lets one component absorb several regimes – the identifiability trap learn_mixture_structure() closes with field_estimators and get_dpm_mixture() now closes by default.

ragged governs rows that do not all carry the same number of fields, a shape that is genuinely ambiguous between a table with a malformed row and variable-length sequence data. 'auto' (the default) decides on the arity evidence: an overwhelmingly dominant arity means a malformed table and raises ContractError naming the offending row, a bare majority is fitted as a sequence with a warning saying so, and widely spread arities are fitted as a sequence silently. 'sequence' takes the sequence reading unconditionally and without comment – the escape hatch for variable-length data whose lengths happen to be nearly constant.

A field carrying a native +inf/-inf is fitted, not rejected – wrapped as a missingness sentinel with its own fitted rate, same as None/NaN – and a UserWarning discloses it (see _warn_nonfinite_reclassified_as_missing()); pass an explicit estimator instead to get the reject-on-non-finite behavior documented for the family-level default route.

An unrecognized scalar type, an ambiguous bool/number mix, and an identifier-like/high- cardinality field are all frozen instead: the empirical categorical over the values observed here, held fixed by IgnoredDistribution. This is silent (no warning) because it is the correct, unsurprising answer for a field of this shape – but a value not seen here later scores -inf under it (CategoricalDistribution’s documented default_value), which is worth knowing before scoring held-out or production data through the result; see DatumNode.get_estimator()’s docstring for the full policy and mixle.inference.optimize()’s docstring for what this means for a fitted model’s score.

Parameters:

  • pseudo_count (float | None)

  • emp_suff_stat (bool)

  • use_bstats (bool)

  • modality (str | None)

  • ragged (str)

  • unimodal_leaves (bool)

get_prototype(data, *, seed=None, p=0.1, pseudo_count=1.0, emp_suff_stat=True, use_bstats=False, modality=None, ragged='auto')[source]

Infer a model structure and return an initialized prototype distribution.

Where get_estimator() returns the estimator used for fitting, this returns a concrete unfitted distribution whose tree mirrors the detected families. Use it when the inferred model shape should be inspected, customized, or passed to optimize(data, prototype) as a prototype.

proto = get_prototype(records) # see the inferred composite structure model = optimize(records, proto) # fit it (or pass proto to fit(…))

seed makes the randomized initialization reproducible; p is the per-observation keep-probability of the vectorized initializer. Remaining arguments mirror get_estimator().

Parameters: