mixle.utils.automatic.profiling module

Data profiling and model recommendation for automatically-typed data.

Profiles sequences of observations to recommend per-field leaf estimators, measure unconditional pairwise dependency hints, and assemble the composite estimator via DatumNode. Estimator builders are imported from .factories.

class MarginalFieldProfile(path, role, count, missing_count, missing_fraction, observed_count, kind, recommendation, bits_per_obs=None, entropy_bits=None, cardinality=None, unique_fraction=None, effective_cardinality=None, is_constant=False, top_mass=None, numeric_mean=None, numeric_var=None, integer_min=None, integer_max=None, integer_density=None, model_scores_bits=<factory>, model_score_gap_bits=None, validation_scores_bits=<factory>, validation_recommendation=None, validation_score_gap_bits=None, validation_count=0, validation_notes=<factory>, gof_ks=None, gof_pvalue=None, notes=<factory>)[source]

Bases: object

Marginal evidence for one detected scalar field or structural feature.

Parameters:

  • path (tuple[Any, ...])

  • role (str)

  • count (int)

  • missing_count (int)

  • missing_fraction (float)

  • observed_count (int)

  • kind (str)

  • recommendation (str)

  • bits_per_obs (float | None)

  • entropy_bits (float | None)

  • cardinality (int | None)

  • unique_fraction (float | None)

  • effective_cardinality (float | None)

  • is_constant (bool)

  • top_mass (float | None)

  • numeric_mean (float | None)

  • numeric_var (float | None)

  • integer_min (int | None)

  • integer_max (int | None)

  • integer_density (float | None)

  • model_scores_bits (dict[str, float])

  • model_score_gap_bits (float | None)

  • validation_scores_bits (dict[str, float])

  • validation_recommendation (str | None)

  • validation_score_gap_bits (float | None)

  • validation_count (int)

  • validation_notes (list[str])

  • gof_ks (float | None)

  • gof_pvalue (float | None)

  • notes (list[str])

robust_recommendation()[source]

The model choice to actually trust, combining the in-sample (BIC) pick with the held-out validation check rather than leaving their disagreement as a note for a human to notice.

Held-out generalization is stronger evidence than an in-sample penalized-likelihood score: a flexible family (a 3-parameter shape family, a 2-component mixture) can win BIC by fitting noise/outliers in the training data that do not repeat in held-out data, which is exactly the failure mode BIC’s asymptotic parameter-count penalty does not always catch at small-to-moderate n. So: agree with recommendation whenever there is no held-out evidence, or the two already agree; defer to validation_recommendation only when it disagrees by a DECISIVE margin (the same AMBIGUOUS_SCORE_GAP_BITS threshold already used to flag a close call elsewhere in this module) – a marginal, ambiguous validation preference is not enough evidence to overturn BIC.

Return type:

str

model_weights()[source]

Return Schwarz (BIC) model weights over the scored candidates, summing to 1.

From per-observation code lengths L_i (model_scores_bits) and the observed count n, BIC_i = 2 * n * ln2 * L_i so the Schwarz weight is w_i proportional to exp(-0.5 * (BIC_i - min BIC)) = exp(-n * ln2 * (L_i - min L)) – an approximate posterior probability over the candidate models. Empty when nothing was scored.

Return type:

dict[str, float]

summary()[source]

Return a JSON-serializable summary of the marginal field evidence.

Return type:

dict[str, Any]

class MarginalSelectionDecision(path, training_recommendation, validation_recommendation, validation_score_gap_bits, selected_recommendation, validation_overrode)[source]

Bases: object

Immutable evidence and final family choice for one marginal field.

Parameters:

  • path (tuple[Any, ...])

  • training_recommendation (str)

  • validation_recommendation (str | None)

  • validation_score_gap_bits (float | None)

  • selected_recommendation (str)

  • validation_overrode (bool)

class AutomaticSelectionResult(estimator, decisions)[source]

Bases: object

The estimator returned to callers together with every decision that produced it.

Parameters:

  • estimator (ParameterEstimator)

  • decisions (tuple[MarginalSelectionDecision, ...])

class ValueArraySamplingReceipt(input_weight, input_cardinality, output_count, output_cardinality, cap, approximated, method, max_cdf_error_bound)[source]

Bases: object

Evidence describing a bounded expansion of weighted numeric values.

Parameters:

  • input_weight (float)

  • input_cardinality (int)

  • output_count (int)

  • output_cardinality (int)

  • cap (int)

  • approximated (bool)

  • method (str)

  • max_cdf_error_bound (float)

class PairwiseDependencyHint(left, right, mi_bits, adjusted_mi_bits, bic_gain_bits, normalized_mi, left_entropy_bits, right_entropy_bits, joint_count, method, p_value=None, notes=<factory>)[source]

Bases: object

Unconditional pairwise dependency hint measured from encoded values.

Parameters:
summary()[source]

Return a JSON-serializable summary of the dependency hint.

Return type:

dict[str, Any]

class StructureProfile(estimator, fields, pairwise_hints, warnings, sampled_rows, total_rows, dependency_tree_edges=<factory>, dependency_residual_edges=<factory>, dependency_redundancy_ratio=0.0, encoded_pairwise_fields=0, pairwise_fields_available=0, pairwise_pairs_available=0, pairwise_pairs_checked=0, pairwise_pair_strategy='none', selection=None)[source]

Bases: object

Structure-analysis result returned by analyze_structure.

Parameters:

  • estimator (ParameterEstimator)

  • fields (list[MarginalFieldProfile])

  • pairwise_hints (list[PairwiseDependencyHint])

  • warnings (list[str])

  • sampled_rows (int)

  • total_rows (int)

  • dependency_tree_edges (list[PairwiseDependencyHint])

  • dependency_residual_edges (list[PairwiseDependencyHint])

  • dependency_redundancy_ratio (float)

  • encoded_pairwise_fields (int)

  • pairwise_fields_available (int)

  • pairwise_pairs_available (int)

  • pairwise_pairs_checked (int)

  • pairwise_pair_strategy (str)

  • selection (AutomaticSelectionResult | None)

recommend()[source]

Return the estimator selected by the structure-analysis pass.

Return type:

ParameterEstimator

summary()[source]

Return a JSON-serializable summary of the structure profile.

Return type:

dict[str, Any]

explain()[source]

Render human-readable explanation lines for field and dependency choices.

Return type:

list[str]

format_path(path)[source]

Format a tuple path as a compact JSONPath-like field reference.

Parameters:

path (tuple[Any, ...])

Return type:

str

analyze_structure(data, pairwise=True, max_pairwise_fields=32, max_pairwise_pairs=512, max_cardinality=128, num_bins=8, sample_size=5000, validate_marginals=True, validation_fraction=0.25, max_validation_rows=1000, validation_min_count=30, validation_seed=17, mi_threshold_bits=0.05, bic_gain_threshold_bits=0.0, pairwise_permutations=0, permutation_alpha=0.05, dependency_tree=True, rng=None, pseudo_count=1.0, emp_suff_stat=True, use_bstats=False, modality=None)[source]

Profile data and return marginal recommendations plus pairwise hints.

Integer marginals are compared by BIC-style average code length. Pairwise hints report plug-in MI, finite-sample adjusted MI, and BIC edge gain. Pairwise hints are deliberately unconditional and encoded through low-overhead empirical/quantile codes. They are useful evidence, not proof of topology: latent classes or states can explain the same bit gains. Marginal validation is a bounded deterministic train/validation split over scalar fields, meant as a low-overhead predictive sanity check on the BIC choice.

Parameters:

  • pairwise (bool)

  • max_pairwise_fields (int)

  • max_pairwise_pairs (int)

  • max_cardinality (int)

  • num_bins (int)

  • sample_size (int | None)

  • validate_marginals (bool)

  • validation_fraction (float)

  • max_validation_rows (int)

  • validation_min_count (int)

  • validation_seed (int)

  • mi_threshold_bits (float)

  • bic_gain_threshold_bits (float)

  • pairwise_permutations (int)

  • permutation_alpha (float)

  • dependency_tree (bool)

  • rng (RandomState | None)

  • pseudo_count (float | None)

  • emp_suff_stat (bool)

  • use_bstats (bool)

  • modality (str | None)

Return type:

StructureProfile

class DatumNode(parent=None, data=None)[source]

Bases: object

Accumulates type/structure evidence for one slot of the data.

Tuples are treated as fixed-arity records (positional children). Lists, arrays, and other sized iterables are positional only if every observation has the same length (vector semantics); otherwise they are variable-length sequences of a merged element type with a length model. Sets map to a Bernoulli set model. Dicts map to keyed independent records in stats mode.

add_data(x)[source]

Add an iterable of observations to the node profile.

add_datum(x)[source]

Add one observation and update scalar/container child profiles.

copy()[source]

Return a deep copy of this profiling node and its children.

merge(x)[source]

Merge another compatible profiling node into this node.

get_estimator(pseudo_count=1.0, emp_suff_stat=True, use_bstats=False, *, recommendations=None, path=(), modality=None)[source]

Infer and return an estimator for the profiled observations.

What happens to values that are not ordinary numbers. mixle.inference.optimize(), the entry point most callers reach this through, summarizes this policy in its own docstring (see its “Identifier-like columns” and “Missing values” sections) and points back here for the full detail, written out below.

  • None and nan are the two spellings of ABSENT and mean one model: the leaf is wrapped in an OptionalDistribution whose missingness rate is fit from the data, so the model stays a normalized law and still samples. (They must agree: a pandas float Series stores None as nan behind the caller’s back.)

  • +inf and -inf are values rather than absences, and each sign present gets its own fitted-rate wrapper – an atom at the sentinel beside the base family, total mass one. Through 0.7.x these used the transparent (est_prob=False) wrapper instead, which scored the sentinel at log_density == 0.0, probability one, on top of an unscaled base: total mass 2.0, and every such row free in the objective.

  • A scalar column carrying BOTH numbers and strings is modeled as a typed dispatch mixture rather than retyped wholesale as a categorical over the observed values – see get_typed_mixture_estimator().

  • A scalar type this profiler does not recognize (datetime, Decimal, …), an ambiguous bool/number mix, and an identifier-like column are frozen: the empirical categorical over the observed values, held fixed by IgnoredDistribution. A value not seen at profiling time scores -inf there, the same finite-support behavior every automatically fitted categorical has.

Parameters:
normalize_input(data, *, rdd_cap=200000)[source]

Coerce a profiler input to a list of records, accepting more than a bare Python list.

Recognized inputs (each yields the same record stream the profiler/encoder consume):

  • a mixle DataSource (typed/structured) -> its records();

  • a pandas DataFrame (duck-typed via columns/itertuples; pandas is never imported) -> one record per row across its columns (scalar for a single column, tuple otherwise);

  • a Spark RDD -> the first rdd_cap rows (profiling works on a bounded sample);

  • a one-shot iterator (generator, map/filter/zip, a file object) -> materialized to a list, because the profiler iterates the records more than once (schema detection, then fitting) and a one-shot iterator would be exhausted after the first pass;

  • anything else (a list / sequence) is returned unchanged.

Parameters:

rdd_cap (int)

class RaggedRowDiagnosis(modal_width, modal_count, row_count, row_index, row_width, malformed)[source]

Bases: object

What the arity evidence says about rows that do not all carry the same number of fields.

malformed is the verdict: True when one arity is so dominant that the odd rows are a defect in a table (a lost or duplicated delimiter, a blank line), False when the majority is real but not overwhelming, which is the genuinely ambiguous case that gets fitted AND disclosed.

Parameters:

  • modal_width (int)

  • modal_count (int)

  • row_count (int)

  • row_index (int)

  • row_width (int)

  • malformed (bool)

property modal_share: float

Fraction of the rows carrying the dominant arity – the evidence the verdict rests on.

contract_error()[source]

The refusal for a malformed table, naming the offending row and the arity it must have.

Return type:

ContractError

note()[source]

One line naming the arity evidence, the reading taken, and the remedy for the other one.

Return type:

str

diagnose_ragged_rows(rows)[source]

Weigh the arity evidence for rows of differing width, or None when there is nothing to say.

None covers three distinct cases, all of which must stay silent: rows that are not positional containers, rows that all share one arity, and rows whose arities are spread widely enough that no single one is a majority – the fingerprint of real variable-length data.

Parameters:

rows (Sequence[Any])

Return type:

RaggedRowDiagnosis | None

get_estimator(data, pseudo_count=1.0, emp_suff_stat=True, use_bstats=False, *, modality=None, ragged='auto')[source]

Profile data and return the automatically selected estimator.

ragged governs rows that do not all carry the same number of fields, a shape that is genuinely ambiguous between a table with a malformed row and variable-length sequence data. 'auto' (the default) decides on the arity evidence: an overwhelmingly dominant arity means a malformed table and raises ContractError naming the offending row, a bare majority is fitted as a sequence with a warning saying so, and widely spread arities are fitted as a sequence silently. 'sequence' takes the sequence reading unconditionally and without comment – the escape hatch for variable-length data whose lengths happen to be nearly constant.

A field carrying a native +inf/-inf is fitted, not rejected – wrapped as a missingness sentinel with its own fitted rate, same as None/NaN – and a UserWarning discloses it (see _warn_nonfinite_reclassified_as_missing()); pass an explicit estimator instead to get the reject-on-non-finite behavior documented for the family-level default route.

An unrecognized scalar type, an ambiguous bool/number mix, and an identifier-like/high- cardinality field are all frozen instead: the empirical categorical over the values observed here, held fixed by IgnoredDistribution. This is silent (no warning) because it is the correct, unsurprising answer for a field of this shape – but a value not seen here later scores -inf under it (CategoricalDistribution’s documented default_value), which is worth knowing before scoring held-out or production data through the result; see DatumNode.get_estimator()’s docstring for the full policy and mixle.inference.optimize()’s docstring for what this means for a fitted model’s score.

Parameters:

  • pseudo_count (float | None)

  • emp_suff_stat (bool)

  • use_bstats (bool)

  • modality (str | None)

  • ragged (str)

get_prototype(data, *, seed=None, p=0.1, pseudo_count=1.0, emp_suff_stat=True, use_bstats=False, modality=None, ragged='auto')[source]

Infer a model structure and return an initialized prototype distribution.

Where get_estimator() returns the estimator used for fitting, this returns a concrete unfitted distribution whose tree mirrors the detected families. Use it when the inferred model shape should be inspected, customized, or passed to optimize(data, prototype) as a prototype.

proto = get_prototype(records) # see the inferred composite structure model = optimize(records, proto) # fit it (or pass proto to fit(…))

seed makes the randomized initialization reproducible; p is the per-observation keep-probability of the vectorized initializer. Remaining arguments mirror get_estimator().

Parameters: