mixle.utils.automatic.profiling module

Data profiling and model recommendation for automatically-typed data.

Profiles sequences of observations to recommend per-field leaf estimators, measure unconditional pairwise dependency hints, and assemble the composite estimator via DatumNode. Estimator builders are imported from .factories.

class MarginalFieldProfile(path, role, count, missing_count, missing_fraction, observed_count, kind, recommendation, bits_per_obs=None, entropy_bits=None, cardinality=None, unique_fraction=None, effective_cardinality=None, is_constant=False, top_mass=None, numeric_mean=None, numeric_var=None, integer_min=None, integer_max=None, integer_density=None, model_scores_bits=<factory>, model_score_gap_bits=None, validation_scores_bits=<factory>, validation_recommendation=None, validation_score_gap_bits=None, validation_count=0, validation_notes=<factory>, gof_ks=None, gof_pvalue=None, notes=<factory>)[source]

Bases: object

Marginal evidence for one detected scalar field or structural feature.

Parameters:

  • path (tuple[Any, ...])

  • role (str)

  • count (int)

  • missing_count (int)

  • missing_fraction (float)

  • observed_count (int)

  • kind (str)

  • recommendation (str)

  • bits_per_obs (float | None)

  • entropy_bits (float | None)

  • cardinality (int | None)

  • unique_fraction (float | None)

  • effective_cardinality (float | None)

  • is_constant (bool)

  • top_mass (float | None)

  • numeric_mean (float | None)

  • numeric_var (float | None)

  • integer_min (int | None)

  • integer_max (int | None)

  • integer_density (float | None)

  • model_scores_bits (dict[str, float])

  • model_score_gap_bits (float | None)

  • validation_scores_bits (dict[str, float])

  • validation_recommendation (str | None)

  • validation_score_gap_bits (float | None)

  • validation_count (int)

  • validation_notes (list[str])

  • gof_ks (float | None)

  • gof_pvalue (float | None)

  • notes (list[str])

path: tuple[Any, ...]
role: str
count: int
missing_count: int
missing_fraction: float
observed_count: int
kind: str
recommendation: str
bits_per_obs: float | None = None
entropy_bits: float | None = None
cardinality: int | None = None
unique_fraction: float | None = None
effective_cardinality: float | None = None
is_constant: bool = False
top_mass: float | None = None
numeric_mean: float | None = None
numeric_var: float | None = None
integer_min: int | None = None
integer_max: int | None = None
integer_density: float | None = None
model_scores_bits: dict[str, float]
model_score_gap_bits: float | None = None
validation_scores_bits: dict[str, float]
validation_recommendation: str | None = None
validation_score_gap_bits: float | None = None
validation_count: int = 0
validation_notes: list[str]
gof_ks: float | None = None
gof_pvalue: float | None = None
notes: list[str]
model_weights()[source]

Return Schwarz (BIC) model weights over the scored candidates, summing to 1.

From per-observation code lengths L_i (model_scores_bits) and the observed count n, BIC_i = 2 * n * ln2 * L_i so the Schwarz weight is w_i proportional to exp(-0.5 * (BIC_i - min BIC)) = exp(-n * ln2 * (L_i - min L)) – an approximate posterior probability over the candidate models. Empty when nothing was scored.

Return type:

dict[str, float]

summary()[source]
Return type:

dict[str, Any]

class PairwiseDependencyHint(left, right, mi_bits, adjusted_mi_bits, bic_gain_bits, normalized_mi, left_entropy_bits, right_entropy_bits, joint_count, method, p_value=None, notes=<factory>)[source]

Bases: object

Unconditional pairwise dependency hint measured from encoded values.

Parameters:
left: tuple[Any, ...]
right: tuple[Any, ...]
mi_bits: float
adjusted_mi_bits: float
bic_gain_bits: float
normalized_mi: float
left_entropy_bits: float
right_entropy_bits: float
joint_count: int
method: str
p_value: float | None = None
notes: list[str]
summary()[source]
Return type:

dict[str, Any]

class StructureProfile(estimator, fields, pairwise_hints, warnings, sampled_rows, total_rows, dependency_tree_edges=<factory>, dependency_residual_edges=<factory>, dependency_redundancy_ratio=0.0, encoded_pairwise_fields=0, pairwise_fields_available=0, pairwise_pairs_available=0, pairwise_pairs_checked=0, pairwise_pair_strategy='none')[source]

Bases: object

Structure-analysis result returned by analyze_structure.

Parameters:

  • estimator (ParameterEstimator)

  • fields (list[MarginalFieldProfile])

  • pairwise_hints (list[PairwiseDependencyHint])

  • warnings (list[str])

  • sampled_rows (int)

  • total_rows (int)

  • dependency_tree_edges (list[PairwiseDependencyHint])

  • dependency_residual_edges (list[PairwiseDependencyHint])

  • dependency_redundancy_ratio (float)

  • encoded_pairwise_fields (int)

  • pairwise_fields_available (int)

  • pairwise_pairs_available (int)

  • pairwise_pairs_checked (int)

  • pairwise_pair_strategy (str)

estimator: ParameterEstimator
fields: list[MarginalFieldProfile]
pairwise_hints: list[PairwiseDependencyHint]
warnings: list[str]
sampled_rows: int
total_rows: int
dependency_tree_edges: list[PairwiseDependencyHint]
dependency_residual_edges: list[PairwiseDependencyHint]
dependency_redundancy_ratio: float = 0.0
encoded_pairwise_fields: int = 0
pairwise_fields_available: int = 0
pairwise_pairs_available: int = 0
pairwise_pairs_checked: int = 0
pairwise_pair_strategy: str = 'none'
recommend()[source]
Return type:

ParameterEstimator

summary()[source]
Return type:

dict[str, Any]

explain()[source]
Return type:

list[str]

format_path(path)[source]
Parameters:

path (tuple[Any, ...])

Return type:

str

analyze_structure(data, pairwise=True, max_pairwise_fields=32, max_pairwise_pairs=512, max_cardinality=128, num_bins=8, sample_size=5000, validate_marginals=True, validation_fraction=0.25, max_validation_rows=1000, validation_min_count=30, validation_seed=17, mi_threshold_bits=0.05, bic_gain_threshold_bits=0.0, pairwise_permutations=0, permutation_alpha=0.05, dependency_tree=True, rng=None, pseudo_count=1.0, emp_suff_stat=True, use_bstats=False)[source]

Profile data and return marginal recommendations plus pairwise hints.

Integer marginals are compared by BIC-style average code length. Pairwise hints report plug-in MI, finite-sample adjusted MI, and BIC edge gain. Pairwise hints are deliberately unconditional and encoded through cheap empirical/quantile codes. They are useful evidence, not proof of topology: latent classes or states can explain the same bit gains. Marginal validation is a bounded deterministic train/validation split over scalar fields, meant as a cheap predictive sanity check on the BIC choice.

Parameters:

  • pairwise (bool)

  • max_pairwise_fields (int)

  • max_pairwise_pairs (int)

  • max_cardinality (int)

  • num_bins (int)

  • sample_size (int | None)

  • validate_marginals (bool)

  • validation_fraction (float)

  • max_validation_rows (int)

  • validation_min_count (int)

  • validation_seed (int)

  • mi_threshold_bits (float)

  • bic_gain_threshold_bits (float)

  • pairwise_permutations (int)

  • permutation_alpha (float)

  • dependency_tree (bool)

  • rng (RandomState | None)

  • pseudo_count (float | None)

  • emp_suff_stat (bool)

  • use_bstats (bool)

Return type:

StructureProfile

class DatumNode(parent=None, data=None)[source]

Bases: object

Accumulates type/structure evidence for one slot of the data.

Tuples are treated as fixed-arity records (positional children). Lists, arrays, and other sized iterables are positional only if every observation has the same length (vector semantics); otherwise they are variable-length sequences of a merged element type with a length model. Sets map to a Bernoulli set model. Dicts map to keyed independent records in stats mode.

add_data(x)[source]
add_datum(x)[source]
copy()[source]
merge(x)[source]
get_estimator(pseudo_count=1.0, emp_suff_stat=True, use_bstats=False)[source]
Parameters:

  • pseudo_count (float | None)

  • emp_suff_stat (bool)

  • use_bstats (bool)

normalize_input(data, *, rdd_cap=200000)[source]

Coerce a profiler input to a list of records, accepting more than a bare Python list.

Recognized inputs (each yields the same record stream the profiler/encoder consume): * a mixle DataSource (typed/structured) -> its records(); * a pandas DataFrame (duck-typed via columns/itertuples; pandas is never imported) ->

one record per row across its columns (scalar for a single column, tuple otherwise);

  • a Spark RDD -> the first rdd_cap rows (profiling works on a bounded sample);

  • anything else (a list / sequence) is returned unchanged.

Parameters:

rdd_cap (int)

get_estimator(data, pseudo_count=1.0, emp_suff_stat=True, use_bstats=False)[source]
Parameters:

  • pseudo_count (float | None)

  • emp_suff_stat (bool)

  • use_bstats (bool)

get_prototype(data, *, seed=None, p=0.1, pseudo_count=1.0, emp_suff_stat=True, use_bstats=False)[source]

Infer the model structure from raw data and return a prototype distribution.

Where get_estimator() returns the estimator (the thing you fit with), this returns a concrete, initialized-but-unfitted distribution whose tree mirrors the detected families – the thing you fit. It is the “I just have data, show me the model” front door: inspect which families were chosen, tweak them if you like, then optimize(data, prototype) (or prototype straight into fit).

proto = get_prototype(records) # see the inferred composite structure model = optimize(records, proto) # fit it (or pass proto to fit(…))

seed makes the (lightly randomized) initialization reproducible; p is the per-observation keep-probability of the vectorized initializer. Remaining kwargs mirror get_estimator().

Parameters: