mixle.data.sources.pandas_source module

Pandas DataFrame connector + adapters – DataFrame columns into the sequence-encoded stats API.

The adapter never imports pandas (it duck-types df.columns/.loc/.itertuples); the pandas extra is only needed to construct the DataFrame you pass in. read_dataframe wraps a DataFrame as a DataSource so it funnels into the same encoder contract as every other source.

Missing cells are canonicalized by column KIND, not by pandas dtype backend. A gap in a numeric column becomes NaN; a gap in any other column (string, categorical, boolean, datetime, object) becomes None. That holds however pandas spelled the gap – NaN, pd.NA (nullable extension dtypes) or pd.NaT (datetime/timedelta) – so df and df.convert_dtypes() put the SAME missing marker in every cell, the OptionalDistribution fitted from either frame carries the same missing_value, and each model scores the other frame. See dataframe_records().

What this does NOT promise is that the two frames’ present values are spelled alike: a float64 column of whole numbers carries 1.0 where its convert_dtypes() Int64 twin carries 1, and auto-inference reads those as different data (a continuous family versus a count family). That is a property of the dtype the caller chose, not of the missing marker, it predates this canonicalization, and both models still accept both frames’ records.

normalize_pandas_missing(value, marker=None)[source]

Return value with pandas’ own missing sentinels replaced by marker.

pd.NA and pd.NaT are pandas’ missing markers. mixle has two spellings of missing – None, and NaN for numeric data – and this value-level entry point cannot see which column a value came from, so it defaults to None. Callers that DO know the column go through dataframe_records() or column_records() instead, which canonicalize every gap spelling onto the one marker that column’s kind determines (NaN for numbers, None otherwise) rather than rewriting pandas’ sentinels alone. Anything else – including NaN itself, which the profiler already reads as missing – is returned unchanged, and an unchanged container is returned as the same object rather than a copy. Scalars, tuples, lists, and mappings are handled, so record rows of any adapter shape can be normalized with one call.

Parameters:
Return type:

Any

flat_gap_marker(values)[source]

Return the marker a bare flat sequence of scalars should canonicalize its gaps to.

normalize_pandas_missing() cannot see past one value at a time, so it always defaults to None – correct for a genuinely mixed/non-numeric sequence, but wrong for a numeric one: a plain Python list built from list(a_nullable_series) (optimize() and the encode side both fall back to exactly that per-value normalization once the data is no longer a Series or DataFrame, so the column’s own dtype has already been lost) then fits missing_value=None while the identical data still stored as a Series fits missing_value=nan via column_records() – and unlike the Series-vs-Series case (T2-02), a model built either way ends up unable to score/re-encode the very pd.NA values it was fit from, because pd.NA is only sentinel-equivalent to a NaN missing_value, never to None (T1-02).

This makes the same numeric-vs-not judgment _column_gap_plan() makes from a column’s dtype.kind, but from the sequence’s own present values instead, since there is no dtype left to consult. A sequence whose present values are all numbers gets NaN; anything else (non-numeric present values, or nothing present to judge) gets None, same as today.

Parameters:

values (Any)

Return type:

Any

column_records(column)[source]

Canonical observation records for ONE pandas column or Series.

The single-column half of dataframe_records(): the column’s values with every gap spelling (NaN, None, pd.NA, pd.NaT) collapsed onto the one marker that column’s kind determines – NaN for a column of numbers, None for everything else. A column needing no rewriting is returned as tolist() gives it, with no extra copy.

KNOWN GAP, stated rather than implied. A bare Series reaches the fit verbs on its own (optimize(df["spend"])) and does NOT yet route through here: that path still rewrites only pandas’ own sentinels, so the same column fits missing_value=nan as float64 and missing_value=None as Float64 – campaign four’s T2-02 wearing a different hat, at the Series container instead of the frame. Closing it means changing the profiler’s and the encoder’s Series branches together (mixle.utils.automatic.profiling.normalize_input, mixle.inference.estimation._data_records_for_encoding and mixle.lifecycle._tabular_records), because a model whose sentinel came from one of them and records that came from the other reject each other. Until then, optimize(column_records(s)) is the one-call way to fit a Series on the same footing as the identical column inside a frame.

Parameters:

column (Any)

Return type:

list[Any]

dataframe_records(df, fields=None, as_dict=False, *, _dict_keys='logical')[source]

Convert DataFrame columns into observation records for seq_encode.

A single selected field becomes scalar observations. Multiple selected fields become tuple observations in the requested field order, matching the data shape expected by composite distributions. When as_dict=True, each row is returned as a mapping keyed by the selected source field names.

Missing cells are canonicalized by what the column holds, not by its dtype backend: a gap in a column of numbers (any int/uint/float dtype – numpy, nullable or pyarrow – and an object column whose present values are all numbers) becomes NaN; a gap in any other column becomes None. So NaN, None, pd.NA and pd.NaT all collapse onto the one marker that column’s numpy-backed spelling would have produced; df and df.convert_dtypes() mark every gap the same way, and the OptionalDistribution fitted from either frame carries the same missing_value and scores both (campaign four, T2-02). Present values are NOT respelled – an Int64 column still yields int where its float64 twin yields float – so the two frames can still select different families.

That covers both directions of the same defect: the numeric one was loud (a ContractError on the whole batch), the string one silent (pandas 3 spells a plain str column’s gap NaN and its nullable twin’s pd.NA, and the mismatched sentinel scored -inf per row with no error). Gaps nested INSIDE a cell – a tuple/list/mapping observation – are left as the caller wrote them apart from pandas’ own sentinels; they are the wrapped field’s data, not this column’s gap.

Parameters:
Return type:

list[Any]

seq_encode_dataframe(df, fields=None, encoder=None, estimator=None, model=None, num_chunks=1, chunk_size=None)[source]

Sequence-encode selected DataFrame columns with the ordinary stats API.

The frame-shaped counterpart of mixle.stats.seq_encode(), which consumes RECORDS and refuses a frame outright (“expected a sequence of 2-tuples, got DataFrame”). optimize/ fit accept a DataFrame, so the two entry points disagree about what a dataset is and the conversion is the caller’s job: either encode the frame here, or convert it once with dataframe_records() and hand the records to seq_encode yourself. Gaps are canonicalized exactly as dataframe_records() canonicalizes them.

Parameters:

  • df (Any)

  • fields (str | Sequence[Any] | None)

  • encoder (DataSequenceEncoder | None)

  • estimator (ParameterEstimator | None)

  • model (SequenceEncodableProbabilityDistribution | None)

  • num_chunks (int)

  • chunk_size (int | None)

read_dataframe(df, fields=None, *, as_dict=False, structure=EXCHANGEABLE, schema=None)[source]

Wrap a pandas DataFrame’s selected columns as a DataSource (scalar/tuple/dict records).

Parameters:

  • df (Any)

  • fields (str | Sequence[Any] | None)

  • as_dict (bool)

  • structure (SampleStructure)

  • schema (Schema | None)

Return type:

MaterializedSource