mixle.models.partially_observable_markov_decision_process moduleΒΆ

Partially observable Markov decision process helpers.

class PartiallyObservableMarkovDecisionProcessFilterResult(beliefs, log_likelihood, predictive_observation_probs)[source]

Bases: object

Belief trajectories, log likelihood, and predictive observation terms.

Parameters:
beliefs: ndarray
log_likelihood: float
predictive_observation_probs: ndarray
class PartiallyObservableMarkovDecisionProcessFitResult(model, history, validation_history=None)[source]

Bases: FitResult[PartiallyObservableMarkovDecisionProcessModel]

Baum-Welch style fit result for known-action PartiallyObservableMarkovDecisionProcess sequences.

Parameters:
class PartiallyObservableMarkovDecisionProcessModel(transition, observation, initial_belief=None, rewards=None, name=None)[source]

Bases: object

Finite-state PartiallyObservableMarkovDecisionProcess with action-conditioned transitions and observations.

transition[a, i, j] is P(S_t=j | S_{t-1}=i, A_t=a). observation[a, j, o] is P(O_t=o | S_t=j, A_t=a).

Parameters:

  • transition (Any)

  • observation (Any)

  • initial_belief (Any | None)

  • rewards (Any | None)

  • name (str | None)

belief_update(belief, action, observation)[source]

Update a belief after taking action and seeing observation.

Parameters:

  • belief (Any)

  • action (int)

  • observation (int)

Return type:

tuple[ndarray, float]

filter(actions, observations, initial_belief=None)[source]

Run the forward filter and return posterior beliefs and log likelihood.

Parameters:
Return type:

PartiallyObservableMarkovDecisionProcessFilterResult

sequence_log_likelihood(actions, observations, initial_belief=None)[source]

Return log P(observations | actions, model).

Parameters:
Return type:

float

forward_backward(actions, observations, initial_belief=None)[source]

Return state marginals, transition marginals, and sequence log likelihood.

Parameters:
Return type:

tuple[ndarray, ndarray, float]

predict_observation(belief, action)[source]

Return P(O_t | belief, action) before observing O_t.

Parameters:
Return type:

ndarray

expected_reward(belief, action)[source]

Return E[R | belief, action] when rewards were supplied.

Parameters:
Return type:

float

sample(actions, seed=None, initial_belief=None)[source]

Sample latent states and observations for a fixed action sequence.

Parameters:
Return type:

tuple[ndarray, ndarray]

baum_welch_pomdp(sequences, num_states, num_actions, num_observations, initial_model=None, max_its=50, tol=1.0e-8, pseudo_count=1.0e-3, seed=None)[source]

Fit a known-action finite PartiallyObservableMarkovDecisionProcess by Baum-Welch/EM.

Parameters:
Return type:

PartiallyObservableMarkovDecisionProcessFitResult