mixle.task.irl module

Maximum-entropy inverse reinforcement learning (Ziebart et al., 2008): recover a REWARD from expert demonstrations, rather than assume one is given.

This is the complementary direction to mixle.task.rl (which learns a POLICY given a KNOWN reward): given expert trajectories – state sequences produced by an expert acting optimally under some UNKNOWN reward – max_ent_irl() recovers linear reward weights over a state feature map such that the maximum-entropy (Boltzmann-rational) policy induced by those weights matches the expert’s empirical feature expectations. That match is the algorithm’s own optimality certificate (feature-expectation matching at convergence), not a proxy metric graded after the fact.

Differs from mixle.task.plan_model, which fits a Markov chain directly over observed action sequences and models what the expert did. This module additionally explains why by recovering the reward the expert’s behavior is consistent with, over the same GridWorld environment shape.

world = GridWorld(size=5, goal=(4, 4)) demos = [rollout_states(world, expert_policy, start=(0, 0)) for _ in range(20)] result = max_ent_irl(world, demos) result.reward_weights.reshape(world.size, world.size) # recovered per-cell reward

state_features(env)[source]

Default feature map: a one-hot indicator per grid cell (n_states x n_states) – a fully expressive tabular basis, so recovering per-feature weights is equivalent to recovering the per-state reward directly.

Parameters:

env (GridWorld)

Return type:

ndarray

rollout_states(env, policy, *, start=(0, 0))[source]

The state-only trace of a deterministic policy from start (the demonstration format max_ent_irl() expects: what the expert visited, not what it was thinking).

Parameters:
Return type:

list[tuple[int, int]]

class MaxEntIRLResult(reward_weights, policy, history)[source]

Bases: object

The recovered reward, its induced Boltzmann-rational policy, and the convergence trace (||expert_feature_expectation - policy_feature_expectation|| per iteration – should decrease toward zero as the algorithm’s own certificate of fit).

Parameters:
reward(features)[source]

Evaluate the learned linear reward on feature rows.

Parameters:

features (ndarray)

Return type:

ndarray

max_ent_irl(env, expert_trajectories, *, start=(0, 0), gamma=0.9, iterations=150, lr=0.5, features=None)[source]

Recover linear reward weights whose maximum-entropy-optimal policy matches the expert’s empirical feature expectations, via gradient ascent on trajectory likelihood: weights += lr * (expert_feature_expectation - policy_feature_expectation). Requires only expert_trajectories (state sequences); never sees the expert’s true reward or the actions that produced them.

Parameters:
Return type:

MaxEntIRLResult