mixle.task.rl module

Reinforcement learning: tabular Q-learning over a discrete MDP with a KNOWN reward.

This is the classical value-based counterpart to sequence-level optimization routines such as outcome_decomposer and probe_policy: those optimize a whole action sequence’s terminal, oracle-verified score by propose/filter/refit, with no per-step value estimate. This module is the other end of the design space – a per-step Bellman backup, Q(s, a) <- Q(s, a) + alpha * [r + gamma * max_a' Q(s', a') - Q(s, a)] – for problems that ARE naturally modeled as a finite MDP with a known step reward. It composes with mixle.task.irl (the complementary direction: reward FROM demonstrations, not policy from a known reward) via the shared GridWorld environment.

world = GridWorld(size=5, goal=(4, 4)) result = tabular_q_learning(world, episodes=500, seed=0) policy = result.greedy_policy(world) # {state: best action}, the recovered optimal policy

class GridWorld(size, goal, obstacles=<factory>, step_cost=-1.0, goal_reward=10.0, max_steps=100)[source]

Bases: object

A deterministic size x size grid MDP: a goal cell worth goal_reward, a per-step cost of step_cost, and optional impassable obstacles (moving into a wall or obstacle leaves the agent in place, still paying the step cost). The optimal policy is the shortest obstacle-free path to the goal – computable independently via optimal_path_length() (BFS), which is what makes this a closed-form-known-optimum test environment.

Parameters:
property n_states: int

Return the number of states in the square grid.

state_index(state)[source]

Map a (row, column) state to its row-major integer index.

Parameters:

state (tuple[int, int])

Return type:

int

index_state(index)[source]

Map a row-major integer state index back to (row, column).

Parameters:

index (int)

Return type:

tuple[int, int]

states()[source]

Return every grid state in row-major order.

Return type:

list[tuple[int, int]]

transition(state, action)[source]

The deterministic next state for action at state (walls/obstacles are a no-op).

Parameters:
Return type:

tuple[int, int]

reset(start=(0, 0))[source]

Reset the environment to start and return the initial state.

Parameters:

start (tuple[int, int])

Return type:

tuple[int, int]

step(action)[source]

Apply one action and return (next_state, reward, done).

Parameters:

action (str)

Return type:

tuple[tuple[int, int], float, bool]

optimal_path_length(start=(0, 0))[source]

BFS shortest obstacle-free path length from start to goal – ground truth for tests, computed independently of any learning algorithm in this module.

Parameters:

start (tuple[int, int])

Return type:

int

class QLearningResult(q_table, rewards_per_episode)[source]

Bases: object

The fitted Q-table plus the per-episode return trace (the learning curve).

Parameters:
greedy_action_index(state_index)[source]

Return the index of the highest-valued action for state_index.

Parameters:

state_index (int)

Return type:

int

greedy_policy(env)[source]

The recovered deterministic policy: the argmax action at every non-goal state.

Parameters:

env (GridWorld)

Return type:

dict[tuple[int, int], str]

tabular_q_learning(env, *, episodes=500, alpha=0.3, gamma=0.95, epsilon=0.2, seed=None)[source]

Epsilon-greedy tabular Q-learning: episodes full rollouts from env.reset(), each step updating Q(s, a) toward the observed one-step Bellman target.

Parameters:
Return type:

QLearningResult

rollout(env, policy, *, start=(0, 0))[source]

Roll out a deterministic state -> action policy from start; the (state, action) trace (stops at the goal or env.max_steps, whichever first).

Parameters:
Return type:

list[tuple[tuple[int, int], str]]

random_policy(env, rng)[source]

Baseline: an independent uniform-random action at every non-goal state.

Parameters:
Return type:

dict[tuple[int, int], str]