Reinforcement learning · course project · 2025

Three-Player Hexagonal Othello

A three-player Othello variant on a hexagonal board, plus a tabular Q-learning agent trained over 100K episodes to play it. Team of four; I built the RL agent with one teammate.

Source

Win rate: 55%
Training episodes: 100K
Avg reward: −1100−500
Corners held on avg: 3 of 6

The problem

Othello theory assumes two players and a square board. Add a third player and a hex grid and most of it stops applying: a single move can flip tokens belonging to both opponents at once, so the position is no longer zero-sum and the weaker two players have an incentive to gang up on the leader. We wanted to know whether an agent could learn to play this well without being told any of that.

Approach

01Designing the variant

A hexagon of edge length 7 carved out of a 13×19 matrix, with off-board cells masked. The opening position is a balanced 3×3 subgrid with one token per player in every row and column. Without it, turn order alone decides the game and two players can trivially collapse on the third.

02Making the state space tractable

Tabular Q-learning over a board this size only works if states collapse aggressively. Positions are encoded from the acting player's perspective with both opponents mapped to the same value, roughly halving the space, then keyed by SHA-256 of the raw bytes into a dict of SciPy sparse rows over 247 possible actions. The learning rate decays per state-action visit count rather than globally, so rarely-seen positions keep learning while common ones settle.

03Reward shaping

Terminal-weighted: token margin against the strongest opponent, plus 100 for a win and minus 50 for a loss. Trained for 100K episodes against fixed random and greedy opponents with epsilon annealed from 1.0 to about 0.01.

Results

The agent reached a 55% win rate over 40 held-out games in a three-player game where the baseline is 33.3%, with average episode reward climbing from below −1100 to around −500 over training. The more interesting result is qualitative: it converged on holding roughly 3 of the 6 corners, rediscovering the corner-control heuristic that dominates standard Othello strategy without any of it being encoded in the reward.

What it doesn't do

40 evaluation games is a small sample, and the confidence interval on 55% is wide enough that the true edge over baseline is uncertain, even though the direction is clear.
This is tabular Q-learning, not deep RL. It generalizes only to positions it has actually seen, and the Q-table grew large enough to be awkward to distribute. A function approximator was the obvious next step and we didn't get to it.
Trained against fixed random and greedy opponents rather than self-play, so it learned to beat those two policies specifically, not to play the variant well in general.

Next project

Warrant