We present a new approach to formulating and solving heterogeneous agent mod-
els with aggregate risk. We replace the cross-sectional distribution with low-dimensional
prices as state variables and let agents learn equilibrium price dynamics directly from sim-
ulated paths. To do so, we introduce a structural reinforcement learning (SRL) method which
treats prices via simulation while exploiting agents’ structural knowledge of their own in-
dividual dynamics. Our SRL method yields a general and highly efficient global solution
method for heterogeneous agent models that sidesteps the Master equation and handles
models traditional methods struggle with, like those with nontrivial market-clearing con-
ditions. We illustrate the approach in the Krusell-Smith model, the Huggett model with
aggregate shocks, and a HANK model with a forward-looking Phillips curve, all of which
we solve globally within minutes.