New AI-assisted Study on the Gender Pay Gap
The gender pay gap—the widely-cited statistic showing that the median female worker earns less than the median male worker—remains a lightning rod in policy debates.
Many politicians and advocates seize upon simplistic figures (like the oft-repeated claim that women only earn "84 cents on the dollar") to push government regulatory solutions. The implicit claim is that the pay gap is driven by employer bias and discrimination against women, akin to them being paid less than men for the same work. Yet traditional analyses often gloss over crucial details about individual career paths, inadvertently obscuring important drivers of average wage differences between the sexes.
Most research that investigates gender wage gaps beyond the headline statistic uses snapshot data—looking at what men and women earn today, while controlling for broad variables such as education, age, occupation, experience, and demographic factors to try to see how much gap is left “unexplained.” These controls help to get us closer to an apples-to-apples comparison, and some studies find including them lowers the unexplained pay gap to around 5 percent. Yet even that level of aggregation misses nuanced career decisions at the individual level, including previous shifts between occupations, specialization within a job, or significant breaks from the labor force. When such factors are ignored or oversimplified, the unexplained gap attributed to discrimination can still appear artificially large.
In a really interesting new paper (h/t Tyler Cowen via Kevin Lewis), Estimating Wage Disparities Using Foundation Models, economists Keyon Vafa, Susan Athey, and David M. Blei use cutting-edge artificial intelligence techniques to address these limitations. They introduce an AI-driven "foundation model" known as CAREER, trained on the detailed career histories of millions of workers, to better capture the subtle yet influential factors driving gender wage disparities.
CAREER uses transformer technology—a type of AI originally developed for complex language processing tasks—to analyze extensive longitudinal data from the Panel Study of Income Dynamics (PSID). By evaluating peoples’ complete occupational trajectories, rather than just current jobs or broad observable factors, CAREER can tease out hidden drivers of wage differences between men and women that traditional methods usually leave unexplained.
What do Vafa, Athey, and Blei find when the full career histories are considered? Well, the unexplained gender wage gap falls significantly. With their comprehensive AI analysis of career histories, the ratio of women’s to men’s pay rises to 93.4 percent, compared to 88.6 percent via a more traditional analysis. This shows that even among studies that had explained much of the gap using observable characteristics, the detailed and nuanced job histories captured by CAREER can reduce the unexplained portion even further by identifying other patterns that affect pay.
So what sort of factors do most studies omit? In supplementary analysis, the authors cluster career histories to tease out examples.
First, traditional analyses broadly group jobs into categories like "management" or "education," missing critical specialization details. For example, this new analysis shows that managers who previously worked technical roles (e.g., computer scientists, engineers, or engineering technicians) earn more than those whose backgrounds were administrative or clerical. Women less frequently occupy these technical specializations, thus explaining part of the gap. In contrast, managers with previous jobs as bookkeepers or secretaries tend to earn less—and women make up 79 percent of these roles.
Second, the new analysis finds that gender wage differences explained by career history widen with age, highlighting that cumulative career decisions and life choices—such as selecting flexible jobs for family reasons or reducing workload intensity—magnify wage disparities later in life. Younger cohorts (25-34 year olds) generally have smaller pay gaps, suggesting that differences accumulate over time rather than primarily being a consequence of overt discrimination. This isn’t surprising—Nobel Prize winner Claudia Goldin has extensively documented how career paths diverge over time—and yet this dynamic is often missed in snapshot analyses.
Third, broader occupational trajectories matter. For instance, teachers who spent time in administrative or lower-paid support roles earlier in their careers tend to have persistently lower wages, even after transitioning to seemingly similar higher-level positions. Women more often follow these trajectories, making this another vital explanation for wage differences.
Why should such “career history” variables matter, if a man and woman have many of the same demographic characteristics and are ostensibly doing the same job? Well, on the employer side, a business or non-profit may be looking for specific skills for, say, a managerial role. It may be that the broad “codes” used for jobs don’t reflect that someone with a more technical background is doing a slightly different task, or that they just have different skills that are more valuable. Perhaps a school is looking for a math teacher with a strong background in physics—and is willing to pay more to find one. If that pool is more dominated by men, then selecting one will add to the pay gap.
Ultimately, Vafa, Athey, and Blei’s use of AI highlights a deeply Hayekian insight: wage differences reflect highly localized knowledge and individual-specific characteristics that traditional statistical methods fail to capture.
Each person’s career history encapsulates subtle, decentralized information—such as specialized expertise, preferences for flexibility (often related to family or caregiving responsibilities), and unique combinations of skills. These intricacies shape an individual’s reservation wage—the minimum pay someone is willing to accept—and reflect qualities employers recognize, value, and willingly reward. Even this more granular model can’t account for all the factors that determine our earning capabilities.
Sure, there’s no doubt some sex discrimination exists. And, yes, someone concerned about gender equality might follow up by asking whether attitudes towards women shape some downstream factors that affect wages. Yet, the key point here is that by harnessing AI to illuminate these previously hidden dimensions, the authors underscore that wage disparities are affected by nuanced factors related to career history, leaving far less room to attribute them to explicit employer bias against women.