K.E.Y.N.E.S.Knowledge Engine Yielding Nonstop Economic Scenarios

Methods

How the models are tested, what the scores mean, how this work relates to the models the Federal Reserve publishes, and what to be careful about.

Real-time testing

Economic data are revised, often a lot. The first estimate of a month's payroll gain is routinely revised by tens of thousands of jobs, and GDP growth by a point or more. A model tested on today's revised data would look better than it could ever have been in practice, because it would be using information nobody had at the time.

So every result on this site comes from a real-time backtest. We step through history one forecast date at a time. At each date we take the data exactly as they had been published by then, fit the model on those data alone, and record its forecast. Only later is the forecast compared with what actually happened.

data published by the forecast date (as first released) forecast date fit the model, record the forecast not yet known truth published score the forecast Repeat for every week (GDP nowcast) or month (industry models) from 2000 or 2006 to 2026.
  • GDP nowcast: a nowcast every Friday for 82 quarters, 2006Q2 to 2026Q3, with the model re-estimated on the first Wednesday of each quarter. Scored at 56, 28 and 1 day before BEA's first estimate, as in the New York Fed's own evaluation.
  • Industry models: a forecast every month from January 2000 to the latest FRED-MD vintage, 0 to 12 months ahead. Model choices (the number of factors and lags) were made on 2000–2012 only and then kept fixed.
  • Data vintages: the FRED-MD snapshot of each month, and ALFRED's archive of past releases for other series. Where no archive exists for a date, the GDP nowcast leaves the value out and the industry models use today's data cut off at that date, labeled "pseudo-real-time".

Accuracy scores

Mean squared error (MSE) averages the squared forecast misses. Squaring makes large misses count more, and keeps positive and negative misses from cancelling. For GDP, the misses are in percentage points of annualized growth, so an MSE of 1.4 means a typical miss of about \(\sqrt{1.4} \approx 1.2\) points.

Relative error divides a model's root mean squared error by the benchmark's. 0.90 means errors 10% smaller than the benchmark; above 1 means worse. The site shows this as "10% better".

The benchmark is an autoregression (AR): a forecast of a series from its own recent past alone. It is simple but hard to beat, especially in calm periods.

p-values come from Diebold–Mariano tests of whether two forecasts are equally accurate. A p-value below 0.05 means a difference that large would rarely arise by chance. With only a few recessions in the test periods, most differences between good models are not statistically significant.

What counts as the truth: for GDP, BEA's first estimate, its third estimate, or the latest revised figure; for monthly indicators, the first release or the latest vintage. Results can differ by truth: a model that predicts first releases well need not predict the revised figures as well.

For economistsFormulas $$\text{MSE} = \frac{1}{N}\sum_{q=1}^{N} \big(\hat y_{q \mid t} - y_q\big)^2, \qquad \text{relative RMSE} = \frac{\sqrt{\text{MSE}_{\text{model}}}}{\sqrt{\text{MSE}_{\text{AR}}}}$$

Diebold–Mariano statistics use the squared-error loss differential with a Newey–West variance and the Harvey–Leybourne–Newbold small-sample correction. Industry-model scores are on cumulative changes over the horizon. Error bands are scored by coverage: the share of outcomes inside the model's 68% band.

Both models belong to the family of factor-model nowcasts that several Federal Reserve Banks publish. They are a different kind of tool from the Board's structural models, which exist to analyze policy rather than to track data.

ModelInstitutionMethodHow it compares with this work
Staff Nowcast 2.0 (2023)New York Fed Bayesian dynamic factor model with a trend, stochastic volatility and outlier adjustment; 35 series from 1985 Our GDP nowcast replicates it on free data, with regional Fed surveys in place of the ISM indexes and our own priors.
Legacy Staff Nowcast (2016–2021)New York Fed Dynamic factor model with global, soft, real and labor factors, estimated by EM (Bok et al. 2018) The first stage of our GDP work, and the same estimation method as the industry models.
GDPNowAtlanta Fed Bridge equations for 13 GDP components, with a factor model filling in unpublished months Builds GDP up from its parts instead of reading hidden factors. Not yet compared with ours.
CFNAIChicago Fed First principal component of 85 monthly activity indicators An index of current activity, essentially the activity factor of our industry models on its own.
FRB/US, EDO and SIGMAFederal Reserve Board Structural models of how households, firms and markets behave They can answer "what if the Fed raises rates?"; our statistical models cannot.

Methods we build on: FRED-MD (McCracken and Ng 2016) with its transformation codes and outlier rule; factor models estimated by EM with the Kalman smoother on data with gaps (Bańbura and Modugno 2014); the diffusion-index forecasts of Stock and Watson (2002); mixed-frequency aggregation (Mariano and Murasawa 2003); stochastic volatility sampled as in Kim, Shephard and Chib (1998) with the mixture of Omori et al. (2007); and the Staff Nowcast 2.0 paper itself (Almuzara, Baker, O'Keeffe and Sbordone 2023).

Caveats

  • Few recessions. Most gains are consistent in sign but not statistically significant, because the test periods contain few downturns.
  • The GDP nowcast's lead over our older-model replica is not significant, and its lead over the New York Fed's published nowcasts holds only against BEA's first estimate.
  • Error bands of the GDP nowcast are too wide before the pandemic (the 68% band covers 80% of outcomes) and too narrow after it.
  • Priors are our own, since the Staff Nowcast 2.0 paper does not publish its values. The trend's flexibility in particular is set by our prior, not by the data.
  • Post-pandemic results rest on only 10 quarters.
  • Industry models: gains over the AR on national indicators are 2–5% and mostly significant only at the 10% level. Industry price forecasts need the plain growth-rate transformation to beat the AR.
  • These are research models, not official forecasts of any institution.

Glossary

NowcastAn estimate of the present or the very recent past, for a figure not yet published.
FactorA hidden variable that many data series move with, estimated from the data.
LoadingHow strongly a series moves with a factor.
First estimateBEA's first estimate of a quarter's GDP, about a month after the quarter ends (BEA calls it the advance estimate).
VintageThe version of a data set as it was published on a given date.
NewsThe part of a data release the model did not expect: actual minus expected value. Each release moves the nowcast by its news times a weight.
HorizonHow far ahead a forecast is: months ahead, or weeks before the GDP release.
AR benchmarkA forecast of a series from its own recent past only.
Annualized growthA quarter's growth rate expressed as if it continued for a whole year (roughly four times the quarterly rate).
Stochastic volatilityShock sizes that change over time instead of being fixed.
Real-time backtestReplaying history using only the data available at each date.