No I/L – Lucas Critique, Info Relevancies, and Data Gathering
Fernández-Villaverde 21 (Jesús Fernández-Villaverde - Professor of Economics at the University of Pennsylvania, “Artificial Intelligence Can’t Solve the Knowledge Problem”, Public Discourse, 29 July 2021, https://www.thepublicdiscourse.com/2021/07/76963/, MG)
Unfortunately, in many areas of public policy, we do not have access to such a wealth of data and, most likely, we never will. Take the example of setting monetary policy, i.e., the regulation of how much money to supply, typically conducted by national central banks, such as the Federal Reserve in the U.S.
Monetary policy is a relatively straightforward topic with fewer moving parts than other kinds of economic policy. Could machine learning ever replace the Federal Market Open Committee (FOMC, the main policy instrument of the Federal Reserve System)? I am skeptical. For one thing, the FOMC usually has a limited amount of data. In the United States, we only have reliable data for output, consumption, and investment after World War II and, even then, only at a quarterly level. If we count them, from 1947:Q1 (the first “good” observation in terms of the accuracy of our measurement) until 2021:Q2 (the last observation as I write this), we have 298 data points. This is far fewer than acceptable for machine learning techniques.
Furthermore, the US economy has radically changed, which limits the relevance of older data. We have moved from an economy dominated by manufacturing into an economy driven by services, and financial innovations have transformed the relationship between financial and real variables. The evolving structure of the economy shifts the relationships between the data points, making it harder for machine learning to find clear patterns. These structural changes mean that econometricians often do not use observations before the early 1980s when they estimate the effects of monetary policy on output. In fact, such estimates change sharply depending on whether we include early observations. Moreover, the economy is bound to continue to change, meaning we will continue to have to deal with newer and newer data.
Employing individual data (such as consumption data of households or financial transactions) can help us get more observations, but we will still encounter similar problems. Ponder, for example: how informative are the consumption patterns of married couples in the 1990s, in their early 40s, with several kids at home, about the consumption patterns of single individuals in the 2020s, also in their early 40s, without kids? Moreover, there are severe limitations on what individual data can teach us in the absence of detailed explanations about the nature of relationships between different data points.
This additional problem with using microdata is an instance of the Lucas critique, named after Robert Lucas, one of the most influential economists of the last century. The essence of the critique is that it is difficult, if not impossible, to distinguish in data how much behavior is attributable to unobserved characteristics or to the impact of particular public policies.
Machine learning faces the same problem that economists have faced for a century: distinguishing causation and correlation. Moreover, the answer that machine learning provides is valid only under a constant set of circumstances: changes in policy may affect different individuals differently, which gives misleading results about the impact of a policy.