For the past decade, the American Economic Association (AEA) has provided data and code for the hundreds of articles that have appeared in the nine journals produced by the nonprofit, nonpartisan scholarly association. Despite the availability of these materials, provided so others could replicate the authors’ findings, many attempts at replication failed.

As a result, the association in July adopted an updated policy, which was spearheaded by Lars Vilhuber, executive director of Cornell’s Labor Dynamics Institute and the AEA’s inaugural data editor. The new initiative provides for improved guidance on the types of materials required, increased quality control and more review earlier in the publication process.

“The prior policy was to provide the data, but there was little checking beyond making sure some material was provided,” Vilhuber said. “So while we spent enormous amounts of time on editing the journal articles themselves, we spent little to no time on verifying to make sure that the materials the author provided actually work.”

As part of the new review process, a team of 10 to 12 Cornell undergraduate students under Vilhuber’s direction will run the authors’ code in order to verify reproducibility.

“This is a great opportunity for undergraduates to get involved with research that is of the highest quality in economics,” said Vilhuber. “They will come out of this with quite a bit of experience. They assess reproducibility, and in doing so they learn how to do it themselves. 

“I hope to be out of this job in five to 10 years,” he said, “because the goal is that, by then, it will be so trivial that there’s nothing to check.”

Vilhuber will speak at the Policy Simulation Library DC meeting on Wednesday, Sept. 11. Hosted by the American Enterprise Institute’s Open Source Policy Center, the theme for this year’s meeting is “Promoting Transparency and Reproducibility at the American Economic Association.” Register to attend the event or watch online.

Julie Greco is a communications specialist with the ILR School.