News Summary

Epoch AI, a nonprofit organization focused on developing mathematical benchmarks for AI, has faced criticism for not disclosing funding from OpenAI until recently. The organization, which is primarily funded by Open Philanthropy, announced on December 20 that OpenAI had supported the creation of FrontierMath, a benchmark used to test AI’s mathematical skills. This late disclosure has raised concerns about the transparency and objectivity of FrontierMath as a benchmark. Tamay Besiroglu, associate director and co-founder of Epoch AI, acknowledged the lack of transparency as a mistake and expressed the need for more open communication with contributors. Despite the controversy, he assured that OpenAI has a verbal agreement not to use FrontierMath’s problem set for training its AI. Additionally, Epoch AI maintains a separate holdout set for independent verification. However, Epoch AI’s lead mathematician, Ellot Glazer, noted that the organization has yet to independently verify OpenAI’s results with FrontierMath. This situation highlights the challenges in developing empirical AI benchmarks and managing potential conflicts of interest.