News Summary

A recent study highlights the limitations of quantization, a technique used to make AI models more efficient by reducing the number of bits needed to represent information. This method, while beneficial in reducing computational demands, is showing signs of diminishing returns, particularly for models trained on extensive datasets over long periods. The study, involving researchers from Harvard, Stanford, MIT, Databricks, and Carnegie Mellon, found that quantized models tend to perform worse than their unquantized counterparts if the original models were extensively trained. This poses a challenge for AI companies that rely on large models, as quantizing them to cut costs may not always be effective. The cost of AI model inferencing is noted to often surpass that of model training, exemplified by Google’s significant expenditure on inference. As the industry heavily invests in scaling models, the study suggests that training models in low precision could enhance robustness, though extremely low quantization precision might degrade quality unless the models are exceptionally large. The research underscores the necessity for meticulous data curation and new architectures to stabilize low precision training, highlighting that reducing inference costs is not straightforward.