Maximizing AI Efficiency: Understanding AI Quantization and Its Challenges

In the quest to enhance AI efficiency, one effective technique that stands out is AI quantization. However, while quantization offers numerous advantages, it also comes with significant challenges that may soon limit its effectiveness in the rapidly evolving AI landscape.

AI quantization

AI quantization is the process of reducing the number of bits required to represent AI model data. To illustrate, imagine telling time. Instead of saying “12:00:00,” you might simply say “noon.” Both communicate the same idea, but one is far more concise. The level of detail necessary always depends on the context in which it’s used.

Within AI models, several components benefit from quantization, especially model parameters, which are the internal variables these models leverage to predict outcomes and make decisions. As AI models perform millions of computations, using quantized versions with fewer bits for their parameters can significantly ease the mathematical burden and lower computational expenses. It’s important to note that this process differs from “distilling,” focused on selectively pruning model parameters.

Nevertheless, recent developments indicate that AI quantization might entail more drawbacks than previously recognized.

The Shrinking Model Conundrum: Challenges of AI Quantization

A comprehensive study by researchers from Harvard, Stanford, MIT, Databricks, and Carnegie Mellon emphasizes that quantized models frequently underperform, especially if the original unquantized model was trained for long durations with expansive datasets. In essence, there are occasions when training a smaller model might yield better results than attempting to compress a larger one.

This presents substantial hurdles for AI companies that typically train vast models, known for improving output quality, only to try and quantize them to slash serving costs.

Some disturbing effects are becoming evident. Developers and researchers recently noted that applying quantization to Meta’s Llama 3 model produced “more detrimental” effects compared to other models, likely due to its distinctive training methods.

According to Tanishq Kumar, a mathematics student at Harvard and the principal author of the study, “inference remains the number one cost for everyone in AI, and our findings show that one method to reduce costs may not be viable forever.”

Interestingly, running AI models—referred to as inference—often incurs more costs than initial model training. For instance, Google invested around $191 million to train one of its fundamental models. However, if this model generates brief responses for half of Google Search queries, their annual costs could soar to approximately $6 billion.

Prominent AI laboratories have adopted the approach of training models on massive datasets based on the assumption that “scaling up”—enhancing data and computing resources during training—will foster the development of more capable AI systems.

For example, Meta employed approximately 15 trillion tokens for training Llama 3, a staggering increase from the 2 trillion tokens allocated for its predecessor, Llama 2.

Yet, evidence suggests that this scaling up strategy may result in diminishing returns. Recent accounts reveal that Anthropic and Google trained exceptionally large models but did not meet internal performance milestones. Despite this, the industry seems hesitant to drift away from established scaling practices.

Precision in AI: Finding the Balance

As leading AI labs continue to resist the appeal of smaller datasets, is there a potential approach to fortify models against quality degradation? Perhaps! Kumar and his research team discovered that training models using “low precision” could improve their resilience. Let’s break this down.

The term “precision” denotes the amount of digits a numerical data type can accurately depict. Varied data types allow for a range of data values and operations. For instance, FP8 data utilizes merely 8 bits to symbolize a floating-point number.

Most contemporary models are trained with “half-precision” formats using 16 bits, transitioning to 8-bit precision through “post-training quantization.” This process involves converting certain model components into lower precision formats, slightly affecting accuracy. It’s akin to performing rounding calculations, balancing efficiency and accuracy.

Quantized models

Hardware manufacturers, especially Nvidia, endorse low precision in quantized models. Their latest Blackwell chip supports a new 4-bit precision format called FP4, which will be particularly advantageous for data centers facing memory and power restrictions.

However, adopting extremely low quantization precision may not always yield desirable results. Kumar points out that when the original model’s parameter count is not vast, utilizing precisions lower than 7 or 8 bits could seriously undermine quality.

While this concept may appear intricate, the essential takeaway is that AI models remain imperfectly understood, and shortcuts that work in other computational areas may not apply here. Just as you wouldn’t say “noon” to describe the starting time of a 100-meter dash, the details in AI operations deserve careful attention. Kumar highlights that “there are limitations in quantization that cannot be easily circumvented.” His research team aims to contribute to a more nuanced comprehension of higher precision defaults in training and inference.

Models

Kumar acknowledges that their study’s scale was relatively limited; nonetheless, they seek to expand their research with additional models. He emphasizes that one critical insight is clear: reducing inference costs is not straightforward, and it comes with ramifications.

To capture this insight, Kumar concludes, “Bit precision matters, and it isn’t without cost. You can’t simply reduce it indefinitely without adversely impacting model performance. Models have finite capacities; therefore, instead of trying to compress exorbitant tokens into a small model. Significant emphasis should go into meticulous data curation and filtering, ensuring that only top-quality data inputs smaller models. I remain hopeful that future architectures emphasizing stable low precision training will prove essential.”