Researchers have developed a new way to compress the memory used by AI models to increase their accuracy in complex tasks or help save significant amounts of energy.\u00a0<\/p>\n\n\n\n
Experts from University of Edinburgh and NVIDIA found that large language models (LLMs) using memory eight times smaller than an uncompressed LLM scored better on maths, science and coding tests while spending the same amount of time reasoning.\u00a0The method can be used in an alternative way to help LLMs respond to more user queries simultaneously, reducing the amount of power needed per task.<\/p>\n\n\n\n
As well as energy savings, experts say the improvements could benefit AI systems that are used to solve complicated tasks or in devices that have slow or limited memory, such as smart home devices and wearable technology. <\/p>\n\n\n\n
By \u201cthinking\u201d about more complex hypotheses or exploring more hypotheses concurrently, AI models improve their problem-solving abilities. In practice, this is achieved by generating more reasoning threads \u2013 a step-by-step logical process used to solve problems \u2013 in text form. <\/p>\n\n\n\n
The model\u2019s memory \u2013 called the KV cache \u2013 which stores the portions of the threads generated, can act as a bottleneck, as its size slows down the generation of reasoning thread outputs during inference \u2013 the process by which AI models respond to an input prompt, such as answering a user query. <\/p>\n\n\n\n
The more threads there are, and the longer they are, the more memory is required. The larger the memory size used, the longer the LLM takes to retrieve the KV cache from the part of the AI device where it is stored. <\/p>\n\n\n\n
To overcome this, the team developed a method to compress the models\u2019 memory \u2013 called Dynamic Memory Sparsification (DMS). Instead of keeping every token \u2013 the units of data that an AI model processes \u2013 DMS decides which ones are important enough to keep and which ones can be deleted. <\/p>\n\n\n\n
There is a slight delay between the time when the decisions to delete tokens using sparsification are made and when they are removed. This gives the model a chance to pass on any valuable information from the evicted tokens to preserved ones. <\/p>\n\n\n\n
In managing which tokens to keep and which to discard, DMS lets the AI model “think\u201d in more depth or explore more possible solutions without needing extra computer power.<\/p>\n\n\n\n
The researchers tested DMS on different versions of the AI models Llama and Qwen and compared their performance to models without compression. <\/p>\n\n\n\n
The models\u2019 performance was assessed using standardised tests. It was found even with memories compressed to one eighth their original size, LLMs fully retain their original accuracy in difficult tasks while accelerating reasoning compared with non-compressed models. <\/p>\n\n\n\n
In the standardised maths test AIME 24, which served as the qualifier for the United States Mathematical Olympiad, the compressed models performed twelve points better on average using the same number of KV cache reads to produce an answer.<\/p>\n\n\n\n
For GPQA Diamond \u2013 a series of complex questions in biology, chemistry and physics authored by PhD-level experts \u2013 the models performed over eight points better.<\/p>\n\n\n\n
The models were also tested with LiveCode Bench, which measures how well AI models can write code. The compressed models scored on average ten points better than non-compressed models. <\/p>\n\n\n\n