Transfer and Inspiration of Efficient Attention Mechanisms from Long Context Language Models to Speech Recognition: A Systematic Review

Abstract

With the rapid development of large language models (LLMs), efficient attention mechanisms have become a key solution to addressing computational, memory, and bandwidth bottlenecks in Transformer-based architectures. Although efficient attention mechanisms have been widely applied in text modeling, their use in automatic speech recognition (ASR) has not been extensively studied. This study provides a systematic review of efficient attention mechanisms in LLMs and proposes a task- oriented classification framework covering training and inference acceleration, memory optimization, and long-context support. Based on this classification, we analyze the similarities between LLMs and ASR models and explore the feasibility of cross- modal transfer of efficient attention mechanisms. Additionally, we identify key challenges in transferring efficient attention mechanisms from LLMs to ASR models. By transferring efficient attention mechanisms from LLMs to ASR models, this review provides a conceptual foundation and practical insights for developing effective attention mechanisms that support low- latency, scalable, and accurate speech recognition.