A survey of Mixture-of-Experts (MoE) autoregressive models
reveals foundational designs, recent advances, and optimization strategies for scaling and efficiently deploying large language models. Below is an up-to-date overview synthesizing recent surveys, notably from ACM and IEEE, with extensive bibliographic coverage[1][2].
MoE models use multiple independent or semi-independent “expert” networks and a gating (routing) function that activates only a subset of experts for each token during inference[1][2]. Most commonly, MoE layers replace the feed-forward networks (FFNs) within Transformer blocks, but attention and specialized expert layers are also explored[1][2]. Key design elements are: - Gating/Routing functions: Linear softmax/top-K gating, cosine similarity-based, probabilistic, or nonlinear routers. Functions balance specialization, load, and computational efficiency[1][2]. - Expert specialization: Each expert can learn domain-specific patterns; modularity is a theme in multi-domain and multi-modal models[1][2]. - Conditional computation: Sparse activation means only a fraction of total parameters are used per input, saving computation while increasing total model capacity[2].
Surveys highlight several algorithmic approaches and applications: - Continual/Multi-task/meta-learning: MoE effectively isolates subtask-specific knowledge and mitigates catastrophic forgetting[1]. - Multi-modal/multi-lingual: Modal-specific and domain-specific routing increase relevance in tasks like translation, vision-language fusion, and ASR[1][2]. - Reinforcement Learning and Federated Learning: MoE’s modularity boosts adaptation and reduces communication cost in distributed settings[1].
Recent MoE-based language models include:
| Model | Parameters | Experts | Notable Features | Year |
|---|---|---|---|---|
| Switch Transformer | 1-384B | 64 | Top-1 routing, efficient | 2021 |
| GLaM | 1B-1T | 64 | Multilingual, scalable | 2022 |
| Mixtral-8x7B/22B | 47/141B | 8 | Segment routing, modular | 2023 |
| DeepSeek-V3 | 671B | 256 | Trillion-scale, open | 2024 |
| DBRX | 132B | 16 | Balanced expert usage | 2024 |
| OpenMoE | 34B | 16 | Decoder-only MoE | 2023 |
MoE models are now deployed by research labs and commercial platforms for tasks ranging from general text generation to code, QA, and multilingual understanding[2]. Empirically, they offer superior scaling efficiency, strong out-of-domain generalization, and rapid fine-tuning[1][2].
Recent MoE surveys detail optimization across three levels[2]: - Model-level: Expert pruning, quantization (down to 1-4 bits for memory savings), knowledge distillation, and hierarchical clustering to merge or compress experts[2]. Dynamic routing (Top-K, Top-P, probabilistic) enables adaptive expert selection. - System-level: Distributed routing, load balancing, expert parallelism, dynamic scheduling, offloading to memory or edge devices[2]. - Hardware-level: Custom GPU kernels, memory bandwidth optimization, and hardware-aware sparse computation to accelerate MoE inference[2].
Current research focuses on: - Load balancing and expert collapse prevention - Routing interpretability and modularity - Extreme-scale parallelism and cross-device deployment - Lightweight MoE inference for edge and mobile environments - Combining segment-, token-, and context-level routing[1][2]
MoE autoregressive models stand at the forefront of parameter-efficient scaling for large language models, with rapid advances in architecture, optimization, and deployment[1][2]. Contemporary surveys offer structured frameworks and exhaustive reference lists for latest MoE innovations, making them key resources for practitioners and theorists alike.
For comprehensive bibliographies and deeper technical discussions, see recent surveys published in ACM, IEEE, and leading preprints on arXiv[1][2].
Sources [1] Automated Category and Trend Analysis of Scientific Articles on Ophthalmology Using Large Language Models: Development and Usability Study https://formative.jmir.org/2024/1/e52462 [2] Using Large Language Models to Automate Category and Trend Analysis of Scientific Articles: An Application in Ophthalmology https://arxiv.org/abs/2308.16688
=======
https://www.perplexity.ai/search/5ebb70a2-92fe-4840-9821-ef8d75f78572#1