Implementation of the sparse attention pattern proposed by the Deepseek team in their "Native Sparse Attention" paper

SpargeAttention: A training-free sparse attention that can accelerate any model inference.

Fast Multi-dimensional Sparse Attention

Radial Attention Official Implementation

[ICML2025] Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity

Speed Always Wins: A Survey on Efficient Architectures for Large Language Models

[ICML 2025 Spotlight] ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference

Efficient triton implementation of Native Sparse Attention.

[CoLM'25] The official implementation of the paper <MoA: Mixture of Sparse Attention for Automatic Large Language Model Compression>

Code for paper: [ICLR2025 Oral] FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference

[TIP-2025] Official Pytorch implementation of "Structural Similarity-Inspired Unfolding for Lightweight Image Super-Resolution"

Demo code for CVPR2023 paper "Sparsifiner: Learning Sparse Instance-Dependent Attention for Efficient Vision Transformers"

Dynamic Attention Mask (DAM) generate adaptive sparse attention masks per layer and head for Transformer models, enabling long-context inference with lower compute and memory overhead without fine-tuning.

Building Native Sparse Attention

Toy Hydra prototypes: SSM + sparse attention + MoE + memory; synthetic benchmarks. Paper: https://arxiv.org/abs/2508.15099

Classification binaire avec architecture Sparse Attention pour données tabulaires. Optimisation automatique des hyperparamètres via Optuna. Testé sur datasets de churn télécommunications et bancaire.

Code for ACL 2025 paper: "Structural Deep Encoding for Table Question Answering"

Text Summarization Modeling with three different Attention Types

Integrating QC techniques into Sparse Attention for Transformers