Implementation of the sparse attention pattern proposed by the Deepseek team in their "Native Sparse Attention" paper

SpargeAttention: A training-free sparse attention that can accelerate any model inference.

Neighborhood Attention Extension. Bringing attention to a neighborhood near you!

[ICML2025] Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity

Radial Attention Official Implementation

[ICML 2025 Spotlight] ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference

Efficient triton implementation of Native Sparse Attention.

[CoLM'25] The official implementation of the paper <MoA: Mixture of Sparse Attention for Automatic Large Language Model Compression>

Code for paper: [ICLR2025 Oral] FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference

[TIP-2025] Official Pytorch implementation of "Structural Similarity-Inspired Unfolding for Lightweight Image Super-Resolution"

Demo code for CVPR2023 paper "Sparsifiner: Learning Sparse Instance-Dependent Attention for Efficient Vision Transformers"

Dynamic Attention Mask (DAM) generate adaptive sparse attention masks per layer and head for Transformer models, enabling long-context inference with lower compute and memory overhead without fine-tuning.

Building Native Sparse Attention

Text Summarization Modeling with three different Attention Types

Classification binaire avec architecture Sparse Attention pour données tabulaires. Optimisation automatique des hyperparamètres via Optuna. Testé sur datasets de churn télécommunications et bancaire.

Code for ACL 2025 paper: "Structural Deep Encoding for Table Question Answering"

Integrating QC techniques into Sparse Attention for Transformers