End-to-end training of sparse deep neural networks with little-to-no performance loss.
-
Updated
Jan 26, 2023 - Python
#
sparse-training
Here are 19 public repositories matching this topic...
Always sparse. Never dense. But never say never. A Sparse Training repository for the Adaptive Sparse Connectivity concept and its algorithmic instantiation, i.e. Sparse Evolutionary Training, to boost Deep Learning scalability on various aspects (e.g. memory and computational time efficiency, representation and generalization power).
deep-neural-networks sparsity deep-learning scalability randomization neuroevolution deep-learning-algorithms classification evolutionary-algorithms artificial-neural-networks restricted-boltzmann-machine complex-networks deep-learning-papers multi-layer-perceptron generative-models sparse-neural-networks scalable-deep-learning sparse-training sparse-evolutionary-training adaptive-sparse-connectivity
-
Updated
Jul 21, 2021 - Python
[NeurIPS'21] "Chasing Sparsity in Vision Transformers: An End-to-End Exploration" by Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, Zhangyang Wang
pruning model-compression sparse-training vision-transformers efficient-transformers dynamic-sparsity token-slimming
-
Updated
Dec 1, 2023 - Python
[ICML 2021] "Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training" by Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, Mykola Pechenizkiy
sparsity deep-learning generalization sparse-training over-parameterization overparameterization in-time-overparameterization dynamic-sparse-training in-time-over-parameterization
-
Updated
Nov 11, 2023 - Python
[ICML2022] Training Your Sparse Neural Network Better with Any Mask. Ajay Jaiswal, Haoyu Ma, Tianlong Chen, ying Ding, and Zhangyang Wang
-
Updated
Jul 24, 2022 - Python
[Machine Learning Journal (ECML-PKDD 2022 journal track)] Quick and Robust Feature Selection: the Strength of Energy-efficient Sparse Training for Autoencoders
sparsity deep-learning feature-selection autoencoder dimensionality-reduction denoising-autoencoders sparse-neural-networks sparse-training
-
Updated
Oct 2, 2023 - Python
[IJCAI 2022] "Dynamic Sparse Training for Deep Reinforcement Learning" by Ghada Sokar, Elena Mocanu , Decebal Constantin Mocanu, Mykola Pechenizkiy, and Peter Stone.
deep-neural-networks sparsity reinforcement-learning deep-learning deep-reinforcement-learning sparse-neural-networks sparse-training continuous-control-tasks
-
Updated
May 13, 2022 - Python
Actually-sparse dynamic training for PyTorch. CPU-native, Apple Silicon first. Pluggable routers, drop-in SparseLinear.
deep-learning pytorch pybind11 sparse-neural-networks sparse-training apple-silicon neon-intrinsics dynamic-sparse-training cpu-optimization rigl
-
Updated
May 19, 2026 - Python
[ICML 2021] "Selfish Sparse RNN Training" by Shiwei Liu, Decebal Constantin Mocanu, Yulong Pei, Mykola Pechenizkiy
recurrent-neural-networks awd-lstm onlstm sparse-training dynamic-sparse-training awd-mos-lstm sparse-rnn-training
-
Updated
Oct 8, 2021 - Python
Code for "Training Adversarially Robust Sparse Networks via Bayesian Connectivity Sampling" [ICML 2021]
-
Updated
Mar 14, 2022 - Python
moe-engine is a research-grade infrastructure layer for training large Mixture-of-Experts language models at hyperscale. It is designed around one core constraint: at 10K+ GPUs, nodes die continuously. The system must keep training alive end-to-end — routing correctly, checkpointing durably, and resuming without operator intervention.
machine-learning fault-tolerance pytorch triton moe distributed-training mixture-of-experts sparse-training llm-training production-infrastructure
-
Updated
Jul 1, 2026 - Python
Implementation for the paper "SpaceNet: Make Free Space For Continual Learning" in PyTorch.
deep-neural-networks deep-learning incremental-learning sparse-representations brain-inspired continual-learning catastrophic-forgetting sparse-neural-networks sparse-training life-long-learning
-
Updated
Feb 28, 2021 - Python
PyTorch Implementation of TopKAST
-
Updated
Dec 28, 2022 - Python
[Machine Learning Journal (ECML-PKDD 2022 journal track)] A Brain-inspired Algorithm for Training Highly Sparse Neural Networks
-
Updated
Feb 20, 2023 - Python
[TMLR] Supervised Feature Selection with Neuron Evolution in Sparse Neural Networks
python deep-learning keras efficiency high-dimensional-data feature-selection neural-networks sparse-neural-networks sparse-training dynamic-sparse-training
-
Updated
Feb 12, 2023 - Python
This is the repository for the SNN-22 Workshop paper on "Generalization and Memorization in Sparse Neural Networks".
-
Updated
Feb 15, 2023 - Python
This project is subproject of the COMFORT.
-
Updated
Jun 30, 2026 - Jupyter Notebook
Staged Embarrassment Learning (SEL) is a bio-inspired framework for efficient Deep Learning. Inspired by a child’s rapid correction after a mistake, SEL uses dynamic gradient sparsity to focus compute on high-loss "embarrassing" samples . It achieves up to 99% FLOPs reduction, making it ideal for Edge AI.
computer-vision pytorch resnet cifar10 curriculum-learning edge-ai green-ai sparse-training model-optimization efficient-deep-learning gradient-masking
-
Updated
Apr 23, 2026 - Jupyter Notebook
The most compute-efficient training framework for ResNet architectures. 99.1% FLOPs reduction. 100% Local.
computer-vision pytorch resnet cifar10 curriculum-learning edge-ai green-ai sparse-training model-optimization efficient-deep-learning gradient-masking
-
Updated
May 28, 2026 - Jupyter Notebook
Improve this page
Add a description, image, and links to the sparse-training topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the sparse-training topic, visit your repo's landing page and select "manage topics."