lottery ticket hypothesis best paper

Navigation Menu

Search code, repositories, users, issues, pull requests..., provide feedback.

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly.

To see all available qualifiers, see our documentation .

• Notifications You must be signed in to change notification settings

A reimplementation of "The Lottery Ticket Hypothesis" (Frankle and Carbin) on MNIST.

google-research/lottery-ticket-hypothesis

Folders and files.

Repository files navigation

The lottery ticket hypothesis.

This codebase was developed by Jonathan Frankle and David Bieber at Google during the summer of 2018.

This library reimplements and extends the work of Frankle and Carbin in "The Lottery Ticket Hypothesis: Finding Small, Trainable Neural Networks" ( https://arxiv.org/abs/1803.03635 ). Their paper aims to explore why we find large, overparameterized networks easier to train than the smaller networks we can find by pruning or distilling. Their answer is the lottery ticket hypothesis:

Any large network that trains successfully contains a subnetwork that is initialized such that - when trained in isolation - it can match the accuracy of the original network in at most the same number of training iterations.

They refer to this special subset as a winning ticket .

Frankle and Carbin further conjecture that pruning a neural network after training reveals a winning ticket in the original, untrained network. They posit that were pruned after training were never necessary at all, meaning they could have been removed from the original network with no harm to learning. Once pruned, the original network becomes a winning ticket.

To evaluate the lottery ticket hypothesis in the context of pruning, they run the following experiment:

Randomly initialize a neural network.

Train the network until it converges.

Prune a fraction of the network.

To extract the winning ticket, reset the weights of the remaining portion of the network to their values from (1) - the initializations they received before training began.

To evaluate whether the resulting network at step (4) is indeed a winning ticket, train the pruned, untrained network and examine its convergence behavior and accuracy.

Frankle and Carbin found that running this process iteratively produced the smallest winning tickets. That is, the network found at step (4) becomes a new network to train and further prune again at steps (2) and (3). By training, pruning, resetting, and repeating many times, Frankle and Carbin achieved their best results.

This library reimplements Frankle and Carbin's core experiment on fully-connected networks for MNIST (Section 2 of their paper). It also includes several additional capabilities for further examining the behavior of winning tickets.

Getting Started

Run setup.py to install library dependencies.

Modify mnist_fc/locations.py to determine where to store MNIST ( MNIST_LOCATION ) and the data generated by experiments ( EXPERIMENT_PATH ).

Run download_data.py to install MNIST in those locations.

Code Walkthrough

This codebase is divided into four top-level directories: foundations , datasets , analysis , and mnist_fc .

Foundations

The foundations directory contains all of the abstractions and machinery for running lottery ticket experiments.

Dataset and Model

A learning task is represented by two components: a Model to train and a Dataset on which to train that network. Base classes for these abstractions are in foundations/dataset_base.py and foundations/model_base.py . Any networks on which you wish to run the lottery ticket experiment must subclass the ModelBase class in foundations/model_base.py ; likewise, any datasets on which you wish to train data must subclass the DatasetBase class in foundations/dataset_base.py . foundations.model_fc implements a generic fully-connected model.

Model objects in this codebase have two special features that distinguish them from normal models:

• masks : arrays of 0/1 values that are multiplied by tensors of network parameters to permanently disable particular parameters.
• presets : specific values to which network parameters should be initialized.

Masks are the mechanism by which weights are pruned. To prune a weight, set the value of the corresponding position in the mask to 0. Presets are the mechanism by which a network can be initialized to specific values, making it possible to perform the "reset" step of the lottery ticket experiment.

ModelBase has a dense method that mirrors the tf.layers.dense method but automatically integrates masks and presets. You should use this method when you build your networks so that weights can properly be managed during the course of the lottery ticket experiment, and you may desire to write similar methods for conv2d, etc. if you want to work with other kinds of layers.

Trainer, Pruner, and Experiment

To train a network, run the train function of foundations.trainer , providing it with a Model and Dataset .

A lottery ticket experiment comprises iteratively training a network, pruning it, resetting it, and training it again. The infrastructure for doing so is in the experiment function of foundations.experiment . This function expects four functions as arguments (in addition to other parameters).

• make_dataset : A function that generates a dataset. This function is called before each training run to re-generate the dataset.
• make_model : A function that generates the mode on which the lottery ticket experiment is to be run. This function is also called after each pruning step to generate the next model to be trained. This function can take masks and presets, which is useful for pruning the network and initializing its parameters to the same values as those of the original network.
• train_model : A function that trains the model generated by make_model on the dataset generated by make_dataset .
• prune_masks : A function that performs a pruning step, updating the previous masks based on network weights at the end of training. foundations.pruner implements the pruner from the original lottery ticket paper.

At a high level, here is how an experiment is structured:

An experiment consists of running the complete lottery ticket process (starting with a network and iteratively training and pruning many times). Experiments often build off of one another, reusing and transforming networks from other experiments for new purposes.

We often perform the same experiment more than once in order to demonstrate repeatability, so there are likely to be multiple trials for each experiment.

When k pruning steps have taken place, a network is said to be pruned to level k.

Training one individual network at one level of one experiment trial is called a run .

In a run, a network is trained for a certain number of training steps, or iterations .

Paths, Saving, and Restoring

The foundations.paths module contains helper functions for managing the locations where data generated by the experiments is stored. Each experiment generates five outputs:

The initial weights, final weights, and masks of the network.

Training, test, and validation loss and accuracy at frequent intervals throughout the training process as both a JSON file and a set of tensorflow summaries.

foundations.paths has functions that create the appropriate filenames for each of these records when provided with the directory in which they should be stored. It also has functions that structure where the results of a particular experiment, trial, and run are stored.

The foundations.save_restore module contains functions for saving networks, masks, and experimental results.

Networks and masks are stored as dictionaries whose keys are layer names and whose values are numpy arrays of the corresponding values for each layer. The standardize function of foundations.save_restore takes as input either a dictionary storing a network or the path to the location where such a network is stored; either way, it returns the dictionary. This function is used throughout the codebase to handle cases where a network could be represented by either a path or a dictionary.

The datasets directory stores specific datasets - children of the DatasetBase class. Right now, only dataset_mnist is present.

MNIST on a Fully-Connected Network

The mnist_fc directory contains the experimental infrastructure for instantiating the foundations on a fully-connected network for MNIST. It has three main components:

Top-level files.

Top-level Files

The top-level files ( train.py , lottery_experiment.py , etc.) contain the infrastructure for running MNIST experiments.

Support infrastructure:

locations.py : locations where datasets and data should be stored.

download_data.py : downloads MNIST and converts them into the formats expected by dataset_mnist.py .

constants.py : constants specific to the MNIST experiments (hyperparameters) and functions that construct locations for storing experimental results.

Infrastructure for running experiments:

train.py : Trains a single network, optionally with masks and presets.

lottery_experiment.py : Performs the lottery ticket experiment, optionally with presets.

reinitialize.py : Runs the random reinitialization ("control") experiment on a particular network.

For each of the scripts for running experiments, there is a corresponding runner which uses Python Fire to make these scripts callable from the command line. For example, you can run runners/lottery_experiment.py to execute it from the command line by using its function arguments as flags.

The argfiles directory contains scripts that generate sets of flags for the runners to perform experiments. The argfile_runner.py script will run the experiments specified in an argfile on a particular runner. For example:

python argfile_runner.py runners/lottery_experiment.py argfiles/lottery_experiment_argfile.py

will run the lottery experiment for each of the sets of flags generated by lottery_experiment_argfile.py .

Disclaimer: This is not an official Google product.

Contributors 2

• Python 100.0%

• DOI: 10.48550/arXiv.2403.04861
• Corpus ID: 268296813

A Survey of Lottery Ticket Hypothesis

• Bohan Liu , Zijie Zhang , +6 authors Bo Hui
• Published in arXiv.org 7 March 2024
• Computer Science, Mathematics

Figures and Tables from this paper

2 Citations

A survey on deep neural network pruning-taxonomy, comparison, analysis, and recommendations, prompt-prompted adaptive structured pruning for efficient llm generation, 123 references, lottery ticket preserves weight correlation: is it desirable or not, efficient lottery ticket finding: less data is more, the elastic lottery ticket hypothesis.

• Highly Influential
• 14 Excerpts

Finding Meta Winning Ticket to Train Your MAML

Evaluating lottery tickets under distributional shifts, playing the lottery with rewards and multiple languages: lottery tickets in rl and nlp, on the existence of universal lottery tickets, when bert plays the lottery, all tickets are winning, dual lottery ticket hypothesis.

• 13 Excerpts

Winning Lottery Tickets in Deep Generative Models

Related papers.

Showing 1 through 3 of 0 Related Papers

The Lottery Ticket Hypothesis

This project explores the Lottery Ticket Hypothesis: the conjecture that neural networks contain much smaller sparse subnetworks capable of training to full accuracy. In the course of this project, we have demonstrated that these subnetworks existed at initialization in small networks and early in training in larger networks. In addition, we have shown that these lottery ticket subnetworks are state-of-the-art pruned neural networks.

The practical goal of this project is to develop sparse neural networks that we can train from scratch or from early in training, creating the opportunity to dramatically reduce the cost of training. The scientific goal of this project is to better understand neural network optimization by empirically studying the behavior of practical, large-scale networks.

Communities

If you would like to contact us about our work, please refer to our members below and reach out to one of the group leads directly.

Last updated Apr 29 '20

Michael Carbin

Jonathan Frankle

Alexander renda, publications.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

Jonathan frankle , michael carbin, send feedback.

Enter your feedback below and we'll get back to you as soon as possible. To submit a bug report or feature request, you can use the official OpenReview GitHub repository: Report an issue

BibTeX Record

Your request couldn't be processed

There was a problem with this request. We're working on getting it fixed as soon as we can.

• Who’s Teaching What
• Subject Updates
• MEng program
• Opportunities
• Minor in Computer Science
• Resources for Current Students
• Program objectives and accreditation
• Graduate program requirements
• Admission process
• Degree programs
• Graduate research
• EECS Graduate Funding
• Resources for current students
• Student profiles
• Instructors
• DEI data and documents
• Recruitment and outreach
• Community and resources
• Get involved / self-education
• Rising Stars in EECS
• Graduate Application Assistance Program (GAAP)
• MIT Summer Research Program (MSRP)
• Sloan-MIT University Center for Exemplary Mentoring (UCEM)
• Electrical Engineering
• Computer Science
• Artificial Intelligence + Decision-making
• AI and Society
• AI for Healthcare and Life Sciences
• Artificial Intelligence and Machine Learning
• Biological and Medical Devices and Systems
• Communications Systems
• Computational Biology
• Computational Fabrication and Manufacturing
• Computer Architecture
• Educational Technology
• Electronic, Magnetic, Optical and Quantum Materials and Devices
• Graphics and Vision
• Human-Computer Interaction
• Information Science and Systems
• Integrated Circuits and Systems
• Nanoscale Materials, Devices, and Systems
• Natural Language and Speech Processing
• Optics + Photonics
• Optimization and Game Theory
• Programming Languages and Software Engineering
• Quantum Computing, Communication, and Sensing
• Security and Cryptography
• Signal Processing
• Systems and Networking
• Systems Theory, Control, and Autonomy
• Theory of Computation
• Departmental History
• Departmental Organization
• Visiting Committee
• News & Events
• News & Events
• EECS Celebrates Awards

Doctoral Thesis: The Lottery Ticket Hypothesis: On Sparse, Trainable Neural Networks

32-D463 (Star Room)

Jonathan Frankle

In this thesis defense, I will present my work on the “Lottery Ticket Hypothesis,” which provides a new perspective on understanding how neural networks learn in practice and how we can make this process more efficient. We have known for decades that it is possible to delete up to 90% of connections from trained neural networks (known as pruning) without any effect on accuracy. In my thesis work, I showed that it is also possible to train such pruned networks from at or near the start, something previous consensus deemed impossible. The takeaway of this finding is that neural networks can successfully learn with far less capacity than we typically provide. This has significant practical and scientific implications. Practically speaking, it sheds light on a new opportunity to dramatically reduce the cost of training the extraordinary models that are increasingly out of reach for all but the best resourced companies. Scientifically speaking, it surprisingly suggests that the capacity necessary for a neural network to learn a function is similar to the capacity necessary to represent it.

I will present the initial work on the Lottery Ticket Hypothesis (ICLR 2019 Best Paper Award), the follow-up work showing how to scale up these findings and providing insights into when and why sparse trainable networks exist (Linear Mode Connectivity and the Lottery Ticket Hypothesis, ICML 2020), and the state of affairs when it comes to exploiting these findings for practical purposes (Pruning Neural Networks at Initialization: Why are we missing the mark?, ICLR 2021). I will close by discussing the implications of this work, including the numerous new research directions it has catalyzed – such as on neural network pruning, efficient training, loss landscape analysis, model averaging for ensembling, and deep learning theory – and the evolution of this empirical approach to understanding and improving deep learning that forms the basis for my startup MosaicML.

For more information please contact: Nathan Higgins,  [email protected]

• Date: Friday, December 9
• Time: 1:30 pm - 3:00 pm
• Category: Thesis Defense
• Location: 32-D463 (Star Room)

Additional Location Details:

Thesis Supervisor: Prof. Michael Carbin

Subscribe to the PwC Newsletter

Join the community, edit social preview.

Add a new code entry for this paper

Remove a code repository from this paper.

Mark the official implementation from paper authors

Add a new evaluation result row.

Remove a task

Add a method, remove a method, edit datasets, stabilizing the lottery ticket hypothesis.

5 Mar 2019  ·  Jonathan Frankle , Gintare Karolina Dziugaite , Daniel M. Roy , Michael Carbin · Edit social preview

Pruning is a well-established technique for removing unnecessary structure from neural networks after training to improve the performance of inference. Several recent results have explored the possibility of pruning at initialization time to provide similar benefits during training. In particular, the "lottery ticket hypothesis" conjectures that typical neural networks contain small subnetworks that can train to similar accuracy in a commensurate number of steps. The evidence for this claim is that a procedure based on iterative magnitude pruning (IMP) reliably finds such subnetworks retroactively on small vision tasks. However, IMP fails on deeper networks, and proposed methods to prune before training or train pruned networks encounter similar scaling limitations. In this paper, we argue that these efforts have struggled on deeper networks because they have focused on pruning precisely at initialization. We modify IMP to search for subnetworks that could have been obtained by pruning early in training (0.1% to 7% through) rather than at iteration 0. With this change, it finds small subnetworks of deeper networks (e.g., 80% sparsity on Resnet-50) that can complete the training process to match the accuracy of the original network on more challenging tasks (e.g., ImageNet). In situations where IMP fails at iteration 0, the accuracy benefits of delaying pruning accrue rapidly over the earliest iterations of training. To explain these behaviors, we study subnetwork "stability," finding that - as accuracy improves in this fashion - IMP subnetworks train to parameters closer to those of the full network and do so with improved consistency in the face of gradient noise. These results offer new insights into the opportunity to prune large-scale networks early in training and the behaviors underlying the lottery ticket hypothesis

Code Edit Add Remove Mark official

Tasks edit add remove, datasets edit, results from the paper edit, methods edit add remove.

• Moscow Tourism
• Moscow Hotels
• Moscow Bed and Breakfast
• Moscow Vacation Rentals
• Flights to Moscow
• Moscow Restaurants
• Things to Do in Moscow
• Moscow Travel Forum
• Moscow Photos
• All Moscow Hotels
• Moscow Hotel Deals

Ticket Machine - worthless slip of paper?? - Moscow Forum

• Europe    
• Russia    
• Central Russia    
• Moscow    

Ticket Machine - worthless slip of paper??

• United States Forums
• Europe Forums
• Canada Forums
• Asia Forums
• Central America Forums
• Africa Forums
• Caribbean Forums
• Mexico Forums
• South Pacific Forums
• South America Forums
• Middle East Forums
• Honeymoons and Romance
• Business Travel
• Train Travel
• Traveling With Disabilities
• Tripadvisor Support
• Solo Travel
• Bargain Travel
• Timeshares / Vacation Rentals
• Central Russia forums
• Moscow forum

Went to Yarloslavskaya. Went to ticket kiosk/machine. Paid 240 rib for express ticket. It's printed a slip of paper with a barcode and a receipt. Not ticket. My friend used different machine. Same result. Waited forever to make sure a ticket wasn't being dispensed. Go to ticket window, ask how do I get a ticket for this? (You cant board d train with this slip). They run us to 3 different windows. I finally gave up bought ticket at window as train was leaving in 4 minutes. Anyone know what I do with this darn slip of paper?? Toss it and call it a day?

The slip of paper with a barcode IS your ticket. Just curious what did you expect - a glossy full-color brochure?

a slip of paper with a barcode

>>>

Do you have a photo? How does it look?

I was denied boarding with the slip of paper. I was tokd I need a ticket...so yeah anything other than that would have been fantastic.

> anything other than that would have been fantastic

Nothing other than that is possible unless you share a photo of your "slip of paper".

The ticket machines dispense valid tickets that look like slips of paper with bar codes. You don't need to exchange them for anything to enter platforms and board trains.

Without a photo we can only speculate:

Did you mistake something else for a ticket machine?

Did you ask for a wrong ticket?

Did you go to a wrong platform?

Did you try to board a wrong train?

Anything else?

I'll never understand why there's always one person that just has to be rude. Be kind.

I'll never understand why some people would see rudeness where there's only a sincere wish to help but never mind.

To try so sum up. What was your destination, please? And what train you wanted to board? Word "Express" is a little unclear here.

If you board suburban trains at Yaroslavskaya you buy a ticket (indeed a small piece of paper with a barcode), enter barcode to the ticket mashine, gates open, and you go to platform/train. Who could deny your boarding then?

Please provide more details to solve and help...

Ticket machines DO work and what they give you after the payment IS a ticket, which should be clear from the information this "slip of paper" contains. It's not just plain white and absolutely empty, right? I can only presume that you misunderstood how to use the ticket.

The bottom line is for “commuter” trains like these folks need to understand that indeed, this IS your ticket, not the larger, more substantial feeling ones we tend to expect and which are used for long-haul or express trains to more distant destinations.

This topic has been closed to new posts due to inactivity.

• cloths to be taken to visit mascow in the month of septemper Aug 17, 2024
• Alternate hotel recommendations Aug 16, 2024
• Help needed regarding taxi at airport please...... Jul 22, 2024
• Bolshoi Tickets. Jul 21, 2024
• Exchange USD to ruble and buy sim card upon arrival Jul 10, 2024
• Train Booking Moscow to St. Peter Jul 10, 2024
• Another Sapsan ticket question. Jul 09, 2024
• Bringing prescription medication to Russia in 2024 Jul 05, 2024
• guide for Moscow (and St.Petersburg) May 30, 2024
• Help needed please... May 30, 2024
• Moscow to St Petersburg train / seating May 28, 2024
• travel to moscow May 04, 2024
• Planning trip to Russia Apr 28, 2024
• SIM card. Russian SIM cards, do they still work in the UK? Apr 09, 2024
• Moscow to St Petersburg train or air?? 32 replies
• New Sapsan Express Train from Moscow to St Petersburg 18 replies
• New year's in moscow 8 replies
• Hop on Hop Off Bus Tour 5 replies
• How do you purchase Bolshoi Ballet tickets at a great price? 2 replies
• Select-a-room.com Are they legitimate? 3 replies
• Weather Moscow and St. petersburg in May 8 replies
• Night train to St Petersburg 3 replies
• ATM Access 12 replies
• Visa needed if on layover at Moscow Airport??????? 15 replies

Moscow Hotels and Places to Stay

• Where can I get initial answers to ANY question?
• Vacation Rentals
• GreenLeaders

This week: the arXiv Accessibility Forum

Help | Advanced Search

Computer Science > Machine Learning

Title: stabilizing the lottery ticket hypothesis.

Abstract: Pruning is a well-established technique for removing unnecessary structure from neural networks after training to improve the performance of inference. Several recent results have explored the possibility of pruning at initialization time to provide similar benefits during training. In particular, the "lottery ticket hypothesis" conjectures that typical neural networks contain small subnetworks that can train to similar accuracy in a commensurate number of steps. The evidence for this claim is that a procedure based on iterative magnitude pruning (IMP) reliably finds such subnetworks retroactively on small vision tasks. However, IMP fails on deeper networks, and proposed methods to prune before training or train pruned networks encounter similar scaling limitations. In this paper, we argue that these efforts have struggled on deeper networks because they have focused on pruning precisely at initialization. We modify IMP to search for subnetworks that could have been obtained by pruning early in training (0.1% to 7% through) rather than at iteration 0. With this change, it finds small subnetworks of deeper networks (e.g., 80% sparsity on Resnet-50) that can complete the training process to match the accuracy of the original network on more challenging tasks (e.g., ImageNet). In situations where IMP fails at iteration 0, the accuracy benefits of delaying pruning accrue rapidly over the earliest iterations of training. To explain these behaviors, we study subnetwork "stability," finding that - as accuracy improves in this fashion - IMP subnetworks train to parameters closer to those of the full network and do so with improved consistency in the face of gradient noise. These results offer new insights into the opportunity to prune large-scale networks early in training and the behaviors underlying the lottery ticket hypothesis

Submission history

Access paper:.

• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

1 blog link

Dblp - cs bibliography, bibtex formatted citation.

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .