Konstantin Mishchenko

Konstantin Mishchenko

Research Scientist

Samsung

Bio

I’m a research scientist at Samsung AI Center in Cambridge, UK. I like using mathematics to make things work in practice, especially in deep learning applications. Beside doing research, I serve as an Action Editor for TMLR, tweet about interesting papers, and give talks about my research. In 2023, I was lucky to receive the Outsdanding Paper Award together with Aaron Defazio for our work on adaptive methods.

Before joining Samsung, I did a postdoc at Inria Sierra with Alexandre d’Aspremont and Francis Bach. I received my PhD in from KAUST, where I worked under the supervision of Peter Richtárik on optimization theory and its applications in machine learning. In 2020, I also interned at Google Brain. I obtained my double degree MSc diploma from École Normale Supérieure Paris-Saclay and Paris-Dauphine, and a BSc from Moscow Institute of Physics and Technology.

My interests and hobbies tend to change every couple of years or so. Right now, I’m studying electronic music production using Ableton Live, and I hope to release some music online in the future. Feel free to shoot me an email if you want to chat in person about research or music, go to a museum, or maybe just take a walk in London!

Note that I’m currently not taking any interns.

Interests
  • Optimization
  • Deep learning
  • Federated and distributed learning
Education

  • PhD in Computer Science, 2021

    KAUST

  • MSc in Data Science, 2017

    École normale supérieure Paris-Saclay and Paris-Dauphine

  • BSc in Computer Science and Physics, 2016

    Moscow Institute of Physics and Technology

Experience

 
 
 
 
 
Samsung
Research Scientist
Samsung
Jan 2023 – Present Cambridge, UK
Working on federated learning and embedded AI systems as a member of the Distributed AI team.
 
 
 
 
 
Inria Sierra
Postdoc
Inria Sierra
Dec 2021 – Dec 2022 Paris, France
Conducted research on adaptive, second-order, and distributed optimization.

Recent Posts

Research directions in optimization
In this post, I share my thoughts on some interesting research direction. Not an exhaustive list, just something I came up with in 30 minutes when responding to an email from a student.
Last updated on Sep 26, 2024 3 min read
Invited lecture at Cambridge University
Today I gave an invited lecture at the University of Cambridge in the Federated Learning class taught by Nic Lane. My presentation was structured around the optimization aspects of federated learning, with the emphasis on what theoretical aspects of a method matter when we are trying to understand how it works in practice.
Last updated on Feb 21, 2024 1 min read
Thoughts on doing PhD in optimization
In this post, I would like to share some thoughts on whether it’s a good idea to choose optimization as one’s PhD topic. The text was written as a response to a MSc student, who reached out asking this question, but my response is general enough to share it here for anyone thinking about doing a PhD.
Last updated on Feb 21, 2024 4 min read
DoWG accepted at NeurIPS
Our work on an extension of DoG with weighted gradients got accepted for presentation at NeurIPS this year! If you want to try our method, a pytorch implementation is available on github. I hope to see more papers building upon DoG, DoWG, D-Adaptation, and Prodigy, we have barely scratched the surface on what can be done, and some of these methods are already being used in practice.
Last updated on Sep 24, 2023 1 min read
DoWG accepted at NeurIPS
ICML Outstanding Paper Award
I’m delighted to share that Aaron Defazio and I received the ICML Outstanding Paper Award for our work on D-Adaptation. The associated github repository of our paper has been quite popular and we are working hard on making extensions that will make adaptive methods even more useful for deep learning. Our first extension, Prodigy, is available on github as well and has been performing even better than D-Adaptation in our experiments. Expect more updates from us pretty soon!
Last updated on Sep 24, 2023 1 min read
ICML Outstanding Paper Award
See all posts

Recent Papers

Quickly discover relevant content by filtering publications.
Hao Mark Chen, Wayne Luk, Ka Fai Cedric Yiu, Rui Li, Konstantin Mishchenko, Stylianos I. Venieris, Hongxiang Fan (2024). Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference.

PDF Cite Slides arXiv

Aaron Defazio, Xingyu Yang, Harsh Mehta, Konstantin Mishchenko, Ahmed Khaled, Ashok Cutkosky (2024). The Road Less Scheduled.

PDF Cite arXiv

Aaron Defazio, Ashok Cutkosky, Harsh Mehta, Konstantin Mishchenko (2023). When, Why and How Much? Adaptive Learning Rate Scheduling by Refinement.

PDF Cite arXiv

Konstantin Mishchenko, Aaron Defazio (2023). Prodigy: An Expeditiously Adaptive Parameter-Free Learner.

PDF Cite Code Poster Slides Video arXiv

Konstantin Mishchenko, Rustem Islamov, Eduard Gorbunov, Samuel Horváth (2023). Partially Personalized Federated Learning: Breaking the Curse of Data Heterogeneity.

PDF Cite arXiv

Ahmed Khaled, Konstantin Mishchenko, Chi Jin (2023). DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Method.

PDF Cite Code Slides arXiv

Blake Woodworth, Konstantin Mishchenko, Francis Bach (2023). Two Losses Are Better Than One: Faster Optimization Using a Cheaper Proxy.

PDF Cite arXiv ICML

Aaron Defazio, Konstantin Mishchenko (2023). Learning-Rate-Free Learning by D-Adaptation.

PDF Cite Code arXiv ICML

Konstantin Mishchenko, Slavomír Hanzely, Peter Richtárik (2023). Convergence of First-Order Algorithms for Meta-Learning with Moreau Envelopes.

PDF Cite arXiv

Nikita Doikov, Konstantin Mishchenko, Yurii Nesterov (2022). Super-Universal Regularized Newton Method.

PDF Cite Code Slides arXiv

Samuel Horváth, Konstantin Mishchenko, Peter Richtárik (2022). Adaptive Learning Rates for Faster Stochastic Gradient Methods.

PDF Cite arXiv

Konstantin Mishchenko, Francis Bach, Mathieu Even, Blake Woodworth (2022). Asynchronous SGD Beats Minibatch SGD Under Arbitrary Delays.

PDF Cite Code Slides arXiv

Konstantin Mishchenko, Grigory Malinovsky, Sebastian Stich, Peter Richtárik (2022). ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!. ICML.

PDF Cite Code Video arXiv ICML

Grigory Malinovsky, Konstantin Mishchenko, Peter Richtárik (2022). Server-Side Stepsizes and Sampling Without Replacement Provably Help in Federated Optimization.

PDF Cite arXiv

Konstantin Mishchenko (2021). Regularized Newton Method with Global $O(1/k^2)$ Convergence.

PDF Cite Code Slides Video arXiv

Konstantin Mishchenko, Bokun Wang, Dmitry Kovalev, Peter Richtárik (2021). IntSGD: Adaptive Floatless Compression of Stochastic Gradients. ICLR.

PDF Cite Code Poster Slides arXiv ICLR

Konstantin Mishchenko, Ahmed Khaled, Peter Richtárik (2021). Proximal and Federated Random Reshuffling. ICML.

PDF Cite Code Slides Video arXiv ICML

Konstantin Mishchenko, Ahmed Khaled, Peter Richtárik (2020). Random Reshuffling: Simple Analysis with Vast Improvements. NeurIPS.

PDF Cite Code Poster Slides arXiv NeurIPS

Adil Salim, Laurent Condat, Konstantin Mishchenko, Peter Richtárik (2020). Dualize, Split, Randomize: Toward Fast Nonsmooth Optimization Algorithms. JOTA.

PDF Cite Poster arXiv JOTA

Dmitry Kovalev, Konstantin Mishchenko, Peter Richtárik (2019). Stochastic Newton and Cubic Newton Methods with Simple Local Linear-Quadratic Rates.

PDF Cite Poster arXiv NeurIPS

Yura Malitsky, Konstantin Mishchenko (2019). Adaptive Gradient Descent without Descent. ICML.

PDF Cite Code Poster Slides arXiv ICML Video

Konstantin Mishchenko (2019). Sinkhorn Algorithm as a Special Case of Stochastic Mirror Descent.

PDF Cite Poster Slides arXiv NeurIPS

Ahmed Khaled, Konstantin Mishchenko, Peter Richtárik (2019). First Analysis of Local GD on Heterogeneous Data.

PDF Cite Slides arXiv NeurIPS

Ahmed Khaled, Konstantin Mishchenko, Peter Richtárik (2019). Tighter Theory for Local SGD on Identical and Heterogeneous Data. AISTATS.

PDF Cite Slides arXiv AISTATS

Konstantin Mishchenko, Mallory Montgomery, Federico Vaggi (2019). A Self-supervised Approach to Hierarchical Forecasting with Applications to Groupwise Synthetic Controls.

PDF Cite arXiv ICML

Xun Qian, Alibek Sailanbayev, Konstantin Mishchenko, Peter Richtárik (2019). MISO is Making a Comeback With Better Proofs and Rates.

PDF Cite arXiv

Saeed Soori, Konstantin Mishchenko, Aryan Mokhtari, Maryam Mehri Dehnavi, Mert Gürbüzbalaban (2019). DAve-QN: A Distributed Averaged Quasi-Newton Method with Local Superlinear Convergence Rate. AISTATS.

PDF Cite arXiv AISTATS

Konstantin Mishchenko, Dmitry Kovalev, Egor Shulgin, Peter Richtárik, Yura Malitsky (2019). Revisiting Stochastic Extragradient. AISTATS.

PDF Cite Slides arXiv AISTATS

Konstantin Mishchenko, Peter Richtárik (2019). A Stochastic Decoupling Method for Minimizing the Sum of Smooth and Non-Smooth Functions.

PDF Cite Slides arXiv

Samuel Horváth, Dmitry Kovalev, Konstantin Mishchenko, Sebastian Stich, Peter Richtárik (2019). Stochastic Distributed Learning with Gradient Quantization and Double Variance Reduction. Optimization Methods and Software.

PDF Cite arXiv

Konstantin Mishchenko, Filip Hanzely, Peter Richtárik (2019). 99% of Worker-Master Communication in Distributed Optimization Is Not Needed. UAI.

PDF Cite arXiv UAI

Konstantin Mishchenko, Eduard Gorbunov, Martin Takáč, Peter Richtárik (2019). Distributed Learning with Compressed Gradient Differences.

PDF Cite arXiv

Konstantin Mishchenko, Peter Richtárik (2018). A Stochastic Penalty Model for Convex and Nonconvex Optimization with Big Constraints.

PDF Cite Poster arXiv

Filip Hanzely, Konstantin Mishchenko, Peter Richtárik (2018). SEGA: Variance Reduction via Gradient Sketching. NeurIPS.

PDF Cite arXiv NIPS

Konstantin Mishchenko, Franck Iutzeler, Jérôme Malick, Massih-Reza Amini (2018). A Delay-tolerant Proximal-Gradient Algorithm for Distributed Learning. ICML.

PDF Cite ICML

Konstantin Mishchenko, Franck Iutzeler, Jérôme Malick (2018). A Distributed Flexible Delay-tolerant Proximal Gradient Algorithm. SIOPT.

PDF Cite arXiv SIAM

Contact