Posts
Our work on an extension of DoG with weighted gradients got accepted for presentation at NeurIPS this year! If you want to try our method, a pytorch implementation is available on github. I hope to see more papers building upon DoG, DoWG, D-Adaptation, and Prodigy, we have barely scratched the surface on what can be done, and some of these methods are already being used in practice.
I’m delighted to share that Aaron Defazio and I received the ICML Outstanding Paper Award for our work on D-Adaptation. The associated github repository of our paper has been quite popular and we are working hard on making extensions that will make adaptive methods even more useful for deep learning. Our first extension, Prodigy, is available on github as well and has been performing even better than D-Adaptation in our experiments. Expect more updates from us pretty soon!
I was informed that I got selected as an Expert Reviewer for Transactions on Machine Learning Research (TMLR). As the email “Expert Reviewers are reviewers for TMLR who have done exemplary work in evaluating TMLR submissions. They have stood out in various ways, including by writing detailed reviews, engaging deeply with authors and performing their work in a timely fashion.”
The full list of TMLR Expert Reviewers is available here.
Today I’m giving an online talk at AIDRC Seminar Series of Technology Innovation Institute. The talk announcement can be found on the seminar’s website along with the abstract. In short, The topic of my presentation is our 2022 ICML paper ProxSkip and its several extension that were done by other authors.
My paper on Regularized Newton got accepted for publication at SIAM Journal on Optimization (SIOPT). The main result of this work is to show that one can globalize Newton’s method by using regularization proportional to the square root of the gradient norm. The corresponding method achieves global acceleration over gradient descent and it converges with the $O(1/k^2)$ rate of cubic Newton.