Menu

2022 Conference

News about ICLR 2022

Reflection on the DEI Initiative at ICLR 2022

750 participants, 55 proposals, 8 months, 1 workshop, 15 speakers. A new and thriving research community.

By ICLR 2022 Diversity, Equity & Inclusion Co-Chairs, Rosanne Liu and Krystal Maughan

ICLR 2022 has successfully come to an end. Having served as the DEI Chairs in the organizing committee, for the first time for both of us, we’d like to take this chance to reflect on this year’s effort—ups and downs, advances and setbacks, and most importantly, lessons and memories.

Last summer, how it all started

When we were invited to serve as the DEI chairs last summer, we were excited to make some real changes: we both have seen enough lip service, enough short-term, one-off engagements that while net positive, still didn’t amount to lasting changes. One of the central questions we asked ourselves was how we could really support underrepresented, underprivileged, and first-time submitters in a long-term capacity.

We started by first asking each other: how were you supported? How would you have liked to be supported?

More generally: how do people discover a love for research? How does someone decide to pursue research and what shapes that decision? Why do people stay in research? Is it the result of their first taste or experience in research, who they do research with, that ordains the course? What kind of support network do they have, or lack?

These were our answers.

Krystal: 

“I did not know that someone could have a career in research. I am a non-traditional student; a professor encouraged me to apply for grad school after I volunteered for the ACM Principles of Programming Languages (POPL) conference as the only non-PhD student.  Someone took a chance on me. Pivotal moments can have a lasting impact on someone’s decision to consider graduate school. These moments are a result of a field’s environment (whether positive or negative), access to opportunity, and some degree of luck. I want to minimize the need for that degree of luck and make the opportunity to pursue research accessible.

Why do people stay in research? The support of peers, and multi-layered or structural levels of support (including financial support) help us to persevere. Being able to look ahead and see persons like yourself who have succeeded makes a difference. We wanted to build this regardless of each person’s history or opportunity to find this in their own community. We wanted to build a network of mentors and early career researchers who advocate for themselves, have the freedom to take risks, and celebrate their accomplishments. We wanted them to have a space where they can see that we all experience ups and downs in research, but we can bounce back, find support, and keep going.“

Rosanne: 

“I never really loved research until I met a great mentor and worked with a supportive team, and that was way past my “supposed to love research” years—I had already graduated my PhD and started an industry job. I consider myself non-traditional for that reason: I followed a “traditional” path but did not hear the calling until I’m out of the path. But that’s when I realized not all of us have to go through the same growth trajectory to become good at something. I am actually really good at this research thing even as a late bloomer. What precipitates the flip from thinking that I suck to knowing that I excel is just one positive experience. 

I can’t stand thinking how many people are held back by the lack of that one crucial positive, self-perception-altering experience, not knowing what they could be good at, had they had a little bit of initial luck.“


It is from there that we started converging to the idea of a concrete program that integrates DEI efforts with the core of scientific research, the main track of the conference, and moves from short-term, one-off engagements (for example, giving out free registrations) to long-term nurturing and support of the whole community. 

The 1st CSS Program

We announced the CoSubmitting Summer program with a blog post, last summer, the beginning of a 8-month coworking program for underrepresented researchers (definition here) to submit to ICLR or other sister venues.

The program was a success! At the ICLR 2022 Opening Session, we summarized the 1st CSS program: it attracted 750+ participants to the Slack, and an additional 20+ experienced researchers willing to serve as mentors. Together, we wrote research proposals, formed teams, sought actively mentorship actively, and had recurring meetings to shape our own research paths. The program sponsored 55 projects in total, all of them led by at least one underrepresented minority (URM), with a total amount of $17,650 spent; amount funded per project ranges from $200 to $500. After 8 months, this resulted in 15+ submitted papers, not just to ICLR, but to other sister conferences and workshops. 

Most importantly, we hope this initiative helps plant seeds for many more future scientific advancements contributed to the entire academic society. Since launched, we are aware that a number of similar initiatives followed suit, including new submission mentoring and mentor-matching programs at AISTATS and other conferences. We are really happy to see the influence of this program going further than we initially expected.

We hosted a CSS Workshop to celebrate the milestone on the last day of ICLR 2022. After the workshop, we received a heartwarming Kudoboard from attendees. Thank you!

Challenges

This was our first time launching such a large-scale experiment of open collaboration, which means the first iteration of this program is not without challenges. Some of our challenges involved keeping the momentum, resolving team conflicts, and finding solutions to reimbursing participants in locations not traditionally represented at ICLR. We also learned that our initial timeline goal of initially submitting to ICLR wasn’t as feasible; although, we had wanted this effort to make contributions to the whole research community, not just limited to ICLR, and judging by the result, a lot of CSS-initiated projects do end up getting submitted to, and accepted by other sister venues.

Looking Forward

Where do we go from here? The most important thing to ensure is not to have this program be a one-off engagement. In asking our initial questions, we found that the community we had built was not only thriving, but could be sustained long-term. Many expressed a need for long-term mentorship and support in pursuing research, which they had grown to love. When we look at who we choose to work with in our research careers, how do we pick our collaborators? Who do we give chances to? We often choose those we like working with, those we know, and those with a shared methodology. Freedom to fail then becomes a privilege for those who do not fit our shared methodology. How might an individual who comes with a different perspective complement one’s research methodology and what might collaborators learn from this process? How might an individual be given the space and opportunity to learn and fail in order to become an excellent researcher? We want to challenge researchers to collaborate with less homophily, and instead embrace difference of perspectives and intellectual curiosity to solve problems.

A final word

We would like to extend gratitude to our co-organizers for the Broadening Participation initiative and the 1st CoSubmitting Summer workshop: Tom Burns, Ching Lam Choi, Arun Raja and Aya Salama, presenters of the workshop: En-Shiun Annie Lee, Sarubi Thillainathan, Edward Elson Kosasih, Mats L. Richter, Emily McMilin, Andrija Djurisic, Raza Hashmi, Qiang Li, Fatemeh Siar, Laura Fee Nern, Edwin Arkel Rios, Nuredin Ali, Krupal Shah, Connor McCurley, Peiyuan Zhou and Tian H Teh, as well as all the participants of the initiative, and the organizing committee of ICLR 2022, especially Katja Hofmann, Feryal Behbahani and Andrea Brown who graciously and tirelessly supported us.

Announcing the ICLR 2022 Outstanding Paper Award Recipients

,

By ICLR 2022 Senior Program Chair Yan Liu and Program Chairs Chelsea Finn, Yejin Choi, Marc Deisenroth

We are delighted to announce the recipients of the ICLR 2022 Outstanding Paper Awards!

First, we would like to thank the members of the ICLR community, including reviewers, area chairs, and senior area charis, who provided valuable discussions and feedback to guide the award selection. 

In addition, we would like to extend a special thanks to the Outstanding Paper Award Selection Committee, which consisted of Andreas Krause (ETHZ), Atlas Wang (UT Austin), Been Kim (Google Brain), Bo Li (UIUC), Bohyung Han (SNU), He He (NYU), and Zaid Harchaoui (UW), for generously sharing their time and expertise for making the final selection. 

Outstanding Paper Awards

The following seven papers are chosen as recipients of the Outstanding Paper Award, due to their excellent clarity, insight, creativity, and potential for lasting impact. Additional details about the paper selection process are provided below. 

The award recipients are (in order of paper ID):

Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models

By Fan Bao, Chongxuan Li, Jun Zhu, Bo Zhang

Defusion probabilistic model (DPM), a class of powerful generative models, is a rapidly growing topic in machine learning. This paper aims to tackle the inherent limitation of the DPM models, which is the slow and expensive computation of the optimal reverse variance in DPMs. The authors first present a surprising result that both the optimal reverse variance and the corresponding optimal KL divergence of a DPM have analytic forms with respect to its score function. Then they propose Analytic-DPM, a novel and elegant training-free inference framework that estimates the analytic forms of the variance and KL divergence using the Monte Carlo method and a pretrained score-based model. This paper is significant both in terms of its theoretical contribution (showing that both the optimal reverse variance and KL divergence of a DPM have analytic forms) and its practical benefit (presenting a training-free inference applicable to various DPM models), and will likely influence future research on DPMs.

*This paper will be presented in the Oral Session 4 on Probabilistic Models & Vision on Apr 28 8am GMT (1am PST)

Hyperparameter Tuning with Renyi Differential Privacy

By Nicolas Papernot, Thomas Steinke

This paper provides new insights into an important blind spot of most of the prior analyses of the differential privacy of learning algorithms, namely the fact that the learning algorithm is run multiple times over the data in order to tune the hyperparameters. The authors show that there are situations in which part of the data can skew the optimal hyperparameters, henceforth leaking private information. Furthermore, the authors provide privacy guarantees for hyperparameter search procedures within the framework of Renyi Differential Privacy. This is an excellent paper considering the everyday use of learning algorithms and its implications in terms of privacy for society, and proposing ways to address this issue. This work will provide the foundation for many follow-up works on differentially private machine learning algorithms. 

*This paper will be presented in the Oral Session 1 on Learning in the Wild & RL on Apr 26 12am GMT (Apr 25 5pm PST).

Learning Strides in Convolutional Neural Networks

By Rachid Riad, Olivier Teboul, David Grangier, Neil Zeghidour

This paper addresses an important problem that anyone using convolutional networks has faced, namely setting the strides in a principled way as opposed to trials and errors. The authors propose a novel and very clever mathematical formulation for learning strides and demonstrate a practically useful method that achieves state-of-the-art experimental results in comprehensive benchmarks. The main idea is DiffStride, the first downsampling layer with learnable strides that allows one to learn the size of a cropping mask in the Fourier domain, effectively performing resizing in a way that is amenable to differentiable programming. This is an excellent paper that proposes a method that will likely be part of commonly used tool boxes as well as courses on deep learning.

*This paper will be presented in the Oral Session 2 on Understanding Deep Learning on Apr 26 8am GMT (1am PST).

Expressiveness and Approximation Properties of Graph Neural Networks

By Floris Geerts, Juan L Reutter

This elegant theoretical paper shows how questions regarding the expressiveness and separability of different graph neural networks GNN architectures can be reduced to (and sometimes substantially simplified by) examining their computations in tensor language, where these questions connect to well-known combinatorial notions such as the treewidth. In particular, this paper provides an elegant way to easily obtain bounds on the separation power of GNNs in terms of the Weisfeiler-Leman (WL) tests, which have become the yardstick to measure the separation power of GNNs. The proposed framework also has implications for studying approximability of functions through GNNs. This paper has the potential to make a significant impact for future research by providing a general framework for describing, comparing and analyzing GNN architectures. In addition, this paper provides a toolbox with which GNN architecture designers can analyze the separation power of their GNNs, without needing to know the intricacies of the WL-tests.

*This paper will be presented in the Oral Session 2 on Understanding Deep Learning on Apr 26 8am GMT (1am PST).

Comparing Distributions by Measuring Differences that Affect Decision Making

By Shengjia Zhao, Abhishek Sinha, Yutong (Kelly) He, Aidan Perreault, Jiaming Song, Stefano Ermon

This paper proposes a new class of discrepancies that can compare two probability distributions based on the optimal loss for a decision task. By suitably choosing the decision task, the proposed method generalizes the Jensen-Shannon divergence and the maximum mean discrepancy family. The authors demonstrate that the proposed approach achieves superior test power compared to competitive baselines on various benchmarks, with compelling use cases for understanding the effects of climate change on different social and economic activities, evaluating sample quality, and selecting features targeting different decision tasks. Not only is the proposed method intellectually elegant, the committee finds that the paper is exceptional for its empirical significance, as the fact that the method allows a user to directly specify their preferences when comparing distributions through the decision loss implies an increased level of interpretability for practitioners. 

*This paper will be presented in the Oral Session 4 on Probabilistic Models & Vision on Apr 28 8am GMT (1am PST).

Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path

By X.Y. Han, Vardan Papyan, David L. Donoho

This paper presents new theoretical insights on the “neural collapse” phenomenon that occurs pervasively in today’s deep net training paradigm. During neural collapse, last-layer features collapse to their class-means, both classifiers and class-means collapse to the same Simplex Equiangular Tight Frame, and classifier behavior collapses to the nearest-class-mean decision rule. Instead of the cross-entropy loss that is mathematically harder to analyze, the paper demonstrates a new decomposition of the mean squared error (MSE) loss in order to analyze each component of the loss under neural collapse, which in turn, leads to a new theoretical construct of “central path”, where the linear classifier stays MSE-optimal for feature activations throughout the dynamics. Finally, by studying renormalized gradient flow along the central path, the authors derive exact dynamics that predict neural collapse. In sum, this paper provides novel and highly inspiring theoretical insights for understanding the empirical training dynamics of deep networks.

*This paper will be presented in the Oral Session 2 on Understanding Deep Learning on Apr 26 8am GMT (1am PST).

Bootstrapped Meta-Learning

By Sebastian Flennerhag, Yannick Schroecker, Tom Zahavy, Hado van Hasselt, David Silver, Satinder Singh

Meta-learning, or learning to learn, has the potential to empower artificial intelligence, yet meta-optimization has been a considerable challenge to unlocking this potential. This paper opens a new direction in meta-learning, beautifully inspired from TD learning, that bootstraps the meta-learner from itself or another update rule. The theoretical analysis is thorough, and the empirical results are compelling, with a new state-of-the art for model-free agents on the Atari ALE benchmark and demonstrate that it yields both performance and efficiency gains in multi-task meta-learning. The committee believes that this paper will inspire a lot of people.

*This paper will be presented in the Oral Session 3 on Meta Learning and Adaptation on Apr 27 4pm GMT (9am PST).

Outstanding Paper Honorable Mentions

Understanding over-squashing and bottlenecks on graphs via curvature

By Jake Topping, Francesco Di Giovanni, Benjamin Paul Chamberlain, Xiaowen Dong, Michael M. Bronstein

Most graph neural networks (GNNs) use the message passing paradigm, which suffers from the “over-squashing” phenomenon, where the distortion of information flowing from distant nodes limits the efficiency of message passing, which in turn, has been heuristically attributed to graph bottlenecks. Drawing insights from discrete differential geometry, this paper provides a precise description of the over-squashing phenomenon in GNNs and analyzes how it arises from bottlenecks in the graph. In particular, the authors introduce a new edge-based combinatorial curvature and prove that negatively curved edges are responsible for the over-squashing issue. Moreover, the authors demonstrate an elegant approach to reducing these negative effects by rewiring the graph according to this curvature notion. The paper has a potential to make considerable impact by importing tools from differential geometry for analyzing GNNs, and the rewiring approach may suggest new directions for improving the empirical performance of GNNs.

*This paper will be presented in the Oral Session 2 on Structured Learning on Apr 26 8am GMT (1am PST).

Efficiently Modeling Long Sequences with Structured State Spaces

By Albert Gu, Karan Goel, Christopher Re

Modeling long sequences is a central challenge in representation learning across various tasks and modalities, and the dominant architecture over the past years has been Transformers. This paper investigates a surprising alternative to Transformers, by proposing the Structured State Space Sequence model (S4). The S4 model is basically a new and clever parameterization of the state space model (SSM), that can address the prohibitive computation and memory requirements of SSM, while maintaining the theoretical strengths of SSM for handling long-range dependencies. S4 demonstrates impressive empirical results on multiple domains including vision, text, and audio. Unlike most work that tries to make Transformers more efficient, this work takes a completely different approach by focusing on the less studied state space models. The technical elegance and empirical strengths of the proposed approach inspire a new research direction.

*This paper will be presented in the Oral Session 2 on Structured Learning on Apr 26 8am GMT (1am PST).

PiCO: Contrastive Label Disambiguation for Partial Label Learning

By Haobo Wang, Ruixuan Xiao, Yixuan (Sharon) Li, Lei Feng, Gang Niu, Gang Chen, Junbo Zhao

This paper studies Partial Label Learning (PLL), an important problem in real-world applications where each training example is labeled with a coarse candidate set due to label ambiguity. This paper aims to reduce the performance gap between PLL and the supervised counterpart, by addressing two key challenges in PLL—representation learning and label disambiguation—in one coherent framework. Specifically, the authors propose PiCO, a new framework that combines contrastive learning with prototype-based label disambiguation. The paper includes an interesting theoretical interpretation to justify their framework from an expectation-maximization (EM) perspective. The empirical results are particularly impressive, as PiCO significantly outperforms current state-of-the-art in PLL and even achieves comparable results to fully supervised learning. 

*This paper will be presented in the Oral Session 1 on Learning in the Wild & RL on Apr 26 12am GMT (Apr 25 5pm PST).

Selection Process

The Outstanding Paper Committee determined a selection process with the goal of identifying an equivalence class of outstanding papers that represent the breadth of excellent research being conducted by the ICLR community.

The committee began with an initial pool of 58 papers including all papers that were explicitly nominated for an award by area chairs or senior area chairs as well as all papers that were slated for an oral presentation. The committee used three phases of down-selection. During Phase 1, each paper was assigned to one primary reader to determine if the paper should move to Phase 2, and in addition, committee members optionally endorsed other papers that were outside their assignments to move to Phase 2. We had 20 shortlisted papers after Phase 1. During Phase 2, each paper was assigned with an additional secondary reader for closer examination, and in addition, committee members optionally endorsed other papers that were outside their assignments to move to Phase 3. We had 10 shortlisted papers after Phase 2. During Phase 3, all remaining papers were considered and approved by the entire committee based on the rankings and nomination notes shared by the readers from Phase 1 and 2. After Phase 3, the committee decided 7 papers in the equivalence set for the Outstanding Paper Awards, and additional 3 papers for the Honorable Mentions. At all phases, the committee members could read or endorse papers only if they do not have conflicts of interests, either based on the domain conflicts, or based on the personal relationship conflicts (e.g,. friends or former advisors). In order to promote honest and fair evaluations among the selection committee, the paper assignments were kept confidential so that judgments are not biased by the presence of other committee members who have conflicts of interests with some of the papers in the pool. 

Mentorship Program for New Reviewers at ICLR 2022

, ,

Summary

For ICLR 2022, we introduced a mentoring scheme for reviewers, who are new to reviewing for machine learning conferences. Below, we describe the motivation, the implementation, and the result.

Motivation

Finding reviewers for conferences or journals is challenging: The reviewing system is almost a closed circle with unclear/undefined processes to join, and where members of that circle are frequently overburdened with review requests. For example, at ICLR 2022 we merged reviewer databases from NeurIPS 2021, ICML 2021, and ICLR 2021 to reach out to more than 11,000 reviewers. We believe this kind of approach is fairly common to reach some critical mass in terms of reviewers. However, this also means that reviewers in those databases are contacted over and over again. On the other hand, as the machine learning community is growing, there’s no clear path for new reviewers to join. Sometimes potential reviewers reach out to the program chairs and ask whether they can join or where supervisors (who are already in the pool) add their PhD students to the pool. These approaches do not scale, and we miss out on many potential reviewers who could do a great job, but who cannot enter the reviewer pool through the ‘usual means’. 

This year, we had an open call for reviewer self-nominations, and we received approximately 700 requests from people, who did not get an invitation to review. About 250 of these applications were requests from reviewers with great experience in reviewing for machine learning conferences, and who should have been included in the original invites, but who fell through the cracks for some reason. However, there were also more than 350 requests from people who have either never (or very little) reviewed for major machine learning conferences. For these ‘new reviewers’, we decided to establish a mentoring program, where they are matched with an experienced and empathetic mentor, who supports them with the writing of a review.

Implementation

We hand-picked 30 mentors (who were neither reviewers nor area chairs at ICLR) to support approximately 300 new reviewers. Based on their domain expertise, new reviewers will be assigned a single paper submitted to ICLR 2022 for which they drafted a review within 10 days. We provided the new reviewers with examples of good and bad reviews, a checklist and a template to simplify the creation of the review.

When the review was done, they reached out to their assigned mentor, who provided feedback on the quality of the review. For example, they checked whether feedback is constructive and helpful or whether the language is polite. The mentors were not expected to have read the paper, but only to provide feedback on the reviews themselves. Mentors and new reviewers had about 10 days to iterate after which the mentor made a decision whether the review was of sufficiently high quality (at this point the mentorship program ended). If that was the case, the review (and the reviewer) entered the regular pool of reviewers for ICLR, which allowed them to participate in discussions. 220 ‘new’ reviewers entered the reviewer pool in this way; the timing was synced with the submission of the regular reviews.

Reviewer Mentors

We want to express our gratitude to the reviewer mentors: Arno Solin, Sergey Levine, Martha White, Hugo Larochelle, James R. Wright, Amy Zhang, Sinead Williamson, Nicholas J. Foti, Samy Bengio, Nicolas Le Roux, Matthew Kyle Schlegel, Hugh Salimbeni, Andrew Patterson, Andrew Jacobsen, Mohammad Emtiyaz Khan, Ulrich Paquet, Carl Henrik Ek, Iain Murray, Cheng Soon Ong, Markus Kaiser, Savannah Jennifer Thais, Finale Doshi-Velez, Irina Higgins, Mark van der Wilk, Edward Grefenstette, Brian Kingsbury, Pontus Saito Stenetorp, David Pfau, Danielle Belgrave, Mikkel N. Schmidt.

Statistics

‘Regular’ reviewerMentee
Average review length501 words859 words
Average number of discussion comments1.541.73
Average comment length128 words165 words
Table 1: Performance of ‘regular’ reviewers and reviewer mentees measured in terms of review length and interactions during the rebuttal period.

We looked at some basic statistics to see how reviewer mentees compare with ‘regular’ reviewers (see table above). It turns out that reviews by reviewer mentees were longer on average, compared with reviews that came through the regular reviewer pool. Furthermore, on average, mentees also showed more engagement in the discussion (see average number of comments and average length of comments). While the discussion period was outside the mentorship program, i.e., no further feedback from the mentors was provided, we assume that the increased level of engagement is related to a lower paper load (1 paper for mentees, 1.87 papers on average across all reviewers) and the fact that reviewer mentees were fairly comfortable with the paper they reviewed. 

Survey Feedback

A survey conducted in early February 2022 (i.e. after the decision notifications) amongst all reviewer mentees revealed (130 respondents):

  • Mentees were predominantly PhD students (65%); see Figure above
  • Most mentees had reviewed never (53%) or once (26%) for a major machine learning conferences
  • Goals (>95% (strongly) agree) and the process (>91% (strongly) agree) of the mentorship program were clear 
  • Reviewer guidelines were useful (>93% (strongly) agree)
  • Paper assignments based on affinity scores were good (>85% (strongly) agree), although 4% of the respondents found the assigned papers too far from their research expertise.
  • Mentees had enough time to review the paper (>93% (strongly) agree)
  • Mentor was responsive and provided useful feedback (90% (strongly) agree)
  • 77% of the respondents were promoted to ‘regular’ reviewer by their corresponding mentors
  • Mentees found the mentorship program useful (>93% (strongly) agree)
  • Mentees recommend that the mentorship program should be offered again (>96% (strongly) agree)

The success of the mentorship program very much relies on the mentors and their timely feedback. If communication between mentor and mentee is stuck, the learning experience is somewhat reduced, which a few people pointed out. However, the vast majority of respondents highly valued the discussions and communication with mentors, which helped them to write a good review. 

Summary

In 2022, we piloted a mentorship program for reviewers with little experience reviewing for machine learning conferences. To support these new reviewers, we assigned them an experienced reviewer to support them in writing a high-quality review. Most of the reviews were of very high quality, so that they entered the regular reviewing pool/cycle. Overall, the mentorship program received overwhelmingly positive feedback.

Announcing ICLR 2022 Keynote Speakers

,

By ICLR 2022 Senior Program Chair Yan Liu and Program Chairs  Chelsea Finn, Yejin Choi, Marc Deisenroth

We are thrilled to host the following keynote speakers at ICLR 2022:

  • John H. Amuasi (Kwame Nkrumah University of Science and Technology)
  • Jenny L. Davis (The Australian National University)
  • Been Kim (Google Brain)
  • Pushmeet Kohli (DeepMind)
  • Kunle Olukotun (Stanford University)
  • Doina Precup (McGill University & Mila; DeepMind Montreal) 
  • Cordelia Schmid (Inria; Google)
  • H. Sebastian Seung (Princeton University; Samsung Research)

The keynote talks will cover a diverse set of research topics, ranging from fundamental research in deep learning, such as deep reinforcement learning, interpretability, computer vision, AI ethics, and ML systems, to breakthroughs of deep learning in scientific domains, such as protein structure prediction, neuroscience, and global health and infectious diseases.  At ICLR 2022, each invited speaker will give a 30-min talk, followed by a live Q&A. We hope that the talks will provoke us into thinking deeply about both the technical foundations of deep learning and its increasing impact on society.

We solicited keynote speaker nominations from area chairs and senior area chairs of ICLR 2022 as well as the public. Among more than 60 nominations, we prioritized candidates who not only have conducted exciting research but also have contributed to diversity in the community. We would like to thank all ACs/SACs who submitted nominations of excellent speaker candidates and the Diversity Equity & Inclusion Chairs, Krystal Maughan and Rosanne Liu for their valuable input. 

We cannot wait to see the invited talks and participate in the live Q&A for all speakers at ICLR 2022. 

Announcing the Accepted Workshops at ICLR 2022

By ICLR 2022 Workshop Chairs, Feryal Behbahani and Vukosi Marivate

We want to thank everyone who submitted a proposal. We were very impressed by the quality of these proposals and thrilled to be able to include some of them as part of the ICLR program. We received 45 total submissions and accepted 19 workshops.  

We also want to thank our team of reviewers, ICLR program committee and ICLR 2021 Workshop Chairs for their kind support. 

The workshops will be held virtually (like the rest of the  conference) in April 2022.

Selection Process

Ultimately we could not accept every proposal but we worked to balance a few goals. We wanted to have a good representation of theory, applications and innovative multidisciplinary workshops to broaden the ICLR community. Additionally, as we have a single day for workshops, we wanted each workshop to be able to attract a good number of attendees, hence we tried to avoid having workshops with very similar topics.

A team of invited volunteer reviewers reviewed each workshop proposal which was an important part of the selection process. We strived to have a good balance between topics and took into account the diversity of the speakers and organising team across multiple factors such as gender, institution, seniority and geographic location.

We will not be releasing the reviews directly, but will be making available short meta-reviews on CMT to summarise the feedback, highlighting the strengths and opportunities for improvement.

We are looking forward to all your contributions to the workshops!

Accepted Workshops

3rd Workshop on practical ML for Developing Countries: learning under limited/low resource scenarios

AfricaNLP 2022: NLP for African languages

AI for Earth and Space Science

Deep Generative Models for Highly Structured Data

Deep Learning for Code

Deep Learning on Graphs for Natural Language Processing

Emergent Communication: New Frontiers

From Cells to Societies: Collective Learning Across Scales

Gamification and Multiagent Solutions

Generalizable Policy Learning in the Physical World

Geometrical and Topological Representation Learning

GroundedML: Anchoring Machine Learning in Classical Algorithmic Theory

Machine Learning for Drug Discovery (MLDD)

PAIR^2Struct: Privacy, Accountability, Interpretability, Robustness, Reasoning on Structured Data

Setting up ML Evaluation Standards to Accelerate Progress

Socially Responsible Machine Learning

Wiki-M3L: Wikipedia and Multimodal & Multilingual Research

Workshop on Agent Learning in Open-Endedness

Workshop on the Elements of Reasoning: Objects, Structure and Causality

Additionally we are very excited to announce that there will be a workshop on CoSubmitting Summer (CSS) Program which will be the official conclusion of the 1st CSS program featuring the 55 submitted projects. 

Blog Posts as Conference Contributions

By the Blog Post Track chairs: Bubeck, Sebastien, Microsoft; Dobre, David, Mila; Gauthier, Charlie, Mila; Gidel, Gauthier, Mila; Vernade, Claire, DeepMind

Motivations

The Machine Learning community is currently experiencing a reproducibility crisis and a reviewing crisis [Littman, 2021]. Because of the highly competitive and noisy reviewing process of ML conferences [Tran et al., 2020], researchers have an incentive to oversell their results, slowing down the progress and diminishing the integrity of the scientific community. Moreover with the growing number of papers published and submitted at the main ML conferences [Lin et al., 2020], it has become more challenging to keep track of the latest advances in the field.

Blog posts are becoming an increasingly popular and useful way to talk about science [Brown and Woolston, 2018]. They offer substantial value to the scientific community by providing a flexible platform to foster open, human, and transparent discussions about new insights or limitations of a scientific publication. However, because they are not as recognized as standard scientific publications, only a minority of researchers manage to maintain an active blog and get visibility for their efforts. Many are well-established researchers (Francis Bach, Ben Recht, Ferenc Huszár, Lilian Weng) or big corporations that leverage entire teams of graphic designers designer and writers to polish their blogs (Facebook AI, Google AI, DeepMind, OpenAI). As a result, the incentives for writing scientific blog posts are largely personal; it is unreasonable to expect a significant portion of the machine learning community to contribute to such an initiative when everyone is trying to establish themselves through publications.

Our goal is to create a formal call for blog posts at ICLR to incentivize and reward researchers to review past work and summarize the outcomes, develop new intuitions, or highlight some shortcomings. A very influential initiative of this kind happened after the second world war in France. Because of the lack of up-to-date textbooks, a collective of mathematicians under the pseudonym Nicolas Bourbaki [Halmos 1957], decided to start a series of textbooks  about the foundations of mathematics [Bourbaki, 1939]. In the same vein, we aim at providing a new way to summarize scientific knowledge in the ML community.

Our Idea: Blog post Conference Track

Due to the large diversity of topics that can be discussed in a blog post, we decided to restrict the range of topics for this call for blog posts. We identified that the blog posts that would bring to most value to the community and the conference would be posts that distill and discuss previously published papers.

Call for blog posts on papers previously published at ICLR

The call for blog post would take the following form:

  • Write a post about a paper previously published at ICLR, with the constraint that one cannot write a blog post on work that they have a conflict of interest with. This implies that one cannot review their own work, or work originating from their institution or company. We want to foster productive discussion about ideas, and prevent posts that intentionally aim to help or hurt individuals or institutions.

  • Blogs will be peer-reviewed (double-blind, see Section 2.5) for quality and novelty of the content: clarity and pedagogy of the exposition, new theoretical or practical insights, reproduction/extension of experiments, etc.

  • The posts will be published under a unified template (see Section 2.4 and Section 2.5) and hosted on the conference website or our own Github page.

Positive Impact for the Community

We believe having this call for blog posts as a conference track would increase the posts’ visibility, impact, and credibility, while simultaneously providing benefits to the conference.

  • Adoption: we think that, with the conference’s stamp, such a format will be more broadly recognized and adopted by the community.

  • Accessibility: maintaining a blog is time consuming , and requires many blog posts to gain a stable following. By allowing researchers to publish a single post, we will permit occasional blog writers to publish their ideas, something that is relatively impossible right now. Moreover, it will make this format accessible to more independent/junior blog writers that do not have a company or a research lab to support them.

  • Synchronization: the fast evolving field of ML advances at the paces of its conferences. By following the same pace the blog posts will add value and momentum to the conference. It will benefit from the same advantages of conferences with respect to scientific journals: faster publication process and cross-fertilization of ideas.

Positive Impact for the Conference

We develop the potential positive impact of a blog post track for the conference itself:

  • Increases the value of the papers submitted to ICLR: blog posts will discuss previously published papers, thus increasing their visibility and quality.

  • Incentivizes researchers to submit their best research to ICLR: high quality work will likely get highlighted in future years in a blog post.

  • Improves reproducibility and transparency: the blog post track will identify and publicly document pitfalls and "tricks" that were not clearly communicated in the original publication.

  • Provides a scientific value by itself: such blog posts will reproduce and extend results of previously published papers. They will distill important theoretical and practical ideas improving their adoption and impact.

  • Tests of time: this track will provide a sort of crowd-sourced test of time at a shorter timescale than the current test of times awards.

  • Promotes accessibility: because many of this track’s blog posts will vulgarize past content, this track will make the conference broadly more accessible (to students, non-natives, and, more generally, non-experts in the field).

Submission Format

Our goal is to avoid heavily engineered, professionally-made blog-posts—Such as the “100+ hours” mentioned as a standard by the Distill guidelines—to entice ideas and clear writing rather than dynamic visualizations or embedded javascript engines.

As a result, we restrict submissions to the Markdown format. We believe this is a good trade-off between complexity and flexibility. Markdown enables users to easily embed media such as images, gifs, audio, and video as well as write mathematical equations using MathJax, without requiring users to know how to create HTML web pages. This (mostly) static format is also fairly portable; users can download the blog post without much effort for offline reading or archival purposes. More importantly, this format can be easily hosted and maintained through GitHub.

Submission Process

A full copy of the track’s blogs will always be publicly available as a GitHub repository.

The process for creating and submitting a blog post is as follows:

  1. Entrants will fork this repository and make their fork private. Failure to do so will result in the submission being rejected, as it breaches the double-blind review process.

  2. Users will modify their fork as they see fit; they will add their post along with any media files it might require. Since this is a full fork, they will be able to view their own copy of the blog. This means that they will be able to see exactly how their post will look and behave on the main website.

  3. Once completed, entrants will anonymize their blog post (i.e. strip their name, affiliation, etc).

  4. Entrants will download a ZIP of their anonymized fork (see figure below), and submit the ZIP to our OpenReview venue.

Download instructions image

  1. Once accepted, entrants will de-anonymize their post, make their fork public again, and make a Pull Request on Github from their fork to the main blog, allowing us to pull in their new blog post in a transparent way.

Once the submission period has ended, the GitHub repository of our track will be temporarily made private for the duration of the conference, allowing the conference to host the website. After the conference, the GitHub repository will be made public again to allow viewers to fork and download its contents.

The potential Pitfalls of our Blog Post Track

In this section we identify potential issues arising with such a track and explain how to mitigate them:

  1. Adversarial Blog Posts: Since the guidelines are to write a blog post on a previously published paper, one may expect some researcher to try to use bad faith arguments to criticize a concurrent paper through one of these blog post. We do not think this will happen, because these blog posts will be public and thus researchers would discredit themselves by using bad faith arguments.

  2. Too many/few submissions: As this is a new track, it may be difficult to predict the volume of submissions. The fact that there are currently many independent blog posts on the web is a good indicator that there will be positive interest. To get a better estimate of the volume of potential submissions, we intend to leverage social media to gauge the interest of the ML community in such a track; this will allow us to gather a large enough reviewing committee.

  3. Reviewing: Once again as this is a new track, it may be unclear how to judge blog posts during a review process. We will recruit a large reviewing committee and define clear guidelines for the reviewing process. Our primary focus will be on the originality of the perspective and the novelty of the ideas, insights, and experiments. For instance, posts that reuse less content from the original paper (results, direct quotes) will be scored more favourably than those that use more.

  4. Too many posts on the same paper: We may mitigate this by only selecting a small numbers of blog posts on the same paper. This could actually be a strength since this can encourage discussion and highlight different perspectives on the same work. Moreover, we could explicitly state that we will have this hard limit (e.g., accepting a maximum of 3 blog posts on the same paper) to entice researchers to submit blog posts on papers that have less visibility.

Related Initiatives

We mainly address our difference with respect to Distill, the ML Retrospectives Workshop, a Tutorial Track, and other workshops discussing alternative formats for publications.

Distill.

Created in 2016, Distill is an online scientific journal based on blog post publications. We address our differences with respect to Distill:

  • Visualizations: Blog posts should take advantage of the fact that they’re not paperbound, and use innovative visualisations. But the process of creating the intricate, dynamic visualisations associated with Distill posts is a daunting for most authors. Creating blog posts should be more easily accessible to newer authors and researchers. Sometimes, being able to embed videos and gifs is enough.

  • Content: Distill does not target the same type of content as our track. Distill aims at presenting new research, and at making this research more accessible. We want our blog post track to incentivize researchers to revisit and discuss on other researcher’s works, in a more natural way than scientific papers allow. Such a practice would undoubtedly be useful for the community, both as a short-term “test of time”, and also as a way to extract the key ideas from lengthy articles.

  • Limited adoption by the community: we believe that since Distill is not associated with a big conference track, its widespread adoption is hindered. This lack of association confines it to a small subset of the community that is already familiar with blog posts.

  • Leveraging the momentum of the conference: Distill describes itself as a scientific journal. A large amount of the publications in the ML community are conference papers. A blog post track that follows conferences would be better suited to follow the pace of the community.

ML-Retrospective Workshop.

A recurrent workshop in the ML community is the ML Retrospectives Workshop (NeurIPS 2019, 2020 and ICML 2020). This workshop is a venue for researchers to talk about their previous work in a more open and transparent way. More precisely, emphasis has recently been put on addressing:

  • Flaws or mistakes in the paper’s methodology

  • Limitations in the applicability of the work

  • Changes in understanding or intuition

We share the ultimate goal of “making research more human”, but with a completely different format. We believe that the constraint to write about someone else’s work using natural language will channel fruitful discussions and provide more visibility to previously published papers.

Tutorial Track.

We believe that our proposed blog post track differentiates itself from a tutorial track because tutorials operate at different scales. On the one hand, a tutorial regarding a whole topic (e.g. GANs, adversarial examples, Random matrix theory in ML) contains a long talk, slides, and potentially exercises to get familiar with the topics. It is usually made by a team of expert researchers on the topic. On the other hand, the call for blog posts we propose focuses on a single publication. It regards a single paper that can concern a more precise and recent topic (e.g., a specific paper that addresses mode collapse on GANs, a novel technique to perform adversarial training, etc.) and could be written by a single researcher (once again making it more accessible to junior researchers).

Previous workshops on rethinking publication formats.

Recently, the Rethinking ML Papers Workshop at ICLR 2021 fuelled the discussion (see references therein for related past workshops). The presenters discussed the importance of accessibility, web demonstrations, visualization and blog posts (among others). One particularly related discussion was the talk by Lilian Weng (time=4h25mins) on the usefulness of blog posts to get up-to-date with the field of ML.

In alignment with these initiatives, this new track is another step in the direction of making research more human.

Bibliography

[Littman, 2021] Michael L Littman. Collusion rings threaten the integrity of computer science research. Communications of the ACM, 2021.

[Tran et al., 2020] David Tran, Alex Valtchanov, Keshav Ganapathy, Raymond Feng, Eric Slud, Micah Goldblum, and Tom Goldstein. An open review of openreview: A critical analysis of the machine learning conference review process. arXiv, 2020.

[Lin et al., 2020] Hsuan-Tien Lin, Maria-Florina Balcan, Raia Hadsell, and Marc’Aurelio Ranzato. What we learned from neurips2020 reviewing process. Medium https://medium.com/@NeurIPSConf/what-we-learned-from-neurips-2020-reviewing-process-e24549eea38f, 2020.

[Brown and Woolston, 2018] Eryn Brown and Chris Woolston. Why science blogging still matters. Nature, 2018.

[Halmos 1957] Paul R Halmos. Nicolas bourbaki. Scientific American, 1957.

[Bourbaki, 1939] Nicolas Bourbaki. Elements of mathematics. Éditions Hermann, 1939.

Broadening Our Call for Participation to ICLR 2022

By ICLR 2022 Diversity, Equity & Inclusion Co-Chairs, Rosanne Liu and Krystal Maughan

In 2022, in an effort to broaden the diversity of the pool of participants to ICLR 2022, we are starting a program specifically assisting underrepresented, underprivileged, independent, and particularly first-time ICLR submitters. We hope this program can help create a path for prospective ICLR authors—who would not otherwise have considered participating in, or working on a submission to ICLR—to join the ICLR community, find project ideas, collaborators, mentorship and computational support throughout the submission process, and establish valuable connections and first-hand training during their early career.

Our goal is to provide underrepresented minorities, especially first-timers, and independent researchers, a taste of first-hand research experiences within a community, and a clear target to work towards. For that we need experienced researchers to join this program to provide starter ideas, ongoing feedback, lightweight mentorship, all the way to fully engaged collaboration. 

To this end, we are starting the CoSubmitting Summer (CSS) program that will begin today on August 11th and will run for an intensive 8 weeks until the October 5th submission deadline for the ICLR 2022, with an option to extend for 4 more months until close to the ICLR 2022 conference date. Successful projects will be showcased in a dedicated CSS workshop at ICLR, with an option that some of them, completed within 8 weeks, will be submitted to ICLR to be considered as full papers.

Throughout the program, participants, both first-time and experienced submitters, will gather in groups online and work together to discuss and, if they so choose, collaborate on a paper for submission, based on aligned topics of research interest. 

There are five phases of the CoSubmitting Summer (CSS) program:

  • Phase 1 (August 11 – August 25): idea gathering and proposal writing. This phase has no gatekeeping. Anyone who’s interested can join our Slack, brainstorm a research topic, form an initial team, and write a proposal about the research project. The only requirement is that to be qualified for the ensuing phases, each proposal should contain at least one teammate that comes from a historically disadvantaged, underrepresented, or underprivileged background. Independent researchers (i.e. those without a research employment or formal research affiliation) and first-time submitters are also considered qualified. Meanwhile, experienced researchers are asked to contribute starter ideas in the Request-for-Plot (RFP) format, to help researchers get started. Further details on the proposal will be shared on the CSS Slack.

  • Phase 2 (August 25 – August 30): team selection and initial mentorship matching. After a light review, mainly to ensure that proposals meet our minimum diversity requirement, qualified proposals are selected and assigned a CSS team number. CSS teams will be qualified to receive starter computational support and a stipend to cover project needs (e.g. travel expenses for in-person meetings). CSS teams are committed to report progress and present intermediate results at our weekly centralized research meetings for three consecutive weeks. Experienced submitters can join this stage to take an overview of all existing teams and projects and choose to give feedback, help shape, or officially join a team to help with the submission.

  • Phase 3 (August 30 – October 5, with an option to extend to April next year): decentralized project meetings within teams. Each team can freely decide its meeting cadence and working style. No global control is necessary, after this point, although global support remains open. 

  • Phase 4 (October 5): submit to ICLR 2022!

  • Phase 5 (November 2021 – April 2022): rest, relax, and regroup to submit to the CSS workshop.

We recruit two types of participants to engage in this program:

  • Prospective submitters: those who have never submitted to ICLR or similar conferences before. We expect them to lead or co-lead this submission, as this gives them the necessary first-hand experience of research. 
  • Experienced submitters: those who have experiences submitting to ICLR or similar venues. We expect them to serve as allies, mentors, and/or supporting collaborators in this program. They can join as late as Phase 2 or 3 in the program.

We also recruit volunteers (can be either prospective or experienced submitters) that help us ensure a smooth running of the program.

Our intention is for these groups to engage in thoughtful discussions and diverse perspectives, while providing opportunities for collaboration.

This is our first year in running this program. If you are interested in participating in this opportunity, please join our Slack (essential if you are joining Phase 1), and/or fill out this form (more for those who are more comfortable communicating via email, and/or those that are joining later phases, although Phase 1 participants are more than welcome to fill the form too). 

ICLR supports efforts to increase diversity, inclusion and the participation of traditionally underrepresented individuals in STEM. We strongly encourage participants from these groups to apply.

We thank Katja Hofmann, Emtiyaz Khan, Sebastian Ruder, Pablo Samuel Castro, Sara Hooker, Jane Wang and Marc Deisenroth for feedback on earlier drafts of this post.