Mandy Wang

By ICLR 2023 Program Chair Mengdi Wang

We are delighted to announce the recipients of the ICLR 2023 Outstanding Paper Awards!

First, we would like to thank the members of the ICLR community, including reviewers, area chairs, and senior area chairs, who provided valuable discussions and feedback to guide the award selection. In addition, we would like to extend a special thanks to the Outstanding Paper Award Selection Committee for generously sharing their time and expertise for making the final selection.

Outstanding Paper Awards

The following four papers are chosen as recipients of the Outstanding Paper Award, due to their excellent clarity, insight, creativity, and potential for lasting impact. Additional details about the paper selection process are provided below. 

Universal Few-shot Learning of Dense Prediction Tasks with Visual Token Matching  

Donggyun Kim, Jinwoo Kim, Seongwoong Cho, Chong Luo, Seunghoon Hong

The paper presents a pipeline for few-shot learning on dense prediction tasks, such as semantic segmentation, depth estimation, edge detection, and keypoint detection. It proposes a simple unified model that can handle all of these dense prediction tasks, and it features several key innovations. This work has the potential to inspire further advancements in dense prediction and the individual ideas presented, such as visual token matching and episodic meta-learning, could be applied to related multi-task learning problems. Quoted from an award committee member: “I am thrilled to see this paper receive the recognition it deserves and would like to emphasize its significance by endorsing it for an outstanding paper award.”

Rethinking the Expressive Power of GNNs via Graph Biconnectivity 

Bohang Zhang, Shengjie Luo, Liwei Wang, Di He

This paper studies biconnectivity properties as expressivity metrics of GNNs. Realizing that only one family of GNNs is expressive enough, the authors propose a new algorithm that leverages inter-node distance, and demonstrate it in both synthetic and real-world data. The committee thinks that this work is innovative, interesting and technically solid. Quoted from an award committee member: “The problem of biconnectivity also seems important and has a wide range of potential applications in theory and other practical problems. “

DreamFusion: Text-to-3D using 2D Diffusion 

Ben Poole, Ajay Jain, Jonathan T. Barron, Ben Mildenhall

The paper presents an effective way to generate 3D models based on text without requiring 3D models as training data. The key idea of the paper is to leverage a diffusion model that was trained for text-based image generation, and to generate a 3D model by back propagating the error signal that is normally used to train the diffusion model into a neural radiance field of the 3D model. Quoted from an award committee member: “The approach is a clever, elegant combination of state-of-the-art components for image generation and 3D modeling that works surprisingly well in practice. The work described in the paper has already inspired a variety of follow-up work, including effective approaches for text-based 3D video generation.” 

Emergence of Maps in the Memories of Blind Navigation Agents 

Erik Wijmans, Manolis Savva, Irfan Essa, Stefan Lee, Ari S. Morcos, Dhruv Batra

The paper presents novel, interdisciplinary work that builds on insights in cognitive science and machine learning. It develops insights on the representations learned by artificial “blind” navigating agents and the ways in which these learned representations may enable effective navigation. The authors present a rigorous investigation that systematically untangles relevant research questions. Quoted from a member of the award committee: “I hope that the demonstrated rigor in building up an argument towards answering questions about learned representations will inform future studies across the ICLR community.”

Outstanding Paper Honorable Mentions

Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning 

Zeyuan Allen-Zhu, Yuanzhi Li

The paper tries to understand distillation by considering a new theoretical perspective. The insight from the authors is that under a natural multi-view structure, without distillation a neural network can be trained to depend only on part of the features, but with distillation, this problem can be alleviated. The authors also presented simplified examples for which this can be proved.  The intuition uncovered by the paper is pointed out as – “a real contribution of the paper” by members of the award committee. The committee believes this is a very interesting theoretical explanation, leading to better understanding of the effectiveness of distillation.

Mastering the Game of No-Press Diplomacy via Human-Regularized Reinforcement Learning and Planning  

Anton Bakhtin, David J Wu, Adam Lerer, Jonathan Gray, Athul Paul Jacob, Gabriele Farina, Alexander H Miller, Noam Brown

The paper’s broad topic is the development of algorithms for turn-based, multistage, multiplayer games. The paper proposes a self-play like strategy to figure out a good “equilibrium” and tests the algorithms on a complex multiplayer board game popular among human players. The new idea is to merge equilibrium seeking strategies with behavior cloning: If an algorithm needs to be competitive while interacting with human players, it makes sense for the algorithm to use strategies that human players will recognize. Quoted from a member of the award committee: “Given that multiplayer games are quite poorly understood, the fact that the algorithm developed is shown to be competitive with humans is very promising.”

On the duality between contrastive and non-contrastive self-supervised learning 

Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, Yann LeCun

The source of the unreasonable effectiveness of self-supervised learning methods is under-interrogated, and in particular there exist distinct families/lineages of methods that appear to have nothing in common but perform similarly in practice. This paper shows that they do have something non-obvious in common. Quoted from a member of the paper award committee: “ I believe this should be read by anyone seriously working on unsupervised learning.”

Conditional Antibody Design as 3D Equivariant Graph Translation 

Xiangzhe Kong, Wenbing Huang, Yang Liu

The paper studies antibody design given the antigen, the antibody’s heavy chain and light chain. Authors formulate the problem as a conditional graph generation task with the goal of generating the amino acids in CDRs and their 3D conformation for 4 heavy atoms in each residual amino acid. There are many technical contributions in this paper which the committee finds interesting and potentially useful, and the method shows significant improvement over prior methods in a comprehensive set of experiments. The committee believes that follow-up wetlab experiments for the antibody designed using this algorithm, which could be outside the scope of this paper, are promising.

Disentanglement with Biological Constraints: A Theory of Functional Cell Types 

James C. R. Whittington, Will Dorrell, Surya Ganguli, Timothy Behrens

This work shows an interesting connection between machine learning and neuroscience. This paper introduced biologically inspired constraints, nonnegativity and energy efficiency, and mathematically proved these constraints lead to linear network disentanglement. It also shows empirically that the same constraints are effective for non-linear cases. It contains other contributions summarized at the end of the introduction section, which are not trivial either. The committee believes this work has the potential to give a big impact on future research in diverse fields of machine learning.

Selection Process

The Outstanding Paper Committee determined a selection process with the goal of identifying a collection of outstanding papers that represent the breadth of excellent research being conducted by the ICLR community.

The committee began with an initial pool of 67 papers including all papers that were explicitly nominated for an award by area chairs or senior area chairs as well as all papers receiving top review scores. The committee used two phases of down-selection. During Phase 1, each paper was assigned to one primary reader to determine if the paper should move to Phase 2, and in addition, committee members optionally endorsed other papers that were outside their assignments to move to Phase 2. We had ~20 shortlisted papers after Phase 1. During Phase 2, each paper was assigned with additional secondary readers for ranking and comparing with other candidate papers. After Phase 2, each remaining candidate paper has received at least three independent evaluations. Then we shortlisted 9 papers based on the rankings and endorsement notes shared by the committee members from Phase 1 and 2. At all phases, the committee members could read or endorse papers only if they do not have conflicts of interests, either based on the domain conflicts, or based on the personal relationship conflicts (e.g,. friends or former advisors). In order to promote honest and fair evaluations among the selection committee, the paper assignments were kept confidential so that judgments are not biased by the presence of other committee members who have conflicts of interests with some of the papers in the pool. 

Paper Award Committee

Yann Dauphin, Alex Dimakis,  Aleksandra Faust, David Warde-Farley, Michel Galley, Bohyung Han,  Shafiq Rayhan Joty, James Kwok, Mohammad Emtiyaz Khan,Peter Koniusz, Vladlen Koltun, Katja Hofmann, Lei Li, Lihong Li, Hsuan-Tien Lin, Laurens van der Maaten, Tim Rocktaeschel,  Daniel M Roy,  Suvrit Sra, Matthias Seeger, Yale Song, Csaba Szepesva, Yuandong Tian, Yair Weiss, Scott Yih,   Ming-Hsuan Yang, Tong Zhang, Denny Zhou, Jun Zhu.