Menu

Carl Vondrick

About Carl Vondrick

Posts by Carl Vondrick:

Assisting ICLR 2025 reviewers with feedback

(This post is written by James Zou, Associate Program Chair)

Obtaining constructive and high-quality peer reviews at AI conferences has become increasingly challenging due to the rapidly rising volume of paper submissions. For example, ICLR experienced year-over-year submission increases of 47% in 2024 and 61% in 2025. As submission numbers grow, the demand on reviewers increases, often leading to inconsistent review quality. To help, for ICLR 2025 we are introducing a review feedback agent that identifies potential issues in reviews and provides feedback to reviewers for improvements.  

The goal of this system is to help make reviews more constructive and actionable for authors. The review feedback agent will provide suggestions on three potential categories of issues in reviews. We curated these categories by compiling public comments and evaluating reviews from previous ICLRs to identify common issues. 

The feedback areas are:

  1. Encouraging reviewers to rephrase vague review comments, making them more actionable for the authors.
  2. Highlighting sections of the paper that may already address some of the reviewer’s questions.
  3. Identifying and addressing unprofessional or inappropriate remarks in the review.

We provide an example of a comment that would be flagged by our system for each category and an example feedback below.

The feedback system will not replace any human reviewers. The agent will not write reviews or make automated edits to reviews. Rather, it will serve as an assistant, providing optional feedback that reviewers can choose to incorporate or disregard. Every ICLR submission will be assessed solely by human reviewers, and the final acceptance decisions will be made by ACs, SACs, and reviewers, as in previous ICLR conferences.

The feedback will be provided for a randomly selected subset of initial reviews to enable comparisons and assess its impact. Reviewers will have an opportunity to update their review, should they wish, before the reviews are available to the authors. Feedback will be sent to the reviewer within a few hours of submitting a review. There will be no feedback on subsequent reviewer responses, and no further interaction between the reviewer and the feedback system. Moreover, the feedback will only be visible to the reviewer and the ICLR program chairs; it will not be shared with other reviewers, authors, or ACs and will not be a factor in the acceptance decisions. We have designed the feedback system using a pipeline of multiple LLMs to minimize hallucinations and enhance the quality of the feedback. The system has been carefully tested on publicly available ICLR 2024 reviews. 

After the final paper decisions have been released, we will distribute an anonymous, voluntary survey to authors, reviewers, and ACs to gather feedback. We will carefully analyze the responses to assess the pilot system’s impact and to guide improvements for future iterations. 

Nitya Thakkar, Mert Yuksekgonul, Jake Silberg, James Zou (Associate Program Chair)
The Review Feedback Agent Team

Carl Vondrick, Rose Yu, Violet Peng, Fei Sha, Animesh Garg
ICLR 2025 Program Chairs

Extended partnership pilot with TMLR for ICLR 2025

Last year, the International Conference on Learning Representations (ICLR) hosted a select subset of papers published at the Transactions on Machine Learning Research (TMLR) as ICLR poster presentations. We have received positive feedback on this pilot program from both participants at ICLR as well as the presenters of these TMLR papers. We are thus running another year of the TMLR-to-ICLR pilot program at the upcoming ICLR (2025).

In 2024, we only considered TMLR papers with outstanding or featured certifications. This year, we will be expanding the pool of TMLR papers invited for a poster presentation at ICLR. Over the past few months we have been asking the action editor (AE) of each submission whether they find it appropriate for presentation at ICLR. For ICLR 2025, we will use this AE recommendation together with other signals, such as reviews and reviewer scores, in addition to outstanding and feature certifications, in order to invite the authors of approximately 100 TMLR papers to present their work at ICLR in 2025. For this year’s pilot, we will consider TMLR papers that have been published for at most two years prior to ICLR 2025 and were not presented at any previous ICLR or similar conference venue.

We plan to send out invitations to the authors of the selected TMLR papers later this year, with more details. 

With this continued pilot program, we hope to highlight quality work from TMLR to a broader research community and to understand better how journals and conferences can collaborate with each other to improve the publication paradigm in machine learning.

The announcement for the first pilot of TMLR at ICLR in 2024: https://blog.iclr.cc/2023/10/06/authors-of-tmlr-publications-with-featured-and-outstanding-certifications-at-iclr-2024/