Ethics Review Process for ICLR 2023
By ICLR Program Chairs and Ethics Review Chair
In the interest of transparency, below is the ethics review process we used in ICLR 2023, as major AI/ML conferences are beginning to attend to this crucial work. This write-up is also meant to help raise awareness about this important yet thorny domain. Ethics is now more urgent given the rapid pace of AI/ML research and deployment in an unprecedented range of applications, which potentially raise new or uncommon ethical issues.
1) Determining Review Lead: Ethics Committee or Program Chairs?
Ethics review often embodies blurry boundaries that intersect with policy decisions, technical domain expertise, and legal concerns. This is especially true given the dynamic landscape in AI, where new developments are rapid where there are few precedence cases to follow. In order to make our review process more streamlined, we devised a filtering criteria to do a first-order division of the work between the Program Chairs and the Ethics Committee.
General Guidelines: If a paper requires ethics discussion with ethics experts, it goes to the Ethics Committee; if it’s a matter of policy issues (e.g. plagiarism, dual submission, collusion), it goes to the program committee. In reality, some cases require discussion between the Ethics Committee and Program Chairs to clarify and address the interplay of technical details and risk estimation in addition to policy setting considerations.
Detailed Guidelines
- Plagiarism concern: the paper goes to Program Chairs
- Technology/methods/models with potential ethical biases (e.g. large language models, pre-trained models, generation models): If no deployment and no immediate harm the paper goes to Program Chairs, else Ethics Committee.
- Human subjects: If no obvious risks, the paper goes to Program Chairs (with potential help from Senior Area Chairs), else the paper goes to the Ethics Committee.
- Data usage: If no legitimate rights in using the data, then the paper goes to senior program chair, general chair and ICLR legal team
- Possible offensive language appearing in the paper: the paper goes to Ethics Committee
- Other ethical concerns that are tricky to conclude: the paper goes to Ethics Committee
The Program Chair team (4 chairs and 1 senior chair) went through 190 papers with ethics flags and adopted the guidelines above to determine which papers would be escalated to the Ethics Committee, and which would stay with the Program Chair team.
2) Ethics Review or Related Follow-Up Actions:
Ethics review or related follow-up actions were carried out by the Ethics Committee and/or Program Chairs based on 1). Comments were given out on OpenReview to discuss and clarify with authors. Suggestions were also given out on OpenReview if there are potential concerns.
At least 20 papers went through detailed review and discussions among the Ethics Committee members, where some of them the Program Chairs weighed in on technical details and policy decisions. During global calibration meetings between Program Chairs and Senior Area Chairs, papers of ethical concerns (whether there are ethics flags or not) were part of the meeting agenda to help address ethical concerns unflagged by technical reviewers.
3) New Experiment: Conditional Acceptances:
This year we tried a new experiment: For papers that are technically sound but may lack clarity on certain ethical aspects, authors had to address concerns and suggestions from the Ethics Committee before their paper status could be considered to be converted from Conditional Acceptance to Acceptance. For ICLR 2023, we had five such cases; remaining papers were either directly accepted upon clarification on OpenReview and did not require additional revisions, or were rejected due to technical merit.
Staffing and Resource Allocation Considerations: For ICLR 2023, we prioritized conducting ethics review on papers with an average score greater than 4.5 points out of the 1 to 10 point scale while global calibration of technical reviews continued. If there were papers under consideration of acceptance despite an average score lower than 4.5 points, the above ethics review process would be activated to ensure all papers accepted runs through a thorough ethics review.
Note that in addition to the primary goal of ensuring all papers follow the guidelines in ICLR’s code of conduct, we hope the ethics review process has also helped make authors, reviewers, area chairs and the community at large more aware of the on-going ethical considerations in machine learning research, so we can have more open discussions on how to handle unforeseen cases and adopt proper practices.