Menu

Rosanne Liu

About Rosanne Liu

Posts by Rosanne Liu:

Tiny Papers Strike Back: 2023 Reflections and 2024 Announcement

1. Philosophy: why we do this

The year 2023 will be remembered as one where unprecedented and unparalleled advances on diversity, equity and inclusion (DEI) are made, by having a major machine learning conference successfully held in the African continent, for the first time in history. According to the Closing Remark, this simple change of location (while acknowledging that changes like this are never really simple) has resulted in a more than 13-fold increase (from 16 to 261) of African-based attendees.

As DEI Chairs this year, even though the majority of work pushing the DEI frontier was done when the conference location was decided, we still carry the responsibility of furthering the status quo. In January, the chairs (Rosanne, Krystal and Tom), with the support of the General Chair, Yan Liu, after meeting weekly for a few months, launched a Tiny Papers Track, an experimental paper submission track to welcome underrepresented, and non-traditional researchers into the ICLR community. As its name suggests, the track enforces a simplified format: no more than 2 pages of main content, to encourage the sharing of early results, pilot ideas, and preliminary findings.

Like any paper submission track, this effort involves putting out a call for submission, opening a submission portal, setting up a peer review system where reviewers, area chairs (ACs), and emergency reviewers and ACs are recruited, assigned tasks and consulted for decisions.

But unlike most paper tracks, we adopt an approach that embraces the principles of affirmative action, entailing the deliberate provision of advantages and preferences to historically disadvantaged populations. To that end, we ask every submission to include at least one key author that comes from a URM background, in terms of, for example, race, gender, age, geographical location, distability, or neurodiversity status. This involves adding a required section, an URM statement, to the paper submission, with an acknowledgment sentence that can be as simple as: The authors acknowledge that at least one key author of this work meets the URM criteria of the ICLR 2023 Tiny Papers Track. This honor system of self-identification respects individuals’ privacy and is aligned with a pluralistic view of advantage, representation, and systemic bias.

2. Practice: how we did this for ICLR 2023

The initiative was brainstormed, designed, and implemented for four months, before announcing to the world on January 23, 2023, and submission portal open February 1st, and submission deadline Feb 28th.

We received 219 submissions over a wide variety of topics. 580 unique total authors were involved in making submissions, with 526 of them making 1 submission, and 54 making 2 or more submissions.

We also recruited reviewers on the same day of the announcement. In the end, a total of 173 reviewers, 42 meta-reviewers (ACs) worked with us to ensure that all paper submissions have 3 or more reviews submitted. Reviewers asked to evaluate according to the CCR: Clear, Correct, & Reproducible criteria. Instead of the traditional “accept” or “reject” decision, we also adopt a novel categorization of decisions that consist of:

  • Invite to present (notable)
  • Invite to present
  • Invite to archive
  • Invite to revise

As the names suggest, those decisions reflect whether a submission meets the quality to be presented at our Tiny Papers Showcase Day, archived, or if it requires revision. Below is the distribution of the initial decision.

For those invited to revise, after revision they are given final decisions of either “Invite to archive” or “Not invite to archive”. 34 out of 58 (58.6%) submissions were changed categories from “Invite to revise” to “Invite to archive”. In the end, 194 of 219 (88.6%) submissions were accepted to archive. All archived papers can be found at DBLP.

On May 5, 2023, the last day of ICLR 2023, we hosted Tiny Papers Showcase Day, in Kigali, Rwanda. 86 papers and their authors were invited to give an oral presentation and/or poster presentation. It was a lovely day.

3. Prospect for 2024

Post-ICLR 2023, we received lots of inquiries: “Is Tiny Papers happening again? At ICLR or elsewhere?” Today, we can finally answer affirmatively, for ICLR 2024, to be held at Vienna, Austria.

We made a few changes this year to improve the program:

  • Moved deadlines much earlier. An area for improvement for Tiny Papers 2023 was that it took too long to plan and we were left with too little time to execute. With such lessons learned, we moved this year’s deadlines months earlier. 
  • Appointed dedicated Tiny Papers Chairs who will work with DEI chairs to ensure the success of this program.
  • Shared deadlines and reviewer pool with blog post track. The blog post track has been running for 3 years and has demonstrated great success and momentum. It also has the same spirit of encouraging alternative formats of scientific publications. Therefore this year we are combining forces.

For more information, read the Call for Tiny Papers 2024. Hope to see your submissions there!

Thomas F Burns and Krystal Maughan
ICLR 2024 Tiny Papers Chairs

Rosanne Liu
ICLR 2023-2024 DEI Chair

Reflection on the DEI Initiative at ICLR 2022

750 participants, 55 proposals, 8 months, 1 workshop, 15 speakers. A new and thriving research community.

By ICLR 2022 Diversity, Equity & Inclusion Co-Chairs, Rosanne Liu and Krystal Maughan

ICLR 2022 has successfully come to an end. Having served as the DEI Chairs in the organizing committee, for the first time for both of us, we’d like to take this chance to reflect on this year’s effort—ups and downs, advances and setbacks, and most importantly, lessons and memories.

Last summer, how it all started

When we were invited to serve as the DEI chairs last summer, we were excited to make some real changes: we both have seen enough lip service, enough short-term, one-off engagements that while net positive, still didn’t amount to lasting changes. One of the central questions we asked ourselves was how we could really support underrepresented, underprivileged, and first-time submitters in a long-term capacity.

We started by first asking each other: how were you supported? How would you have liked to be supported?

More generally: how do people discover a love for research? How does someone decide to pursue research and what shapes that decision? Why do people stay in research? Is it the result of their first taste or experience in research, who they do research with, that ordains the course? What kind of support network do they have, or lack?

These were our answers.

Krystal: 

“I did not know that someone could have a career in research. I am a non-traditional student; a professor encouraged me to apply for grad school after I volunteered for the ACM Principles of Programming Languages (POPL) conference as the only non-PhD student.  Someone took a chance on me. Pivotal moments can have a lasting impact on someone’s decision to consider graduate school. These moments are a result of a field’s environment (whether positive or negative), access to opportunity, and some degree of luck. I want to minimize the need for that degree of luck and make the opportunity to pursue research accessible.

Why do people stay in research? The support of peers, and multi-layered or structural levels of support (including financial support) help us to persevere. Being able to look ahead and see persons like yourself who have succeeded makes a difference. We wanted to build this regardless of each person’s history or opportunity to find this in their own community. We wanted to build a network of mentors and early career researchers who advocate for themselves, have the freedom to take risks, and celebrate their accomplishments. We wanted them to have a space where they can see that we all experience ups and downs in research, but we can bounce back, find support, and keep going.“

Rosanne: 

“I never really loved research until I met a great mentor and worked with a supportive team, and that was way past my “supposed to love research” years—I had already graduated my PhD and started an industry job. I consider myself non-traditional for that reason: I followed a “traditional” path but did not hear the calling until I’m out of the path. But that’s when I realized not all of us have to go through the same growth trajectory to become good at something. I am actually really good at this research thing even as a late bloomer. What precipitates the flip from thinking that I suck to knowing that I excel is just one positive experience. 

I can’t stand thinking how many people are held back by the lack of that one crucial positive, self-perception-altering experience, not knowing what they could be good at, had they had a little bit of initial luck.“


It is from there that we started converging to the idea of a concrete program that integrates DEI efforts with the core of scientific research, the main track of the conference, and moves from short-term, one-off engagements (for example, giving out free registrations) to long-term nurturing and support of the whole community. 

The 1st CSS Program

We announced the CoSubmitting Summer program with a blog post, last summer, the beginning of a 8-month coworking program for underrepresented researchers (definition here) to submit to ICLR or other sister venues.

The program was a success! At the ICLR 2022 Opening Session, we summarized the 1st CSS program: it attracted 750+ participants to the Slack, and an additional 20+ experienced researchers willing to serve as mentors. Together, we wrote research proposals, formed teams, sought actively mentorship actively, and had recurring meetings to shape our own research paths. The program sponsored 55 projects in total, all of them led by at least one underrepresented minority (URM), with a total amount of $17,650 spent; amount funded per project ranges from $200 to $500. After 8 months, this resulted in 15+ submitted papers, not just to ICLR, but to other sister conferences and workshops. 

Most importantly, we hope this initiative helps plant seeds for many more future scientific advancements contributed to the entire academic society. Since launched, we are aware that a number of similar initiatives followed suit, including new submission mentoring and mentor-matching programs at AISTATS and other conferences. We are really happy to see the influence of this program going further than we initially expected.

We hosted a CSS Workshop to celebrate the milestone on the last day of ICLR 2022. After the workshop, we received a heartwarming Kudoboard from attendees. Thank you!

Challenges

This was our first time launching such a large-scale experiment of open collaboration, which means the first iteration of this program is not without challenges. Some of our challenges involved keeping the momentum, resolving team conflicts, and finding solutions to reimbursing participants in locations not traditionally represented at ICLR. We also learned that our initial timeline goal of initially submitting to ICLR wasn’t as feasible; although, we had wanted this effort to make contributions to the whole research community, not just limited to ICLR, and judging by the result, a lot of CSS-initiated projects do end up getting submitted to, and accepted by other sister venues.

Looking Forward

Where do we go from here? The most important thing to ensure is not to have this program be a one-off engagement. In asking our initial questions, we found that the community we had built was not only thriving, but could be sustained long-term. Many expressed a need for long-term mentorship and support in pursuing research, which they had grown to love. When we look at who we choose to work with in our research careers, how do we pick our collaborators? Who do we give chances to? We often choose those we like working with, those we know, and those with a shared methodology. Freedom to fail then becomes a privilege for those who do not fit our shared methodology. How might an individual who comes with a different perspective complement one’s research methodology and what might collaborators learn from this process? How might an individual be given the space and opportunity to learn and fail in order to become an excellent researcher? We want to challenge researchers to collaborate with less homophily, and instead embrace difference of perspectives and intellectual curiosity to solve problems.

A final word

We would like to extend gratitude to our co-organizers for the Broadening Participation initiative and the 1st CoSubmitting Summer workshop: Tom Burns, Ching Lam Choi, Arun Raja and Aya Salama, presenters of the workshop: En-Shiun Annie Lee, Sarubi Thillainathan, Edward Elson Kosasih, Mats L. Richter, Emily McMilin, Andrija Djurisic, Raza Hashmi, Qiang Li, Fatemeh Siar, Laura Fee Nern, Edwin Arkel Rios, Nuredin Ali, Krupal Shah, Connor McCurley, Peiyuan Zhou and Tian H Teh, as well as all the participants of the initiative, and the organizing committee of ICLR 2022, especially Katja Hofmann, Feryal Behbahani and Andrea Brown who graciously and tirelessly supported us.

Broadening Our Call for Participation to ICLR 2022

By ICLR 2022 Diversity, Equity & Inclusion Co-Chairs, Rosanne Liu and Krystal Maughan

In 2022, in an effort to broaden the diversity of the pool of participants to ICLR 2022, we are starting a program specifically assisting underrepresented, underprivileged, independent, and particularly first-time ICLR submitters. We hope this program can help create a path for prospective ICLR authors—who would not otherwise have considered participating in, or working on a submission to ICLR—to join the ICLR community, find project ideas, collaborators, mentorship and computational support throughout the submission process, and establish valuable connections and first-hand training during their early career.

Our goal is to provide underrepresented minorities, especially first-timers, and independent researchers, a taste of first-hand research experiences within a community, and a clear target to work towards. For that we need experienced researchers to join this program to provide starter ideas, ongoing feedback, lightweight mentorship, all the way to fully engaged collaboration. 

To this end, we are starting the CoSubmitting Summer (CSS) program that will begin today on August 11th and will run for an intensive 8 weeks until the October 5th submission deadline for the ICLR 2022, with an option to extend for 4 more months until close to the ICLR 2022 conference date. Successful projects will be showcased in a dedicated CSS workshop at ICLR, with an option that some of them, completed within 8 weeks, will be submitted to ICLR to be considered as full papers.

Throughout the program, participants, both first-time and experienced submitters, will gather in groups online and work together to discuss and, if they so choose, collaborate on a paper for submission, based on aligned topics of research interest. 

There are five phases of the CoSubmitting Summer (CSS) program:

  • Phase 1 (August 11 – August 25): idea gathering and proposal writing. This phase has no gatekeeping. Anyone who’s interested can join our Slack, brainstorm a research topic, form an initial team, and write a proposal about the research project. The only requirement is that to be qualified for the ensuing phases, each proposal should contain at least one teammate that comes from a historically disadvantaged, underrepresented, or underprivileged background. Independent researchers (i.e. those without a research employment or formal research affiliation) and first-time submitters are also considered qualified. Meanwhile, experienced researchers are asked to contribute starter ideas in the Request-for-Plot (RFP) format, to help researchers get started. Further details on the proposal will be shared on the CSS Slack.

  • Phase 2 (August 25 – August 30): team selection and initial mentorship matching. After a light review, mainly to ensure that proposals meet our minimum diversity requirement, qualified proposals are selected and assigned a CSS team number. CSS teams will be qualified to receive starter computational support and a stipend to cover project needs (e.g. travel expenses for in-person meetings). CSS teams are committed to report progress and present intermediate results at our weekly centralized research meetings for three consecutive weeks. Experienced submitters can join this stage to take an overview of all existing teams and projects and choose to give feedback, help shape, or officially join a team to help with the submission.

  • Phase 3 (August 30 – October 5, with an option to extend to April next year): decentralized project meetings within teams. Each team can freely decide its meeting cadence and working style. No global control is necessary, after this point, although global support remains open. 

  • Phase 4 (October 5): submit to ICLR 2022!

  • Phase 5 (November 2021 – April 2022): rest, relax, and regroup to submit to the CSS workshop.

We recruit two types of participants to engage in this program:

  • Prospective submitters: those who have never submitted to ICLR or similar conferences before. We expect them to lead or co-lead this submission, as this gives them the necessary first-hand experience of research. 
  • Experienced submitters: those who have experiences submitting to ICLR or similar venues. We expect them to serve as allies, mentors, and/or supporting collaborators in this program. They can join as late as Phase 2 or 3 in the program.

We also recruit volunteers (can be either prospective or experienced submitters) that help us ensure a smooth running of the program.

Our intention is for these groups to engage in thoughtful discussions and diverse perspectives, while providing opportunities for collaboration.

This is our first year in running this program. If you are interested in participating in this opportunity, please join our Slack (essential if you are joining Phase 1), and/or fill out this form (more for those who are more comfortable communicating via email, and/or those that are joining later phases, although Phase 1 participants are more than welcome to fill the form too). 

ICLR supports efforts to increase diversity, inclusion and the participation of traditionally underrepresented individuals in STEM. We strongly encourage participants from these groups to apply.

We thank Katja Hofmann, Emtiyaz Khan, Sebastian Ruder, Pablo Samuel Castro, Sara Hooker, Jane Wang and Marc Deisenroth for feedback on earlier drafts of this post.