By the Blog Post Track chairs: Bubeck, Sebastien, Microsoft; Dobre, David, Mila; Gauthier, Charlie, Mila; Gidel, Gauthier, Mila; Vernade, Claire, DeepMind
The Machine Learning community is currently experiencing a
and a reviewing crisis [Littman, 2021]. Because of the highly competitive and noisy
reviewing process of ML conferences [Tran et al., 2020], researchers have an incentive to
oversell their results, slowing down the progress and diminishing the
integrity of the scientific community. Moreover with the growing number
of papers published and submitted at the main ML conferences [Lin et al., 2020], it has
become more challenging to keep track of the latest advances in the
Blog posts are becoming an increasingly popular and useful way to talk
about science [Brown and Woolston, 2018]. They offer substantial value to the scientific community
by providing a flexible platform to foster open, human, and transparent
discussions about new insights or limitations of a scientific
publication. However, because they are not as recognized as standard
scientific publications, only a minority of researchers manage to
maintain an active blog and get visibility for their efforts. Many are
well-established researchers (Francis Bach,
Ben Recht, Ferenc
Weng) or big corporations that
leverage entire teams of graphic designers designer and writers to
polish their blogs (Facebook AI,
OpenAI). As a result, the incentives for
writing scientific blog posts are largely personal; it is unreasonable
to expect a significant portion of the machine learning community to
contribute to such an initiative when everyone is trying to establish
themselves through publications.
Our goal is to create a formal call for blog posts at ICLR to
incentivize and reward researchers to review past work and summarize the
outcomes, develop new intuitions, or highlight some shortcomings. A very
influential initiative of this kind happened after the second world war
in France. Because of the lack of up-to-date textbooks, a collective of
mathematicians under the pseudonym Nicolas Bourbaki [Halmos 1957], decided to start a
series of textbooks about the foundations of mathematics [Bourbaki, 1939].
In the same vein, we aim at providing a new way to summarize scientific knowledge in the ML community.
Our Idea: Blog post Conference Track
Due to the large diversity of topics that can be discussed in a blog
post, we decided to restrict the range of topics for this call for blog
posts. We identified that the blog posts that would bring to most value
to the community and the conference would be posts that distill and
discuss previously published papers.
Call for blog posts on papers previously published at ICLR
The call for blog post would take the following form:
Write a post about a paper previously published at ICLR, with the
constraint that one cannot write a blog post on work that they have
a conflict of interest with. This implies that one cannot review
their own work, or work originating from their institution or
company. We want to foster productive discussion about ideas, and
prevent posts that intentionally aim to help or hurt individuals or
Blogs will be peer-reviewed (double-blind, see
for quality and novelty of the content: clarity and pedagogy of the
exposition, new theoretical or practical insights,
reproduction/extension of experiments, etc.
The posts will be published under a unified template (see
and hosted on the conference website or our own Github page.
Positive Impact for the Community
We believe having this call for blog posts as a conference track would
increase the posts’ visibility, impact, and credibility, while
simultaneously providing benefits to the conference.
Adoption: we think that, with the conference’s stamp, such a
format will be more broadly recognized and adopted by the community.
Accessibility: maintaining a blog is time consuming , and requires
many blog posts to gain a stable following. By allowing researchers
to publish a single post, we will permit occasional blog writers to
publish their ideas, something that is relatively impossible right
now. Moreover, it will make this format accessible to more
independent/junior blog writers that do not have a company or a
research lab to support them.
Synchronization: the fast evolving field of ML advances at the
paces of its conferences. By following the same pace the blog posts
will add value and momentum to the conference. It will benefit from
the same advantages of conferences with respect to scientific
journals: faster publication process and cross-fertilization of
Positive Impact for the Conference
We develop the potential positive impact of a blog post track for the
Increases the value of the papers submitted to ICLR: blog posts will
discuss previously published papers, thus increasing their
visibility and quality.
Incentivizes researchers to submit their best research to ICLR: high
quality work will likely get highlighted in future years in a blog
Improves reproducibility and transparency: the blog post track will
identify and publicly document pitfalls and "tricks" that were not
clearly communicated in the original publication.
Provides a scientific value by itself: such blog posts will
reproduce and extend results of previously published papers. They
will distill important theoretical and practical ideas improving
their adoption and impact.
Tests of time: this track will provide a sort of crowd-sourced test
of time at a shorter timescale than the current test of times
Promotes accessibility: because many of this track’s blog posts will
vulgarize past content, this track will make the conference broadly
more accessible (to students, non-natives, and, more generally,
non-experts in the field).
Our goal is to avoid heavily engineered, professionally-made
blog-posts—Such as the “100+ hours” mentioned as a standard by the Distill
guidelines—to entice ideas and clear writing rather than dynamic
As a result, we restrict submissions to the Markdown format. We believe
this is a good trade-off between complexity and flexibility. Markdown
enables users to easily embed media such as images, gifs, audio, and
video as well as write mathematical equations using MathJax, without
requiring users to know how to create HTML web pages. This (mostly)
static format is also fairly portable; users can download the blog post
without much effort for offline reading or archival purposes. More
importantly, this format can be easily hosted and maintained through
A full copy of the track’s blogs will always be publicly available as a GitHub repository.
The process for creating and submitting a blog post is as follows:
Entrants will fork this repository and make their fork private.
Failure to do so will result in the submission being rejected, as it
breaches the double-blind review process.
Users will modify their fork as they see fit; they will add their post
along with any media files it might require. Since this is a full fork,
they will be able to view their own copy of the blog. This means that
they will be able to see exactly how their post will look and behave
on the main website.
Once completed, entrants will anonymize their blog post (i.e. strip their
name, affiliation, etc).
Entrants will download a ZIP of their anonymized fork (see figure
below), and submit the ZIP to our OpenReview venue.
- Once accepted, entrants will de-anonymize their post, make their fork
public again, and make a Pull Request on Github from their fork to the
main blog, allowing us to pull in their new blog post in a transparent
Once the submission period has ended, the GitHub repository of our track will
be temporarily made private for the duration of the conference, allowing the
conference to host the website. After the conference, the GitHub repository will
be made public again to allow viewers to fork and download its contents.
The potential Pitfalls of our Blog Post Track
In this section we identify potential issues arising with such a track
and explain how to mitigate them:
Adversarial Blog Posts: Since the guidelines are to write a blog
post on a previously published paper, one may expect some researcher
to try to use bad faith arguments to criticize a concurrent paper
through one of these blog post. We do not think this will happen,
because these blog posts will be public and thus researchers would
discredit themselves by using bad faith arguments.
Too many/few submissions: As this is a new track, it may be
difficult to predict the volume of submissions. The fact that there
are currently many independent blog posts on the web is a good
indicator that there will be positive interest. To get a better
estimate of the volume of potential submissions, we intend to
leverage social media to gauge the interest of the ML community in
such a track; this will allow us to gather a large enough reviewing
Reviewing: Once again as this is a new track, it may be unclear
how to judge blog posts during a review process. We will recruit a
large reviewing committee and define clear guidelines for the
reviewing process. Our primary focus will be on the originality of
the perspective and the novelty of the ideas, insights, and
experiments. For instance, posts that reuse less content from the
original paper (results, direct quotes) will be scored more
favourably than those that use more.
Too many posts on the same paper: We may mitigate this by only
selecting a small numbers of blog posts on the same paper. This
could actually be a strength since this can encourage discussion and
highlight different perspectives on the same work. Moreover, we
could explicitly state that we will have this hard limit (e.g.,
accepting a maximum of 3 blog posts on the same paper) to entice
researchers to submit blog posts on papers that have less
We mainly address our difference with respect to
Distill, the ML Retrospectives
Workshop, a Tutorial Track, and
other workshops discussing alternative formats for publications.
Created in 2016, Distill is an online scientific
journal based on blog post publications. We address our differences with
respect to Distill:
Visualizations: Blog posts should take advantage of the fact that
they’re not paperbound, and use innovative visualisations. But the
process of creating the intricate, dynamic visualisations associated
with Distill posts is a daunting for most authors. Creating blog
posts should be more easily accessible to newer authors and
researchers. Sometimes, being able to embed videos and gifs is
Content: Distill does not target the same type of content as our
track. Distill aims at presenting new research, and at making this
research more accessible. We want our blog post track to incentivize
researchers to revisit and discuss on other researcher’s works, in a
more natural way than scientific papers allow. Such a practice would
undoubtedly be useful for the community, both as a short-term “test
of time”, and also as a way to extract the key ideas from lengthy
Limited adoption by the community: we believe that since Distill
is not associated with a big conference track, its widespread
adoption is hindered. This lack of association confines it to a
small subset of the community that is already familiar with blog
Leveraging the momentum of the conference: Distill describes
itself as a scientific journal. A large amount of the publications
in the ML community are conference papers. A blog post track that
follows conferences would be better suited to follow the pace of the
A recurrent workshop in the ML community is the ML Retrospectives
Workshop (NeurIPS 2019, 2020 and
ICML 2020). This workshop is a venue for researchers to talk about their
previous work in a more open and transparent way. More precisely,
emphasis has recently been put on addressing:
Flaws or mistakes in the paper’s methodology
Limitations in the applicability of the work
Changes in understanding or intuition
We share the ultimate goal of “making research more human”, but with a
completely different format. We believe that the constraint to write
about someone else’s work using natural language will channel fruitful
discussions and provide more visibility to previously published papers.
We believe that our proposed blog post track differentiates itself from
a tutorial track because tutorials operate at different scales. On the
one hand, a tutorial regarding a whole topic (e.g. GANs, adversarial
examples, Random matrix theory in ML) contains a long talk, slides, and
potentially exercises to get familiar with the topics. It is usually
made by a team of expert researchers on the topic. On the other hand,
the call for blog posts we propose focuses on a single publication. It
regards a single paper that can concern a more precise and recent topic
(e.g., a specific paper that addresses mode collapse on GANs, a novel
technique to perform adversarial training, etc.) and could be written by
a single researcher (once again making it more accessible to junior
Previous workshops on rethinking publication formats.
Recently, the Rethinking ML Papers
Workshop at ICLR 2021 fuelled
the discussion (see references therein for related past workshops). The
presenters discussed the importance of accessibility, web
demonstrations, visualization and blog posts (among others). One
particularly related discussion was the talk by Lilian Weng
on the usefulness of blog posts to get up-to-date with the field of ML.
In alignment with these initiatives, this new track is another step in
the direction of making research more human.
[Littman, 2021] Michael L Littman. Collusion rings threaten the integrity of computer science research. Communications of the ACM, 2021.
[Tran et al., 2020] David Tran, Alex Valtchanov, Keshav Ganapathy, Raymond Feng, Eric Slud, Micah Goldblum, and Tom Goldstein. An open review of openreview: A critical analysis of the machine learning conference review process. arXiv, 2020.
[Lin et al., 2020] Hsuan-Tien Lin, Maria-Florina Balcan, Raia Hadsell, and Marc’Aurelio Ranzato. What we learned from neurips2020 reviewing process. Medium https://medium.com/@NeurIPSConf/what-we-learned-from-neurips-2020-reviewing-process-e24549eea38f, 2020.
[Brown and Woolston, 2018] Eryn Brown and Chris Woolston. Why science blogging still matters. Nature, 2018.
[Halmos 1957] Paul R Halmos. Nicolas bourbaki. Scientific American, 1957.
[Bourbaki, 1939] Nicolas Bourbaki. Elements of mathematics. Éditions Hermann, 1939.