Peer Review Demystified: What, Why, and How

Learnings as an AI & Robotics Associate Editor with 100 Peer Reviews

I will share what I have learnt about the academic peer review process through a personal journey from a hesitant reviewer to an Associate Editor for the IEEE Robotics and Automation Letters (Impact Factor 4.6).

While most traditional science and engineering publications require prior publication experience and academic credentials to serve as reviewers, machine learning and data science might be an exception. A significant driver of the widespread adoption and use of data science has been open-source projects and repositories. Many influential contributors to open-source data science are not always published researchers but possess deep knowledge of the field through practice and experimentation. Additionally, formal academic degrees in machine learning have only existed for a few years, and many current researchers come from diverse backgrounds. I, for example, have a background in Mechanical Engineering.

With the above in mind, I hope that if you are a machine learning practitioner who is curious about the review process and wants to get involved, this article should provide some value.

Table of Content

· My Story
· What is Peer Review?
∘ Shouldn’t Editorial Board Members be the Experts?
· Peer Review Process
· Why You Should Consider Peer Reviewing
· How Can You Get Involved?
∘ Tracking Peer Reviews using Web of Science
∘ Do I need to be a published researcher?
· How Much Time Does it Take?
· Conclusion
· Cold Email Template
· Disclaimer

My Story

In August 2024, I reached 100 verified peer reviews for 9 different academic journals and conferences. Although I performed my first review in 2016, it was not until mid-2022 that I truly started enjoying the process.

As a graduate student (2015–2020), I never really enjoyed reviewing papers. Instead, I mostly did it as an academic obligation when my advisor asked me to do so. Furthermore, I lacked confidence in my ability to critique others’ work, given that I only had few publications under my belt.

After graduating, I found it challenging to stay up-to-date with new research. As a student, reading papers was part of the job. In industry, however, I only read the most popular papers. To stay current with the latest research, fulfill my academic responsibilities, and build a stronger research profile, I began emailing editors of various journals to express my interest in becoming a reviewer. Although I received responses from almost all the journals, only 2–3 assigned me papers initially. Over time, I started receiving review requests from journals I hadn’t contacted as well.

In late 2023, I applied to IEEE RA-L for an Associate Editor role and was eventually selected to serve in the human-robot-interaction track.

In the rest of this article, I will explain:

• the importance of peer reviewing and what the process entails,
• why you should consider reviewing for academic publications and how you can get started
• time commitment and other factors to consider

Finally, I will also share a cold email template that you can use to reach out to editors.

Although there is some controversy over the efficacy of the peer review process, I do not consider myself well-versed enough to comment on that aspect. Instead, I will focus on sharing my experiences and learnings.

What is Peer Review?

Peer review is a crucial tool that, ideally, ensures high-quality scientific work. It is a process used to evaluate the quality, validity, and relevance of research or scholarly work before it is published or accepted for presentation. This evaluation is performed by expert peers in the relevant domain. The peer review process helps ensure that published research is of high quality and contributes meaningfully to the field, maintaining academic standards and credibility.

Publication rely on a network of volunteer peer reviewers for the above. This is primarily due to two reasons:

1. Submission Volume : Academic publications may receive thousands of potential manuscripts each year. For instance, the IEEE Computer Vision and Pattern Recognition Conference (CVPR) received 11,532 submissions in 2024. Even though editorial boards of popular journals/conferences may include a few hundred members, they are far out numbered by the number of submissions. Additionally, most publications have at least 2 rounds of reviews, effectively doubling the number of reviews required.
2. Varied Domain Expertise : Although most publications have a relatively narrow scope, they still cover a vast domain of scientific knowledge within a specific field. To this end, editorial boards comprise of experts from numerous sub domains, but the nature of academic research is highly specific and it is nigh impossible for the editorial staff to have the right expertise to fairly critique every submission.

Shouldn’t Editorial Board Members be the Experts?

Yes, but the scope (or focus areas) of journals is often too broad. For example, consider the scope of the IEEE Robotics and Automation Letters (RA-L), where I serve as an Associate Editor:

publishes peer-reviewed articles that provide a timely and concise account of innovative research ideas and application results, reporting significant theoretical findings and application case studies in areas of robotics and automation.

The phrase areas of robotics and automation describes the wide variety of work the journal focuses on. This may include bio-inspired robotics biomedical robotics, field robotics, human-robot interaction, humanoid robotics, soft robotics — to name a few. In addition, the automation part may be based on machine learning, rule-based methods, or good old control theory.

Most robotics researchers specialize in a particular domain. I myself obtained my doctorate in medical robotics. Within that, I focused on physical therapy assistive robots. Within that, upper limb stroke rehabilitation. And finally within that, I explored the use of advanced deep learning and biomechanical signals for automated assistance. So although on paper I am a so called “expert” on medical robots, I do not have in-depth knowledge of say — use of deep learning for surgical robotics.

However, I do know fellow researchers and colleagues with expertise in these specialized fields. I can rely on their knowledge to provide feedback and recommendations for publications. These peers are essential to the publication process, ensuring that submissions receive informed and comprehensive evaluations.

Peer Review Process

Very briefly, the peer review process usually comprises of the following steps:

• Submission: Author submits a manuscript to a journal or conference.
• Initial Screening: Editor checks if the submission fits the journal’s scope and standards.
• Review Assignment: Editor sends the manuscript to experts (peer reviewers) in the field.
• Review: Reviewers assess the manuscript’s quality, methodology, and significance, providing feedback. They may recommend accept, request revisions, or reject the manuscript.
• Editorial Decision: Editor decides to accept, request revisions, or reject the manuscript based on feedback from multiple reviewers.
• Revisions: If needed, the author revises the manuscript and resubmits it for further review.
• Re-Review: Reviewers re-review revised manuscripts and recommend accept, request revisions, or reject. Most journals only allow a binary accept or reject at this stage. Although this varies.
• Publication: Accepted manuscripts are edited and published.

Why You Should Consider Peer Reviewing?

Peer reviewing can be a gratifying experience and a valuable way to contribute to the advancement of scientific research, even if you are not an active researcher. Here’s a breakdown of why peer review is important:

1. Academic Responsibility: If you are a researcher who publishes papers, the general guideline is to maintain a 3:1peer review-to-publication ratio. This means that for every paper you publish, you should ideally review three papers. This ratio reflects the typical practice where most publications assign three reviewers to each submission.
2. Staying Up to Date: Reviewing papers involves reading work that has not yet been published, often representing the cutting edge of your field. While you are not permitted to disclose or use results from unpublished reviews, you still gain insight into new techniques and current research trends within your area of expertise.
3. Build Research Network and Profile: Serving as a peer reviewer highlights your expertise in a particular field and is an excellent way to expand your research network. It connects you with fellow researchers globally and provides direct access to editorial board members, enhancing your professional visibility and connections.
4. Improve Paper Writing: Most journals allow you to review the feedback provided by other reviewers on the same submission. This exposure offers valuable insights into what fellow researchers consider strong versus weak papers, which can help you refine and enhance your own writing skills.
5. Green Card Criteria: The following is not legal advice and only reflects my personal experience. Please consult an immigration lawyer if you need further information.
   This is relevant if you are an immigrant in the U.S. seeking an Employment-Based (EB) Green Card. Categories such as EB1-A, EB1-B, and EB1-NIW often require “evidence of participation, either on a panel or individually, as a judge of the work of others in the same or allied academic field” as one of the criteria to demonstrate expertise. Therefore, reviewing more papers can strengthen your application and increase your chances of meeting this criteria. In fact, I myself used my peer review background as a criteria for the EB1-B Green Card.

How Can You Get Involved?

Peer reviewing might seem intimidating, but it is quite manageable and resembles the code review process. Similar to creating a pull request that needs to reviewed before merged, a manuscript needs to reviewed before it can be published.

Editors are constantly seeking peer reviewers and are often very receptive if you reach out to them. A straightforward cold email can be very effective. I will provide an email template at the end of this article for you to use.

As long as your aim is to offer unbiased feedback to help authors improve their work, you are approaching the process with the right mindset. Most editors will value and respect your contributions.

Tracking Peer Reviews Using Web of Science

I would highly recommend creating a Web of Science profile. It enables you to get your reviews verified and all in one place. It has a handy export feature that can serve as a proof of your reviewer experiences that is expected by most organizations. It also provides intersting metrics such as the average length of your reviews.

Do I need to be a published researcher?

Not necessarily. While many top journals and conferences require some publication experience, others do not. If you do not have publications and are a millennial (i.e. suffer from imposter syndrome), you can start by reviewing poster or abstract submissions for smaller, local conferences to build your profile and confidence. You can then use this as a crux to graduate to international publications as well as a filler for a lack of publication history.

Keep in mind that each manuscript is typically reviewed by 2–3 reviewers at various career stages. As a newcomer, you may offer a fresh perspective compared to more experienced researchers. Editors value all feedback, and a diverse range of viewpoints is highly beneficial.

How Much Time Does it Take?

The time commitment for peer reviewing varies and is entirely up to you. I typically limit myself to 1–2 papers per month (this includes my AE assignments). In 2023, I reviewed more frequently, but I am now more selective. You can always decline invitations if needed, as peer reviewing is a voluntary activity, and editors respect your time.

The duration of each review also varies. In my experience, reviews can take anywhere from a couple of hours to several days. If a paper is closely related to my research, I can complete it in an afternoon. However, papers that are adjacent to my field or involve complex equations can take longer, especially if they require extensive verification. Personally, I avoid papers with too many equations as I do not enjoy reading them. More pictures, less math, please!

Sometimes I do receive terrible quality papers that appear to be a waste of time. But these usually take the least amount of time to review anyway.

In summary, the time commitment varies, but you can choose the number and type of papers to review based on your preferences and availability.

Conclusion

In conclusion, the peer review process is a crucial component of academic publishing that ensures the quality and integrity of scientific research. The process, while challenging, offers significant benefits, including staying abreast of cutting-edge research, enhancing one’s academic profile, and contributing meaningfully to the scholarly community.

Ultimately, peer reviewing is not only a responsibility but also an opportunity for personal growth and professional development. It provides a platform for researchers to influence the advancement of their field, build valuable networks, and improve their own research skills. While the peer review system is not without its criticisms, it remains a vital component of the research ecosystem, fostering academic rigor and innovation.

Cold Email Template

As promised, here is the cold email template I have used in the past.

SUB: Request to serve as peer reviewer for [Publication Name]

Dear [Editor’s Name],

I hope this message finds you well.

I am writing to express my interest in serving as a peer reviewer for [Publication Name]. Currently, I am a [Your Role] at [Your Organization], with a [Bachelor’s/Master’s/Doctorate] in [Field] from [University]. My areas of expertise include [Expertise 1, Expertise 2, Expertise 3], and I have demonstrated proficiency through [briefly mention any relevant experience or achievements].

I have published articles in [Publication 1, Publication 2, Publication 3] and contributed to open-source projects such as [Project 1, Project 2] and [Blog 1, Blog 2]. I also serve as a reviewer for [Publication 1, Publication 2, Publication 3].

You can find more information about my work on my [Google Scholar and/or GitHub] profile, and I have attached my resume for your reference.

I am confident that my background and expertise make me a suitable candidate for reviewing submissions, particularly in areas related to [List Areas of Interest]. I would be honored to contribute to [Publication Name] and support the advancement of research in these fields.

Thank you for considering my application. I look forward to your response.

Best regards,

[Your Full Name]
[Your Contact Information]

Disclaimer

ChatGPT was used as a proof-reading tool for this article. Minor edits were made based on the feedback. Content was created by author.

Peer Review Demystified: What, Why, and How was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.