360-Degree Post Decision Reviews

An important part of the role of the senior editors is to measure performance of the Journal of Information Systems. There are many different ways that we can understand how well JIS is fulfilling its mission. One of the very important dimensions is to quantity the quality of the author feedback process. Our ambition is to provide a welcoming, productive, and responsive review process. This process involves senior editors, editors and reviewers as well as authors. We now have 360-degree feedback on the manuscript review process.

The concept of 360-degree feedback is widely used in human resource management. Supervisors review subordinates. Subordinates and stakeholders (internal customers) review supervisors. Recently, the US Department of Defense has rolled out 360-degree feedback across the military. The Chief of Staff of the Army Gen. Ray Odierno says “I believe that multi-dimensional feedback is an important component to holistic leader development. By encouraging input from peers, subordinates and superiors alike, leaders can better see themselves and increase self-awareness. ... The ability to receive honest and candid feedback, in an anonymous manner, is a great opportunity to facilitate positive leadership growth.”

At JIS, 360-degree feedback commences shortly after the review process ends, whether the paper is accepted or rejected. Each of the authors and reviewers and the designated editor receive targeted emails that point to a survey on Qualtrics.com. Authors answer questions on the submission process, the nature and quality of the reviews received, and support from the editor and senior editors, where appropriate. For example, authors answer questions, using a Likert scale response, such as “the feedback provided by the review team was constructive” and “the feedback provided by the review team helped me improve the manuscript.” Reviewers also answer questions that are specific to their role in the process. At the conclusion of the survey we ask a set of questions for both authors and reviewers including “How likely is it that you will accept future reviewing requests at JIS?” and “How likely is it that you will recommend to colleagues to submit their research to JIS?”

We will maintain strict confidentially on review responses. As the introduction to the survey notes, “All your responses will be read only by ourselves as Senior Editors and confidentiality will be maintained. Your responses will be aggregated with other responses, to generate high-level performance metrics for the JIS community.”

Data from the 360-degree feedback will assist us in a variety of ways. First, it will help us to understand how well we are managing the review process. How well do authors feel that we are supporting them in the review process? Is the process timely and efficient? Are the views of authors and reviewers aligned? Are we providing appropriate guidance to editors and reviewers? Second, the surveys will provide the foundation for identifying “Outstanding Reviewers” and “Outstanding Editor,” presented at the Annual Meeting of the AAA.

Each year, as senior editors, we will produce a report for the AIS Section Research and Publications Committee as well as for the broader JIS community. We will report aggregated information from the surveys. We will then track our performance on these metrics in succeeding years.

The review process should add value on two key dimensions. First, it assists authors improve the quality and enhance the fit of their papers to the JIS mission. Second, it provides a quality assurance screen that ensures that JIS publishes high quality and relevant papers. We hope that this 360-degree feedback will enhance this value adding. 

Roger Debreceny
Senior Editor