SC Reproducibility Reviewer FAQ

Please also review the Author FAQ

Q. What are my responsibilities when reviewing a submission that has an artifact description appendix, an artifact evaluation appendix, or both?

A. Reviewers are expected to read any appendix that is submitted with a paper. The information in these appendices is often included in any well-written paper. By collecting this information in a uniform way, we hope that the review process will be more effective without increasing the burden on reviewers.

Q. Must I confirm that links in an appendix work?

A. Confirming links in an appendix is not required of every reviewer. Volunteers from the review committee will be sought to validate links provided in appendices. Link validation is required as part of the ACM Artifacts Available badging process.

Q. Must I evaluate the code and results associated with an artifact evaluation appendix?

A. Code and results evaluations associated with an AE appendix are not required of every reviewer. Volunteers from the review committee will be sought to evaluate code and results associated with these appendices. Code and results evaluations are required as part of the ACM Artifacts Evaluated badging process.

Q. How should appendices be used as part of the review process?

A. The absence or poor quality of an appendix cannot be used to categorically disqualify a submission. Furthermore, inaccessibility of artifacts such as source code is not sufficient reason to disqualify a submission. However, during the review process, the presence and quality of appendices may earn one submission higher ranking than another of otherwise similar quality.