View on GitHub

sc-reproducibility

Source website for Supercomputing Conference Series reproducibility resources

Acceptance Criteria for SC Artifact Description Appendix

Use the following tables for determining if a paper:

Table 1

Criterion Yes No
Does the article have an artifact description appendix? Continue. Stop. No award or badge.
Does the appendix have an abstract? Continue Stop. No award or badge.
Does the appendix have only an abstract and the abstract states “This paper contains no computational results.”? Go to Table 2. Go to Table 3.

Table 2 (No Computational Results)

Criterion Yes No
Does the article have computational results? Stop. No award or badge. Stop. Eligible for award but not badge.

Table 3 (With Computational Results)

Criterion Yes No
Does the article have computational results? Continue. Stop. No award or badge.
Does the appendix effectively use its two pages[1] to address the checklist items? Continue. Stop. No award or badge.
Does the appendix provide links to a publicly-available[2] location where the software and data for the paper’s computational experiments can be found? Stop. Eligible for award. Badge earned. Stop. Eligible for award but not badge.

[1] The determination of “effectively use” is the responsibility of the reviewer. We rely on the expertise of the reviewer to assess if the author has put in a good faith effort to use the page space to address the required elements of this appendix.

[2] Public availability is still an emerging definition. Eventually publishers will define this criterion rigorously. For now, we will admit artifacts that are available on known public hosting platforms such as GitHub, Gitlab, Bitbucket or SourceForge. Not acceptable are private or departmental platforms that at are less likely to be maintained in the future.