Reproducibility stands as a fundamental foundation of scientific research. Reproducibility can be seen as the capacity to reproduce the findings of scientific papers and it is key for verifying the outcome of a scientific outcome. The difficulty in reproducing the experiments of scientific research has been a matter of strong debate inside the computer science and the AI communities in recent years. Ensuring the robustness and trustworthiness of scientific research utilizing computational methods and data is of utmost importance, since the inability to replicate experimental outcomes may result in various consequences, including the withdrawal of research and harm to the standing of both the authors and the conference/journal that disseminated it. Moreover, promoting reproducibility facilitates the dissemination, comparison, and adoption of research, thereby potentially augmenting the influence and increasing the prominence of scholarly articles. 

In response to the growing need for rigor and transparency in research, numerous reproducibility initiatives have emerged in the software engineering community, the supercomputing community, and more recently in the machine learning community. These initiatives are designed to oversee and streamline the governance that facilitates the assessment of the reproducibility of the submitted papers. Nonetheless, reproducibility in practice is in its infancy, with numerous barriers that must be addressed. These barriers range from the challenges and constraints imposed by current technologies to the social and governance aspects necessitating a shift in our research practices. While certain technical barriers have been previously analyzed, we revisit them in Task 4.3, alongside an analysis of the social and governance obstacles. These latter barriers represent the challenges encountered in real practice. Furthermore, we introduce a reproducibility process aimed at mitigating these barriers, a process that we presented at ECAI’23. 

This analysis of current reproducibility in practice is essential to understand the requirements that the platform tools, resources and services of the AIoD need to address. Our main objective is that our efforts are more useful for the European AI community. Overall, regarding our reproducibility analysis, our contributions encompass the following relevant points: 

• Revision and extensions of technical barriers for reproducibility. 
• Proposal of social barriers for reproducibility 
• From our findings, we propose a reproducibility process to mitigate the previous barriers.

We leverage on existing know-how in reproducibility to specify a concrete reproducibility process that has been implemented in practice in MLSys, SC23, ICPP23, TPDS, and ICPP24. The process goes beyond the existing recommendations and standardizes the contents of the documentation (i.e., how the software and data artifacts need to be documented). The process ensures that documentation of artifacts follows a standardized format, making it easier for authors to adhere to documentation practices, and for reviewers and third-party researchers to encounter more consistency in the content across submissions.

Reproducibility refers to the degree to which the repetition of an experiment yields consistent outcomes. There is no universally accepted definition of reproducibility; rather, researchers have proposed multiple conflicting definitions. Following discussions with the National Information Standards Organization (NISO), ACM tried to align its terminology and definitions with those commonly used in the broader scientific research community. Consequently, ACM proposed to unify the following terms: repeatability, reproducibility and replicability. 

Repeatability refers to the ability to execute experiments and obtain results consistently within the same team and experimental setup. Reproducibility extends the concept of repeatability to different teams or individuals conducting the experiment under similar conditions. Replicability involves conducting the experiment under different conditions or using different equipment, while still achieving comparable results. 

Consistently obtaining the same results of an experiment can be seen as a binary variable, but rather exists along a spectrum, varying in degrees of reproducibility aspects. ACM is also acknowledging it by promoting three main types of reproducibility badges. Artifacts Evaluated (the artifact has been reviewed), Artifacts Available (the artifact is openly available) and Results Validated (results can be replicated). These badges are regarded as independent in the sense that any of one, two or all can be granted to any given paper depending on review procedures developed by the journal or conference. 

The concept of reproducibility has also been studied in AI. According to Gundersen & Kjensmo, three reproducibility levels can be considered in AI: Experiment reproducibility (R1) refers to the capability of replicating results wherein an experiment yields consistent outcomes when the same AI implementation is executed; Data reproducibility (R2) refers to the ability to replicate results when an experiment is conducted using a different implementation of the AI method, yet generates identical outcomes when applied to the same dataset. Method reproducibility (R3) refers to the capacity to reproduce results when an alternative implementation of the AI method yields the same outcomes when executed on diverse datasets. 

Concerns have been raised about the reproducibility of AI and in particular of machine learning. As discussed by Jean Pinneau (Meta), reproducing many machine learning results is challenging, if not impossible, without access to critical components of the original research, such as datasets, software configurations, and controls for randomization. Despite numerous studies demonstrating this issue, it remains evident that AI researchers often do not share sufficient information to facilitate reproducibility. Mere descriptions of experiments may not suffice to demonstrate the robustness of findings. For instance, a recent survey examining 400 algorithms presented at prominent AI conferences found that only 6 percent of presentations included the algorithm’s code,

and just one third provided test datasets. Similarly, a prior survey conducted by the journal Nature in 2016 among 1,576 machine learning researchers revealed that over 70 percent encountered difficulties when attempting to replicate others’ experiments, with more than half unable to reproduce their own work. 

The actors involved in the reproducibility process, that is in the evaluation of computing artifacts typically include: 

1. Authors: The individuals or research teams who have developed and published a scientific paper or research project along with its associated computing artifacts. They are responsible for providing the necessary documentation and materials to enable others to reproduce their work. 
2. Reviewers: In the context of peer-reviewed publications, reviewers play a crucial role in assessing the reproducibility of computing artifacts. They evaluate the clarity of documentation, the availability of code and data, and the ease with which their colleagues can replicate the results. 
3. Third-party researchers: These are individuals or groups who attempt to reproduce the results of a published study using the provided computing artifacts and documentation. They do not have to be involved in the venue where the paper was published. They may or may not be directly involved with the original research but aim to validate its findings. 
4. Journal Editors and Conference Chairs: Editors and conference chairs oversee the  publication or presentation of research papers. They may play a role in enforcing reproducibility  standards and ensuring that authors adhere to guidelines for sharing computing artifacts.3.  

A software artifact (see Figure 1) is a key element for evaluating reproducibility. It comprises all the elements to reproduce an experiment, such as datasets, code, scripts. Different conferences and journals have developed reproducibility processes and guidelines for evaluating software artifacts. The computational artifact of a paper (see Figure 2) can be seen as an aggregation of other artifacts, of different types, such as datasets, code or configuration. As a whole, all the computational artifacts of a paper include all the elements that support the reproducibility of its experiments, such as software, datasets, environment configuration, mechanized proofs, benchmarks, test suites with scripts, etc. For the evaluation, artifacts must be freezed to guarantee their persistence and immutability.

Figure 1: The reproducibility computational artifacts that accompany a paper and that they allow scientists to reproduce the computational experiments.

An artifact needs to meet several properties that are essential for their evaluation, namely immutability, openness, and persistency (see Figure 2). Immutability refers to an artifact remaining unchanged. It can ensure that it remains unaltered throughout the whole evaluation process. To achieve it, authors are encouraged to use a github repository to store their code, then they need to tag the version to be submitted and, finally, they need to pair it along with Zenodo to generate a Digital Object Identifier (DOI). A DOI, standardized by the International Organization for Standardization, is a persistent identifier used to uniquely identify an artifact, and it is the mechanism that guarantees that the tagged github repository remains unchanged. Openness refers to the fact that the artifacts under evaluation can be easily accessible by standard means, such as utilizing the well-known github interfaces. Persistency implies that artifacts persist over time so that they can be revisited, reevaluated, or referenced in the future. This property is critical for maintaining a historical record and for ensuring that the findings and conclusions of the evaluation can stand the test of time. The usage of open repositories, such as Zenodo and github guarantee the three properties, but there can be other ways to achieve the same three properties.

Figure 2. Properties of a computational artifact 

Once submitted to review, an artifact needs to be immutable and it must be accessible via a permanently persistent and publicly shareable DOI on a hosting platform that supports persistent DOIs and versioning (for example, DataPort, Dryad, FigShare, Harvard Dataverse, or Zenodo). Authors should not provide links or zipped files hosted through personal webpages or shared collaboration platforms, such as Next Cloud, Google Drive, or Dropbox. Please, follow the following requirements to prepare your code and data artifacts: 

• For code artifacts, the right way of providing an immutable and persistent artifact can be achieved by means of Zenodo and FigShare. They provide an integration with GitHub to automatically generate DOIs from Git tags. Therefore, it is possible to host code using version control provided by GitHub and describe the artifact using Zenodo or FigShare. Please, observe that Git itself (or any other control versioning software) does not generate a DOI, and it needs to be paired with Zenodo or FigShare. 
• For dataset artifacts, open repositories such as IEEE DataPort provide an automatically generated DOI to the datasets uploaded. 

In addition to the documentation accompanying the code / data artifacts. If needed, README files can also be attached to the computational artifacts, either containing the same information as in the artifact description or complementing it, for instance, providing further and more detailed instructions and documentation. As a general rule, authors should try to do their best to simplify the reproducibility process, to save committee members the burden of reverse-engineering the authors’ intentions. For example, a tool without a quick tutorial is generally very difficult to use. Similarly, a dataset is useless without some explanation on how to browse the data. For software artifacts, the artifact description and the README should—at a minimum—provide instructions for installing and running the software on relevant inputs. For other types of artifacts, describe your artifact and detail how to “use” it in a meaningful way. 

Importantly, make your claims about your artifacts concrete. This is especially important if you think that these claims differ from the expectations set up by your paper. The reproducibility Committee is still going to evaluate your artifacts relative to your paper, but your explanation can help to set expectations up front, especially in cases that might frustrate the evaluators without prior notice. 

Packaging Methods 

Configuring software dependencies of artifacts can require a significant amount of time for reproducibility purposes. Therefore, to alleviate it, authors should consider one of the following methods to package the software components of their artifacts:

• Virtual Machine/Container: A virtual machine or Docker image containing the software application already set up with the right toolchain and intended runtime environment. You can share your container and virtual machine specification with GitHub. 
• Notebook: You can share your Notebook online with GitHub. Read more at: https://viso.ai/deep-learning/jupyter-notebook-for-machine-learning/ 
• For mechanized proofs, the VM would contain the right version of the relevant theorem prover. We recommend using a format that is easy to work with, such as OVF or Docker images. 
• Binary Installer: Indicate exactly which platform and other run-time dependencies your artifact requires. 
• Live Instance on the Web: Ensure that it is available for the duration of the artifact evaluation process.

The structure and contents of an artifact’s documentation must follow the following sections: 

• Artifact Identification: Including an abstract describing the main contributions of the article  and how the role of the artifact in these contributions. 
• AI Methods: The method documentation includes the AI method that the AI program implements as well as the motivation of why the method is used. As the implementation does not contain the motivation and intended behavior, sharing the implementation of the AI program is not enough. It is important to give a high-level description of the AI method that is being tested. This includes what the AI method intends to do, why it is needed and how it works. To decrease ambiguity, a description of how a method works should contain pseudo code and an explanation of the pseudo code containing descriptions of the parameters and sub-procedures. Sharing of the AI method is the objective of most research papers in AI. The problem that is investigated must be specified, the objective of the research must be clear and so must the research method being used. 
• Datasets: Sharing the data used in the experiment is getting simpler with open data repositories, such as IEEE DataPort or the UCI Machine Learning Repository. Reproducing the results fully requires the procedure for how the data set has been divided into training, validation and test sets and which samples belong to the different sets. Sharing the validation set might not be necessary when all samples in the training set are used or might be hard when the method picks the samples randomly during the experiment. Data sets often change, so specifying the version is relevant. Finally in order to compare results, the actual output of the experiment, such as the classes or decisions made, are required. 
• Experiments 
• Artifact Dependencies and Requirements: Including (i) a description of the hardware resources required, (ii) a description of the operating systems required, (iii) the software libraries needed, (iv) the input dataset needed to execute the code or when the input data is generated, and (v) optionally, any other dependencies or

requirements. Best practices to facilitate the understanding of the descriptions indicate that unnecessary dependencies and requirements should be suppressed from the artifact. 

• Artifact Installation and Deployment Process: Including (i) the process description to install and compile the libraries and the code, and (ii) the process description to deploy the code in the resources. The description of these processes should include an estimation of the installation, compilation, and deployment times. When any of these times exceed what is reasonable, authors should provide some way to alleviate the effort required by the potential recipients of the artifacts. For instance, capsules with the compiled code can be provided, or a simplified input dataset that reduces the overall experimental execution time. On the other hand, best practices indicate that, whenever it is possible, the actual code of software dependencies (libraries) should not be included in the artifact, but scripts should be provided to download them from a repository and perform the installation. 
• Reproducibility of Experiments: Including (i) a complete description of the experiment workflow that the code can execute, (ii) an estimation of the execution time to execute the experiment workflow, (iii) a complete description of the expected results and an evaluation of them, and most importantly (iv) how the expected results from the experiment workflow relate to the results found in the article. Best practices indicate that, to facilitate the understanding of the scope of the reproducibility, the expected results from the artifact should be in the same format as the ones in the article. For instance, when the results in the article are depicted in a graph figure, ideally, the execution of the code should provide a (similar) figure (there are open-source tools that can be used for that purpose such as gnuplot). It is critical that authors devote their efforts to these aspects of the reproducibility of experiments to minimize the time needed for their understanding and verification. 
• Other notes: Optionally, it might include other related aspects that could be important and  were not addressed in the previous points.

There is a growing interest in reproducibility in the computer science community. The ACM Conference on Reproducibility and Replicability was established in 2023, as a conference whose full focus is to bring together experts and practitioners engaged in the advancement and conduct of reproducible science in all computing disciplines. Workshops with focus on reproducibility on AI have been also organized, including the Workshop on Replicability and Reusability in Natural Language Processing: From Data to Software Sharing 2015, the workshop Reproducibility in Machine Learning Research 2017, or the workshop Enabling Reproducibility in Machine Learning MLTrain@RML.

Most importantly, some conferences and journals are already integrating reproducibility evaluations in parallel with paper reviews, such as The International Conference for High Performance Computing, Networking, Storage, and Analysis, SC (introduced in 2015); the Annual Conference on Machine Learning and Systems , MLSys (introduced in 2020); the International Conference on Parallel Processing, ICPP (since 2022); the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE (since 2011); or the journal IEEE Transactions on Parallel and Distributed Systems (introduced in 2018). These conferences and journals are typically based on the Artifact Description (AD) / Artifact Evaluation (AE) submission processes. An Artifact Description form (AD) includes specific documentation on the experiments that appear in the associated paper. The Artifact Evaluation also extends on steps to ensure that the experiments are reproducible. 

SC, TPDS and ICPP 

The International Conference for High Performance Computing, Networking, Storage, and Analysis, (SC) has a long tradition on reproducibility that started in SC15. The process consists of two stages (see Figure 3): A preliminary reproducibility evaluation (Artifact Description stage, AD) in parallel with paper reviews and, once a paper is accepted, a full reproducibility evaluation (Artifact Evaluation stage, AE). In SC19, the AD phase became obligatory for all submissions, whereas the AE phase remains optional to date. However, a significant proportion of accepted papers in SC23 requested the reproducibility evaluation. Among those that did not, technical barriers were often cited as the primary reason. The evaluation of reproducibility requires authors submit their computational artifacts and their documentation, the documentation is typically in the form of an appendix that is published along with the paper. 

The evaluation of reproducibility requires both human and computational resources. In addition to the paper review committees, there is a specialized committee for reproducibility that consists of PhD students, post-docs and researchers that evaluate the reproducibility of submissions. It is, therefore, sensitive to undertake a full reproducibility evaluation only for papers that have been accepted for publication at the AE stage. However, the outcomes of the AD stage can also be useful for the decisions on paper evaluation. This is the reason why there is often a light reproducibility evaluation (AD stage) that occurs in parallel with paper evaluation and then, only for accepted papers, there is a complete reproducibility evaluation. 

Figure 3. SC and ICPP conferences reproducibility process

As a result of the evaluation, reproducibility badges are granted. The objective is to effectively test and evaluate the artifacts in order to assign (if appropriate) the badges requested by the appendix authors. The badges quantify different levels of reproducibility and they are defined by the editorial (SC makes use of ACM badges). Typically, they define whether the artifacts are open and available to third-party researchers (not just the reviewers). It examines whether artifacts used in the research (including data and code) are permanently archived in a public repository that assigns a global identifier and guarantees persistence, and are made available via standard open licenses that maximize artifact availability. Another typical badge evaluates whether the reviewer successfully reproduced the key computational results using the author-created research objects, methods, code, and conditions of analysis. For those cases where the results cannot be successfully reproduced, there is a badge that indicates that the artifacts have been functionally evaluated and that they might be useful to reproduce the experiments in the paper. 

The IEEE Transactions on Parallel and Distributed Systems journal (TPDS) initiated its reproducibility initiative in 2018 and it holds the distinction of being the first IEEE transactions journal to implement a reproducibility badging system. Unlike the conference SC, TPDS only offers the AE stage, as a post-publication process (see Figure 4). The International Conference on Parallel Processing (ICPP) started its reproducibility initiative in 2022. The reproducibility processes in ICPP23 and TPDS were aligned with SC23, facilitating familiarity for researchers from both communities. During this AE phase at SC and ICPP and with the aim of helping reviewers to configure their experiments, an anonymous communication mechanism is enabled within the reproducibility evaluation management system. The communication system allows both reviewers and authors to send messages and clarify different aspects. 

Figure 4 . IEEE T on Parallel and Distributed Systems journal reproducibility process 

Additional Activities Around Reproducibility 

In addition to the ADAE phases, SC also undertakes a reproducibility challenge, thereby a paper from the previous SC edition is selected and provided to the Student Cluster Competition (SCC). The SCC is an annual event that takes place as part of the Supercomputing Conference (SC) and during the event. It is a computing competition that brings together teams of undergraduate students from around the world. Among other things, these teams are challenged to reproduce the experiments of the selected paper. Typically, the students are provided a different hardware setup

from the original one in the paper and they are challenged to obtain similar conclusions but from a completely different technical perspective. During the following SC edition, these students are asked to write a critique reproducibility paper with their findings during the competition, and authors extend their paper with the findings of those critiques. The result is published as a journal’s special issue with special emphasis on reproducibility. 

Similarly, acknowledging the importance of reproducibility in ensuring the integrity of scientific and technical research, as well as the difficulties associated with assessing code and data for reproducibility, TPDS has introduced an innovative approach. In this new approach, community members have the opportunity to submit “critique” papers, sharing their firsthand experiences in attempting to replicate published research results using the provided artifacts, or providing assessments and insights into the published artifacts. These supplementary paper submissions will undergo a thorough review process, and if accepted, they will be linked to the original publication and become citable references. This process aims to contribute to the validation of the reproducibility of the original research findings. This approach has been implemented in two distinct special sections: 

• Transparency and Reproducibility Practice in Large-Scale Computational Science, 2021 • Advancing Adoption of Reproducibility in HPC, 2022 
• The International Conference for High-Performance Computing, Networking, Storage, and Analysis in 2021 and 2022 (SC21 and SC22) 

The ACM Joint European Software Engineering Conference and Symposium on the Foundations  of Software Engineering (ESEC/FSE) 

ESEC/FSE started evaluating reproducibility in 2011, being one of the pioneer conferences. For ESEC/FSE, authors of papers accepted to the ESEC/FSE Technical and Industrial Tracks are encouraged to submit associated artifacts to the ESEC/FSE Artifact Track for evaluation as potential candidates for Artifacts Evaluated – Functional, Artifacts Evaluated – Reusable, and Artifacts Available. As a result of over a decade of reproducibility evaluation and practice in the conference, it was found that articles with granted artifacts do not generally have better visibility in the community, measured in terms of number of citations 

MLSys 

Building on the success of implementing the artifact evaluation process at MLSys’19 and observing positive outcomes from similar endeavors at NeurIPS’19 and ASPLOS’20, where over half of the accepted papers submitted artifacts for evaluation, MLSys’20 implemented a reproducibility initiative at the Conference on Machine Learning and Systems 2020. This conference prioritizes the reproducibility of experimental findings and advocates for the sharing of code and data to facilitate swift validation and comparison of different methodologies within the community. Authors of accepted papers are encouraged to formally document supporting materials, including code, data, models, workflows, and results, utilizing the standardized Artifact Appendix template for

submission to the Artifact Evaluation process (AE). The participation in this process was optional and not considered on the final decision on paper acceptance. The primary goal is to assist authors in validating their experimental findings through collaborative assessment by an independent AE Committee and to aid readers in locating articles with accessible, functional, and verified artifacts. 

ECAI 

Being one of the major AI conferences in Europe, ECAI is also considering reproducibility but not full reproducibility evaluation. ECAI authors must follow the reproducibility guidelines and they need to verify a checklist at the time of paper submission (see Figure 5). Reviewers, on the other hand, will verify that the checklist was completed and a reproducibility score will come out of that. 

Figure 5 . ECAI conference paper evaluation process, image taken from ECAI’22 

The ECAI reproducibility checklist encompasses various crucial aspects. It guides authors to provide conceptual outlines or pseudocode descriptions of introduced AI methods, clear distinction between opinions and objective facts, and the provision of pedagogical references for readers’ comprehension. It also provides guidance when the validation is based on theoretical contributions rather than on experiments. Moreover, it also tackles the documentation of datasets involved. Authors need to provide motivations behind their selection, and their availability and citation. Additionally, for the computational experiments, the reproducibility checklist covers computational experiments, encompassing code accessibility, infrastructure details, and comprehensive evaluation metrics, including the number of runs and statistical significance, and the documentation of hyperparameters, their selection criteria, and the breadth of parameter exploration during development.

Reproducibility Guidelines 

• How to Submit a Study: Step-by-step guide 
• Required Artifacts: Code, datasets, documentation 
• Best Practices: Ensuring reproducibility 
• Evaluation Process: How submissions are reviewed 

• Templates & Examples: Submission formats

Based on standard NISO, AI4Europe will consider the following badges: 

Open Research Objects (ORO) 

This badge signals that author-created digital objects used in the research (including data and code) are permanently archived in a public repository that assigns a global identifier and guarantees persistence, and are made available via standard open licenses that maximize artifact availability 

The following requirements are necessary to receive this badge: 

• Artifacts used in the research (including data and code) are permanently archived in a public repository that assigns a global and unique identifier (DOI) and guarantees persistence and are made available via standard open licenses that maximize artifact availability. DOIs can be acquired via Zenodo, FigShare, Dryad, Software Heritage. If authors already have their code in github, Zenodo provides an integration with Github to automatically generate DOIs from Git tags. 
• Please do NOT provide Dropbox links or gzipped files hosted through personal webpages. 

Note that, for physical objects relevant to the research, metadata about the object should be made available. 

Research Objects Reviewed (ROR) 

This badge signals that all relevant author-created digital objects used in the research (including data and code) were reviewed according to the following criteria: 

• Metadata / Documentation: Are the artifacts sufficiently documented to enable them to be exercised by readers of the paper? 
• Completeness: Do the submitted artifacts include all the key components described in the paper? 
• Exercisability: Do the submitted artifacts include the scripts and data needed to run the experiments described in the paper, and can the software be successfully executed?
• We encourage authors to describe their (i) workflow underlying the paper, (ii) describing some of the black boxes, or a white box (e.g., source, configuration files, build environment), (iii) input data: either the process to generate the input data should be made available, or when the data is not generated, the actual data itself or a link to the data should be provided, (iv) environment (system configuration and initialization, scripts, workload, measurement protocol) used to produce the raw experimental data, and (v) the scripts needed to transform the raw data into the graphs included in the paper. 

Results Reproduced (ROR-R) 

The Research Objects were Reviewed and then, the evaluators successfully reproduced the key computational results using the author-created research objects, methods, code, and conditions of analysis. Note we do not aim to recreate the exact or identical results, especially hardware-based results. However, we do aim to: 

Reproduce Behavior: This is of specific importance where results are hardware-dependent. Bit wise reproducibility is not our goal. If we get access to the same hardware as used by experiments, we will aim to reproduce the results on that hardware. If not, we aim to work with authors to determine the equivalent or approximate behavior on available hardware. For example, if results are about response time, our objective will be to check if a given algorithm is significantly faster than another one, or that a given parameter affects negatively or positively the behavior of a system. 

Reproduce the Central Results and Claims of the Paper: We do not aim to reproduce all the results and claims of the paper. The reproducibility committee will determine the central results of the accepted paper and will work with authors to confirm it. Once confirmed, the badge will be assigned based on the committee being able to reproduce behavior of these central results.

Several factors make certain areas of AI difficult to reproduce, and indeterminism and performance variability are two key challenges in this regard. Many AI techniques, such as deep learning, involve complex algorithms and models with numerous parameters. Even a small change in these parameters or the random initialization of weights can lead to significantly different results. This makes it challenging to reproduce results exactly. 

On the other hand, many AI algorithms incorporate randomness, such as dropout or random initialization. This inherent stochasticity can lead to different outcomes in each run of an experiment. While researchers often report average results over multiple runs, the specific outcomes can still vary. Overall, different technological and social barriers hinder the reproducibility of experiments. Some of them have been identified previously, some others like the infrastructure and middleware barriers and, social and governance barriers are described here, to the best of our knowledge, for the first time. 

Technical Barriers 

• Privacy

The reproducibility of a Deep Neural Network can be significantly impacted by data, architecture or weights privacy limitations. When the original model’s training data are inaccessible or withheld due to privacy concerns, it hampers the ability to replicate experiments accurately, leading to divergent results between the original and the reproduced neural network. Similarly, private architectural details challenge the complete understanding and replication of the model, even if its training data are available. As a result, there is a need for methods ensuring reproducibility and being flexible to those privacy limitations. 

• Proprietary Software 

Proprietary software dependencies (i.e third party libraries) or the experimental code itself that cannot be shared for confidential / business reasons. It makes it more complicated for reviewers and third-party researchers and practitioners to find an alternative functionality to the proprietary software. In some cases, it can be impossible to find a viable alternative software. 

• Hardware Dependencies 

Specific hardware dependencies that can be very difficult to solve, the reproducibility of experiments can require very specific and expensive hardware, or a very significant amount of hardware units, also increasing the economic cost. 

• Randomness Exploited by AI models 

The inherent randomness exploited by AI models, such as in training neural networks with stochastic optimization algorithms or in sampling-based approaches like Monte Carlo methods, poses a significant barrier to reproducibility. These models often rely on random initialization, data shuffling, dropout, or stochastic gradient descent, resulting in slight variations in outcomes with each run. While randomness is crucial for exploration and learning in AI systems, it makes reproducing exact results challenging, as even small changes in initialization or data ordering can lead to divergent model behaviors. Consequently, replicating experiments becomes cumbersome, hindering the validation and comparison of findings across different studies or researchers. 

• Infrastructure & Middleware Dependencies 

Typically, the design and the execution of the whole experimental process undertaken by authors in the distributed and parallel computing community is accomplished in a computing infrastructure that is owned by their institution and that reviewers or third-party researchers cannot access. This can represent a challenge when the computing infrastructure is not easily reproducible, for instance when the experiments require specific configurations of middleware such as SLURM or Condor. Given the inherent complexity of computing infrastructures, this leads to a significant barrier when reviewers or third-party researchers attempt to reproduce the experiments. 

• Reproducibility Tools

At present, the reproducibility process involves the utilization of a multitude of diverse tools, requiring authors to switch between them and also requiring them to fill in different forms. For instance, code repositories like Github lack the provision of a DOI, prompting authors to configure version tagging and subsequently associate their software with Zenodo to acquire a DOI. Datasets, on the other hand, follow a distinct procedure, with some open repositories like Zenodo or IEEE dataport automatically assigning a DOI once a dataset is updated. 

On the other hand, the existing reproducibility evaluation management tools fall short in its design, lacking a dedicated focus on reproducibility from the outset. Consequently, it proves challenging to use and lacks seamless integration with other tools. As an example, the absence of integrated access to the experiment playground for running experiments further highlights these limitations. 

Social and Governance Barriers 

• The lack of a Shared Infrastructure 

Researchers perform their experiments in their own institutional computing infrastructures and that can represent a challenge when reviewers need to recreate a complex distributed / parallel infrastructure. The idea of having a shared computing environment where authors, reviewers, and even third-party researchers and practitioners can reproduce their experiments can really simplify the burdens and complexity of configuring and subsequently executing the experiments. However, no shared computing infrastructure is being offered by editorials. The AIoD could play this role, and it can also serve as a repository to store AI assets that can be valuable for the community. 

• The Novelty of Reproducibility 

In contrast to other facets and activities of research, the reproducibility of experiments is a relatively recent aspect. For the reproducibility of experiments, authors need to prepare their artifacts and their documentation. In the scientific community, there is a well-established consensus on structuring papers and their content, which is instilled early in one’s research career. However, the same cannot be said for reproducibility, its documentation, and computational artifacts, despite the existing guidelines provided for authors and reviewers. For instance, we observed that sometimes important aspects of the artifact documentation were missing: Some artifacts were not in the required maturity, so that they could no be executed by reviewers without a significant debugging and editing; also when the output of the execution of the computational artifacts did not match the experiments in the paper, then documentations did not provide any explanations on it. One of the consequences of this is the significant heterogeneity in the submissions and the disparity of the quality, both in the documentation and in the computational artifacts, leading to more confusion in the reviewers, as there is a lack of a reference of quality to compare.

On the other hand, when a reviewer faces the reproducibility assessment, for the same novelty reasons, a reviewer does not exactly know what to expect, how much effort is reasonable to devote, and the main aspects to evaluate. 

• Author’s Role in Reproducibility 

Reproducibility is still perhaps perceived in the community as a burden that can take significant time and does not provide an obvious benefit. This can be understood from the technical debt perspective. Technical debt is a metaphorical concept in software development that refers to the hidden costs and compromises made during the development process when quick and suboptimal solutions are chosen to meet immediate needs. Just like financial debt, technical debt accumulates interest over time in the form of increased complexity, reduced maintainability, and higher risks. When developers take shortcuts or delay necessary tasks, such as code refactoring, to meet tight deadlines or to quickly release a product, they incur technical debt. This can lead to problems like buggy software, slower development velocity, and increased difficulty in making future enhancements. 

• Reviewers Role in Reproducibility 

Reviewing reproducibility can be a time-consuming process and it is often unclear how much time is reasonable for a reviewer to expend on this. Reproducibility reviewers sometimes encounter poor-quality artifacts, where the code does not compile, some key elements are missing or they require significant effort for the configuration until the experiments can be reproduced. To alleviate this, in the reproducibility management system used at SC, TPDS and ICPP, there is an anonymous messaging system enabled that allows authors and reviewers to interact with each other with the purpose of fixing coding bugs, scripts, etc. However, this often involves a significant time and reviewers are often not engaged on this. 

Furthermore, sometimes the compilation time or the execution time of the experiments exceed the reasonable. This can put pressure on both reviewers and computational resources required. Adequate compensation and time allocation are essential to encourage reviewers to conduct thorough assessments and to provide authors with the opportunity to address the reviewer’s concerns effectively. 

On the other hand, reproducibility reviewers encounter an additional obstacle in the form of insufficient high-quality artifacts to serve as reference materials, and then, it is difficult for them to evaluate a submission based on a standardized idea of quality. Furthermore, the time a reviewer must spend on evaluating a reproducibility submission is a crucial factor in the reproducibility peer review process. We believe that we already have a significant experience in peer review and we should not reinvent the wheel in this regard in reproducibility. When the execution of experiments require a significant amount of time beyond reasonable, authors should additionally provide a

reduced scale shortcut that can help reviewers to reach similar conclusions to the ones reached on the paper. 

• Lack of Recognition 

The role of reproducibility reviewers is not always well-acknowledged within the research community. The contribution of these reviewers in ensuring the robustness of research is invaluable, but their efforts may not receive the same recognition and prestige as traditional peer reviewers. To address this, the scientific community needs to acknowledge and value the contributions of reproducibility reviewers. Recognizing their work through academic incentives and providing a platform for sharing best practices can motivate researchers to invest more in making their work reproducible. 

• The role of the reproducibility chair 

The main barrier that the reproducibility chair can encounter is to find appropriate reproducibility reviewers. This barrier stems from the specialized nature of reproducibility evaluation, requiring reviewers with a nuanced understanding of diverse methodologies and technologies. Moreover, the sheer volume of submissions necessitates a considerable number of reviewers, adding to the strain of finding individuals with both the requisite expertise and availability. This scarcity of reproducibility reviewers often leads to delays in the review process and may compromise the thoroughness of evaluations. Reproducibility evaluation often requires familiarity with a broad array of experimental techniques, analytical tools, and technological platforms. Reviewers without this specialized expertise may struggle to identify potential pitfalls in research methodologies or assess the reliability of experimental results accurately. Consequently, there is a risk of introducing biases or inaccuracies into the review process, undermining its reliability and integrity. Addressing this challenge entails ongoing efforts to enhance reviewer training and expertise in reproducibility assessment, as well as fostering interdisciplinary collaboration to leverage diverse perspectives and skills in the peer review process.

To overcome the technical barriers, it is impractical to expect reviewers or third-party researchers to replicate highly specific computing setups. Ideally, authors should offer a means to either reproduce the infrastructure, such as through Software Defined Infrastructure technologies, or grant reviewers direct access. However, this introduces additional complexities. For instance, system administrators of these infrastructures may also need reviewers to undergo identification procedures for security purposes, potentially compromising the anonymity of the peer review process, which is undesirable. One potential option for providing access to reviewers and third party researchers is through the utilization of a federation of resources. However, these solutions are intricate and will require a considerable amount of time. On the other hand, the current tools for ensuring reproducibility must undergo a redesign and reevaluation to better align with the

requirements of all involved parties, including authors, reviewers, chairs, and third-party researchers. 

To solve the social and governance barriers, it is, therefore, important to provide more visibility to good practices of reproducibility. Activities around reproducibility such as the student challenge competition of SC, the possibility of writing reproducibility critiques that are published at TPDS, or a proper recognition of reproducibility reviewers and reproducibility practices, in general, will definitely contribute to it. There is also a growing need for training reviewers on reproducibility and creating a pool of reviewers with expertise in reproducibility to ensure a more comprehensive and accurate review process. SC, TPDS, and ICPP are already contributing to it, as they have already organized reproducibility committees that provide practical experience to participants. Additionally, we believe that it is essential for the reproducibility process to be clearly outlined in guidelines. These guidelines should offer standardized documentation for the components involved and offer direction to both authors and reviewers throughout each stage of the process. 

Figure 6.: Our view on the Reproducibility Initiative process 

Our view on reproducibility can be summarized in Figure 6. The actors involved, namely authors, reviewers and third-party researchers and practitioners, follow the process that we specified and they make use of a number of tools to implement it and a common playground, a common computing infrastructure where authors deploy their experiments and reviewers can reproduce them. 

Common Playground

Although some experiments can be replicated on any computing infrastructure, one of the common barriers to the reproducibility of experiments is the complexity of configuring the experiments to execute in a distributed infrastructure. To simplify the process and enhance the productivity of reviewers, we believe that a common computing infrastructure can contribute to simplify the process. The main idea is that the authors configure and prepare their experiments in a computing infrastructure that they share with reviewers. Ideally, in the long term, that common infrastructure should also be accessible for third-party researchers and practitioners. This would help contribute to the ultimate goal, that is that the research community can reproduce and reuse the experiments, and not just the reviewers. 

Technical Aspects: Privacy 

Novel methodologies, such as knowledge distillation, can play a pivotal role in bridging the gap of privacy limitations. By integrating these techniques, the scientific community can foster reproducibility in deep learning research while respecting data, architecture, and weights privacy limitations. 

A significant advancement in this area can be achieved through the use of a novel distillation driven framework. This framework establishes a collaborative environment for neural networks, comprising agents capable of exchanging information and learning from one another. Addressing the reproducibility challenge involves successful collaboration between a newly introduced agent and the original agent created by the experiment’s authors. The framework employs three essential mechanisms to enhance reproducibility: 

• enabling data exchange between agents, 
• facilitating the exchange of weights between agents, 
• enabling learning through knowledge distillation techniques. 

The combination of the mechanisms mentioned offer five reproducibility options that offer different levels of privacy to the original agents. The experimental results are promising regarding the framework’s capabilities in achieving reproducibility. The plethora of options offered by the framework enable the reproducibility of the original neural network considering its privacy limitations. The experiments were conducted on a model trained on a binary classification dataset for fire detection, which is reproduced using every different option. The disagreement between the original and the reproduced neural network varies from 3.74% to 5.74%, depending on the original network’s privacy limitations. The weight and architecture transmission is also provided as an option, that reproduces the exact original model but having the minimum privacy possible. The options offered introduce a flexible and fully automated way of achieving reproducibility in neural networks, as most reproducibility methods have strict requirements, which are rarely met. The

potential of integrating this tool in online platforms, such as AI on Demand (AIoD), could provide new solutions regarding the reproducibility problem. 

Social Aspects: Standardized Documentation 

In the context of reproducibility of research, standardized documentation is a key factor in enhancing the quality of artifact descriptions. We believe that adhering to standardized documentation can assist authors in effectively describing their experiments and enhance their efficiency. Additionally, it can help reviewers in their assessment by providing a clearer understanding of what to anticipate. Moreover, it contributes to rendering evaluations more consistent. We believe that the following aspects should be covered in the standardized documentation. 

Artifact Identification is the first crucial step in this process. It involves presenting an abstract that describes the main contributions of the article and the role of computational artifacts in achieving these contributions. This abstract should also provide insights into the software architecture or data models, aiding readers in understanding the significance of these computational artifacts and how they contribute to the reproducibility of the experiments detailed in the article. 

The Reproducibility of Experiments is another critical aspect of standardized documentation. It necessitates a complete description of the experiment workflow that the code can execute. Furthermore, it should include an estimation of the time required to execute this experiment workflow, a comprehensive description of the expected results, and an evaluation of these results. Importantly, authors should clarify how the expected results from the experiment workflow relate to the results presented in the article, ensuring a clear and consistent link. 

Artifact Dependencies and Requirements encompass several elements that need to be clearly outlined. This includes a description of the hardware resources necessary, the operating systems required, and the software libraries needed for executing the code. Additionally, authors should specify the input dataset required to run the code or, when applicable, describe how the input data is generated. Authors can also include any other dependencies or requirements, although it is advisable to omit unnecessary ones from the artifact. 

The Artifact Installation and Deployment Process is a critical phase that authors should document thoroughly. This documentation should outline the process for installing and compiling the required libraries and code. Additionally, authors should provide estimates of the time needed for these installation, compilation, and deployment processes. If these time estimates exceed what is considered reasonable, authors should offer solutions to alleviate the effort required by potential artifact recipients. For example, they can provide pre-compiled code capsules or a simplified input dataset to expedite experimental execution. It’s also best practice to avoid including the actual

code of software dependencies (libraries) in the artifact whenever possible. Instead, authors should provide scripts for downloading these dependencies from a repository and performing the installation. 

Lastly, the documentation should account for any other relevant aspects not covered in the preceding points. These “Other aspects” should encompass any additional information that can be pertinent to understanding and replicating the research. 

In addition to standardized documentation, the process of ensuring research reproducibility involves the establishment of clear guidelines and references to maintain quality. Moreover, common infrastructure and reproducibility tools are essential to create a shared environment that simplifies the replication of experiments. The allocation of resources for this common playground is a key consideration, as it is crucial to determine who bears the cost of these resources. Finally, there is a growing emphasis on data in the context of research reproducibility, underlining the significance of transparent and accessible data as a cornerstone of the reproducibility process. 

Social Aspects: The need to better document datasets for reproducibility On some occasions, experiments in parallel and distributed computing rely on synthetic data. However, whenever datasets are obtained in a real setting, they also need to be properly documented. Aspects like data provenance, information about the dataset’s origin, such as its source, data collection methods, and the timeframe in which the data was collected are important as this metadata would enable users to assess the reliability and credibility of the dataset. It is also important to specify any preprocessing steps applied to the data, such as cleaning, filtering, or transformation, to help users understand how the raw data was processed and any potential impact on the final dataset. Another important aspect to specify is clear definitions and descriptions of the variables present in the dataset, including their units of measurement, data types, and possible values, would enable users to correctly interpret and analyze the data. Besides, it would be beneficial to explain how missing or incomplete data were treated, whether through imputation, exclusion, or other methods. This metadata would allow users to understand any potential biases or limitations resulting from missing data. Documentation of quality assessments performed on the dataset, including measures such as accuracy, completeness, consistency, and reliability, would help users gauge the overall quality and reliability of the data. 

Providing information regarding any privacy or ethical considerations associated with the dataset, such as anonymization techniques used, consent procedures, or adherence to data protection regulations, would ensure compliance with legal and ethical standards. Furthermore, if the dataset has undergone changes or updates, it would be beneficial to include version information and a changelog to track modifications over time. This would enable users to understand the dataset’s evolution and any potential impact on analysis or comparability.

Social Aspects: The anonymity of Peer review should be preserved. 

The traditional peer review process for paper evaluation has a long history. Its origin can be found in the 17th century in the Royal Society of London. Peer review has several essential purposes in academia and science. It serves as a quality assurance mechanism. Reviewers assess the methodology and contributions presented in a manuscript, helping identify errors, biases, and weaknesses. This process ensures that only high-quality and reliable work is published in reputable journals, contributing to the integrity of the academic and scientific literature. When a manuscript passes the peer review process and is accepted for publication, it signifies that the research has met certain standards of quality and reliability. This validation is essential for building trust and credibility within the academic and scientific communities. 

The tradition of peer review has evolved over time and it has consolidated interactions between reviewers and authors. It typically involves multiple steps, such as the initial editorial review, assignment to peer reviewers, feedback and revisions by authors, and a final decision by the journal’s editorial board. We argue that this should be the starting point for setting up the reproducibility evaluation process and it is not a good idea to reinvent a new review procedure. We argue against the implementation of a message system akin to that proposed for SC, ICPP, and TPDS, where reviewers and authors can establish anonymous interactactions, which encourages reviewers to act as software testers. Such an approach amplifies the workload on reproducibility reviewers. Instead, interactions should align more closely with the traditional peer review process, as reproducibility is just another aspect of a paper. 

Contained within the attached appendix are comprehensive details outlining the process and standardized documentation tailored for authors. This comprehensive documentation serves as a roadmap, guiding authors through each step of the submission process, ensuring clarity and consistency. Moreover, it extends its utility beyond authors, offering invaluable guidance to reviewers as well, ensuring that all involved parties are well-equipped to navigate the review process effectively.

In the reproducibility process that we propose, these are the steps that authors need to consider for the proposed process. Maximizing automation throughout the entire process is highly desirable. In this context, authors are to provide all necessary scripts for deploying and installing their computational artifacts, particularly at the Artifact Evaluation (AE) stage. The overarching objective is to facilitate the reproducibility of the paper’s experiments, allowing for the potential reuse of results by other researchers. A common barrier to experiment reproducibility lies in intricate installation, deployment, and configuration setups, which often demand substantial efforts from third-party researchers and practitioners.

AD phase – Fast-check reproducibility evaluation 

• Data Artifact Documentation. Then, the dataset must be documented. It can include methodological aspects about how the dataset was obtained, if there was any preprocessing and transformation, etc. More information is available in the computational artifact(s). 
• Code Artifact Hardware Dependencies. Specify and document hardware requirements (including type and quantity). It could be the case that the code artifact does not have any requirements, then it should be stated. In some cases, the requirements can involve specific CPU, RAM, storage, or special accelerators such as GPUs. More information here (the description (documentation metadata) of the artifact(s)). 
• Code Artifact Software Dependencies. Document the OS required, as well as any third party libraries required to execute the experiments. 
• Experimental Workflow. This is the *most important* step for the documentation of the artifact. In this section, authors include (i) a complete description of the experiment workflow that the code can execute, (ii) an estimation of the execution time to execute the experiment workflow, (iii) a complete description of the expected results and an evaluation of them, and most importantly (iv) how the expected results from the experiment workflow relate to the results found in the article. Best practices indicate that, to facilitate the understanding of the scope of the reproducibility, the expected results from the artifact should be in the same format as the ones in the article. For instance, when the results in the article are depicted in a graph figure, ideally, the execution of the code should provide a (similar) figure (there are open-source tools that can be used for that purpose such as gnuplot). It is critical that authors devote their efforts to these aspects of the reproducibility of experiments to minimize the time needed for their understanding and verification. 

Steps for the AE phase – Full reproducibility evaluation 

To minimize the effort required by reviewers and promote the future reuse of results, it is highly desirable to maximize automation throughout the entire process. The attainment of this goal can be facilitated by packaging the artifacts (such as with docker containers) and offering installation and deployment scripts. 

• Data artifact availability. In case your computational artifacts include any kind of dataset, it must be made stored in an *open*, public, persistent repository, such as Zenodo. These open repositories assigned a Digital Object Identifier (What is a DOI? Check this out) that allows researchers to identify the dataset univocally.
• Data artifact documentation. Then, the dataset must be documented. It can include methodological aspects about how the dataset was obtained, if there was any preprocessing and transformation, etc. More information in the computational artifact(s). 
• Code artifact Hardware dependencies. Specify and document hardware requirements (including type and quantity). It could be the case that the code artifact does not require any particular hardware requirements, then it should be stated. In some cases, the requirements can involve specific CPU, RAM, storage, or special accelerators such as GPUs. More information here (the description (documentation metadata) of the artifact(s)). 
• Code Artifact Software Dependencies. Document the OS required, any third-party libraries required. If using container technologies, as recommended, the documentation of this step can be easily accomplished by looking at the dockerfile. 
• Experimental Workflow. This is the *most important* step for the documentation of the artifact. In this section, authors include (i) a complete description of the experiment workflow that the code can execute, (ii) an estimation of the execution time to execute the experiment workflow, (iii) a complete description of the expected results and an evaluation of them, and most importantly (iv) how the expected results from the experiment workflow relate to the results found in the article. Best practices indicate that, to facilitate the understanding of the scope of the reproducibility, the expected results from the artifact should be in the same format as the ones in the article. For instance, when the results in the article are depicted in a graph figure, ideally, the execution of the code should provide a (similar) figure (there are open-source tools that can be used for that purpose such as gnuplot). It is critical that authors devote their efforts to these aspects of the reproducibility of experiments to minimize the time needed for their understanding and verification, for instance, by stating how the execution outcome matches the experiments in the paper exactly and how long it takes to execute the experiments. Authors need to consider how long it is reasonable for reviewers and third-party researchers to wait for the experiments. For the cases of long run executions, it is advisable to provide an additional reduced-scale version of the experiment that can lead to analogous conclusions. 
• Code Artifact availability. In case your computational artifacts include any type of code, the code must be stored in an *open*, public, persistent repository. We recommend using github that can be subsequently paired with Zenodo. Zenodo provides a DOI to the code, so that it guarantees that it cannot be modified while the reviewers are examining it, and in the future, when third-party researchers and practitioners are reusing it. 
• Packaging Method of the Code Artifact. We strongly recommend container technologies to provide the code, which simplifies the deployment and installation processes and helps document the dependencies. More information about packaging methods here (The computational artifact(s)) 
• Installation & Deployment. Prepare installation and deployment scripts and document the installation and deployment process, including an estimation of the installation and deployment times. The documentation should also include (i) the process description to install and compile the libraries and the code, and (ii) the process description to deploy the code in the resources. The description of these processes should include an estimation of

the installation, compilation, and deployment times. When any of these times exceed what is reasonable, authors should provide some way to alleviate the effort required by the potential recipients of the artifacts. For instance, capsules with the compiled code can be provided, or a simplified input dataset that reduces the overall experimental execution time. On the other hand, best practices indicate that, whenever it is possible, the actual code of software dependencies (libraries) should not be included in the artifact, but scripts should be provided to download them from a repository and perform the installation. 

In the reproducibility process that we propose, these are the steps that reviewers need to consider for the proposed process. Maximizing automation throughout the entire process is highly desirable. In this context, reviewers need to be provided with all necessary computational resources (hardware) and software artifacts, including scripts for deploying and installing their computational artifacts, particularly at the Artifact Evaluation (AE) stage. The amount of time required to deploy, compile and execute the experiments should be reasonable. When experiments require a significant amount of time, authors should provide a reduced scale version that helps reviewers to reach similar conclusions. 

AD phase – Fast-check reproducibility evaluation 

• Data Artifact Documentation. Analyze whether the dataset is well documented, and it includes methodological aspects about how the dataset was obtained, if there was any preprocessing and transformation, etc. 
• Code Artifact Hardware Dependencies. Verify the specification of the hardware requirements (including type and quantity) is complete. It could be the case that the code artifact does not have any requirements, then it should be stated. In some cases, the requirements can involve specific CPU, RAM, storage, or special accelerators such as GPUs. More information here (the description (documentation metadata) of the artifact(s)). 
• Code Artifact Software Dependencies. Verify that the documentation of the OS required is complete, as well as any third-party libraries required to execute the experiments. • Experimental Workflow. This is the *most important* step for the documentation of the artifact. In this section, reviewers must check whether authors included (i) a complete description of the experiment workflow that the code can execute, (ii) an estimation of the execution time to execute the experiment workflow, (iii) a complete description of the expected results and an evaluation of them, and most importantly (iv) how the expected results from the experiment workflow relate to the results found in the article. Best practices indicate that, to facilitate the understanding of the scope of the reproducibility, the expected results from the artifact should be in the same format as the ones in the article. For instance, when the results in the article are depicted in a graph figure, ideally, the execution of the code should provide a (similar) figure (there are open-source tools that can be used

for that purpose such as gnuplot). It is critical that authors devote their efforts to these aspects of the reproducibility of experiments to minimize the time needed for their understanding and verification. 

Steps for the AE phase – Full reproducibility evaluation 

To minimize the effort required by reviewers and promote the future reuse of results, it is highly desirable to maximize automation throughout the entire process. The attainment of this goal can be facilitated by packaging the artifacts (such as with docker containers) and offering installation and deployment scripts. 

• Data artifact availability. In case your computational artifacts include any kind of dataset, it must be made stored in an *open*, public, persistent repository, such as Zenodo. These open repositories assigned a Digital Object Identifier that allows researchers to identify the dataset univocally. Reviewers must verify that the datasets have a DOI assigned to them. 
• Data artifact documentation. Then, the dataset must be documented. It can include methodological aspects about how the dataset was obtained, if there was any preprocessing and transformation, etc. Reviewers must verify that the data artifact was well documented. 
• Code artifact Hardware dependencies. Reviewers must verify that the specification of the hardware dependencies are complete, including type and quantity. It could be the case that the code artifact does not require any particular hardware requirements, then it should be stated. In some cases, the requirements can involve specific CPU, RAM, storage, or special accelerators such as GPUs. More information here (the description (documentation metadata) of the artifact(s)). 
• Code Artifact Software Dependencies. Reviewers must verify that the software dependencies are complete, including the OS required, any third-party libraries required. If using container technologies, as recommended, the documentation of this step can be easily accomplished by looking at the dockerfile. 
• Experimental Workflow. This is the *most important* step for the documentation of the artifact. In this section, authors must have included (i) a complete description of the experiment workflow that the code can execute, (ii) an estimation of the execution time to execute the experiment workflow, (iii) a complete description of the expected results and an evaluation of them, and most importantly (iv) how the expected results from the experiment workflow relate to the results found in the article. Best practices indicate that, to facilitate the understanding of the scope of the reproducibility, the expected results from the artifact should be in the same format as the ones in the article. For instance, when the results in the article are depicted in a graph figure, ideally, the execution of the code should provide a (similar) figure (there are open-source tools that can be used for that purpose such as gnuplot). It is critical that authors devote their efforts to these aspects of the reproducibility of experiments to minimize the time needed for their understanding and verification, for instance, by stating how the execution outcome matches the experiments in the paper

exactly and how long it takes to execute the experiments. Authors need to consider how long it is reasonable for reviewers and third-party researchers to wait for the experiments. For the cases of long run executions, it is advisable to provide an additional reduced-scale version of the experiment that can lead to analogous conclusions. 

• Code Artifact availability. In case your computational artifacts include any type of code, the code must be stored in an *open*, public, persistent repository. We recommend using github that can be subsequently paired with Zenodo. Zenodo provides a DOI to the code, so that it guarantees that it cannot be modified while the reviewers are examining it, and in the future, when third-party researchers and practitioners are reusing it. Reviewers must verify that the code artifacts have a DOI assigned to them. 
• Installation & Deployment.The installation and deployment processes provided by the authors should be easy to follow and take a reasonable amount of time. The documentation provided should also include (i) the process description to install and compile the libraries and the code, and (ii) the process description to deploy the code in the resources. The description of these processes should include an estimation of the installation, compilation, and deployment times. When any of these times exceed what is reasonable, authors should provide some way to alleviate the effort required by the potential recipients of the artifacts. For instance, capsules with the compiled code can be provided, or a simplified input dataset that reduces the overall experimental execution time. On the other hand, best practices indicate that, whenever it is possible, the actual code of software dependencies (libraries) should not be included in the artifact, but scripts should be provided to download them from a repository and perform the installation.