Proposal for amending CHI guides for authors and reviewers

The text below is a community-led effort to improve the Guide to a Successful Submission on the CHI conference website. We are currently working with the CHI Steering Committee to facilitate further revision and adoption of this proposal. We will periodically update the progress on our blog.

Transparency

CHI papers should strive for research transparency regardless of the contribution type and methodology. Different contribution types, (e.g. technical contributions, quantitative studies, and qualitative studies) use different criteria for assessing transparency.

Contributions that are technology-oriented (e.g., a new technique or algorithm) and contributions that are quantitative studies (i.e., experiments with statistically analyzed results) are expected to be verifiable, reproducible (e.g., others should be able to rerun the interactive system or rerun the analysis code with the original data) and replicable (e.g., others should be able to independently recreate the interactive system or rerun the same experiment with different participants). Papers with these contributions should include enough detail for an independent researcher or practitioner to (1) independently evaluate the correctness, validity, and reliability of your software and/or analyses and (2) reproduce and replicate both core technology and experimental methods.

Algorithms and statistical analyses should be described with significant detail. Wherever possible, it’s fine to save space by referring the reader to prior work for particular steps in your analysis, so long as the overall approach remains readable. Pseudocode is extremely helpful where algorithmic contributions are involved.

Transparency is often a great area for “beta-testing” your paper with a colleague or friend. Ask a colleague to read your paper and list back the important steps you used in data collection and analysis. Did he or she leave any steps out? If so, you may need to add more detail or appropriate references.

While some independent researchers may have difficulty fully replicating your work — e.g., if the work requires access to unique user populations or rare or expensive hardware — an independent researcher who has access to these resources should ideally be able to reproduce your work.

Contributions that follow a qualitative research approach (i.e., which most of the time incorporate researchers’ subjective interpretation as part of the method) should be transparent about the various decisions made, and the procedures followed in the design of the research study and reporting of findings. This should include clear explanations of and justifications for the theoretical or conceptual basis for the study, choice of methods employed in every stage of the study, participant-selection process, and procedures followed for data collection and analysis. Researchers should also describe their considerations of the ethical concerns in the study, such as those pertaining to participant anonymity, privacy, and consent, their roles in the study, and data gathering and use. In cases where necessary prior permissions have been obtained to disclose any of the collected data (e.g., observation notes and interview transcripts) and documented researcher notes, making these data available would be welcome additions to the contributions.

The reporting of qualitative research findings should strive to show the “big picture” while also sufficiently contextualizing individual findings. The authors should make explicit how the themes were identified or constructed from the data, and whether each conclusion was drawn from outstanding instances or general trend among participants. They should also articulate any assumptions, preconceptions, or potential biases of the researchers. Communicating the research process in sufficient detail will enable reviewers to assess the rigor of the studies and empower others researchers to adopt the approaches, extend the work, and transfer the findings to other similar settings.

Sharing research material: While the paper should provide as much information as possible to enable verification, reproduction, and replication, some details such as source code, analysis code, detailed hardware specifications, interview protocols, and collected data may not be shareable within the paper itself. Reviewers welcome and even expect all such material to be available. These resources are most reliably shared by posting to a publicly available open-access repository with a persistent identifier (e.g., a registration on the Open Science Framework, an open-access university repository, or an independent repository listed on www.re3data.org). Note that the ACM policy does not limit the use of specific repositories for the purpose of archiving supplementary materials, and that some repositories, including the Open Science Framework, allow anonymous posting of materials for reviewers. In some situations, you may not be able to share material such as sensitive data or proprietary code. In these cases, we advise you to share as much as possible and explicitly state in your paper why the rest cannot be shared. For example, while code for novel algorithms or designs may be protected by intellectual property, code for analyzing study data rarely requires protection, and access to this analysis code can be crucial for assessing the validity of your study’s conclusions. While we don’t expect you to share sensitive data or proprietary code, we encourage you to share as much non-sensitive and non-proprietary code as possible to help reviewers scrutinize, replicate and reproduce your results. This will increase the chances of your paper getting accepted.