Reviewing documents with one or more reviewers

In Sysrev, the review of documents can be done by a single reviewer or collaboratively. For collaborations, Sysrev makes possible blind review (where reviewers do not see the decisions or labels of other reviewers), and provides different review modes to facilitate single or double screening of documents. It also makes it easy to identify where conflicts between reviewers exist, so those conflicts can be resolved by discussion, consensus or a third reviewer.

Review options and settings

There are a number of settings that you can adjust to manage your screening and review process. These are found under Manage -> Settings, in the Options box.

• Article review priority: When reviewers start to look at articles in the Articles tab, they will be automatically presented with an article. The Article Review Priority will determine if the articles provided to reviewers are not yet screened by anyone (Single), screened by one reviewer already (Full), or a mix of these two (Balanced). In this way, you can manage a collaborative review with either single or double screening.
• Review limit: This setting can allow for reviews in which you want more than two reviewers (i.e. Entire team) per document.
• Prefill review answers: When toggled ON, the Auto-labeler will automatically prefill the label answers for each record it processes. In this way, human reviewers can then use the Review tab to quickly review, verify and approve Auto-label answers.
• Submit on completion: This option can be toggled on when reviewers prefer to not have to click Save after completing the required labels. If toggled on, once all REQUIRED labels are completed, they will automatically be saved and the next record will appear. This is probably not a good option if you have some optional labels, or a complex label structure.
• Label Blinding: Toggling this label on will make the decisions of other reviewers invisible. This is a good option for most reviews in which double screening is carried out. This setting can be turned off during conflict resolution.

Once you have adjusted your review settings, don't forget to click the Save Changes button at the bottom of the Options box!

How to review documents

Any member of the project can review documents (i.e. assign labels to documents). To do so, from the main project page, click the Review tab. On the left-hand side, all of the enabled labels will appear. Once all required labels are answered by the reviewer, the blue Save button will become available. Click Save to record label answers. The next record for review will automatically appear in the window. Reviewers can also choose to skip the record.

Note that depending on the Article Review Priority settings described above, the records will be prioritized by whether or not they have already been reviewed by another reviewer. Other than this sorting, the records are randomly assigned.

Note that for the default Include label, reviewers can only choose Yes or No, or skip the record. While you can click on the question mark, that will not register as an answer..

Filtered review

When you set an article filter on the Articles tab, the Review tab becomes Filtered Review. In this way, you can review a subset of articles in your Sysrev project. Common use cases for Filtered Review are:

• Reviewing articles based on the prediction model: After multiple reviewers have labeled articles, the Sysrev prediction model will apply a score indicating the likelihood of a given label answer for the remaining records, for all Boolean and categorical labels. You can then filter, for example, to only articles with a 50% or higher likelihood of inclusion (or exclusion) and then click Filtered Review to focus on relevant (or irrelevant) records. 
• Reviewing articles based on auto-label answers: After running the auto-label on unreviewed articles, you can  screen records in batches based on auto-label answers. For example, you may want to focus screening on records for intervention A, before moving on to intervention B. 

Note that once a user has review an article for inclusion, that article will no longer appear in the Review (or Filtered Review) tab for that user, even if new (required or unrequired) labels are added. 

Also note, reviewing articles in a Filtered Review can affect the prediction model’s scores. Each time a user saves their labeling decisions, the list of articles in the Filtered Review will be updated. As a result, some articles may remain in the list, while others may be removed, depending on how the prediction model is updated.