Understanding the auto label report

Once the auto labeler has been run, an Auto Label report will be generated. The report compares the accuracy of the auto labeler against user generated values. If both a user and the auto labeler review an article, the label analysis will appear in the report.

Note that the report will only be generated if the auto labeler was run on labels and records for which there has already been some human reviewer activity.

The Auto Label report can be found at the very bottom of the project Overview page. A donut chart visualization provides a quick snapshot of the performance of the auto labeler compared to user labels. Clicking on the See the full report link will take you to the full report.

The Auto Label report overview

The graphs at the top of the report provide a snapshot of the of the performance of the auto labeler compared to user labels. In addition to the donut chart, bar graphs provide the number of disagreements and agreements between each user and the auto labeler.

Report scope

You can view the results for the most recent run of the auto labeler (i.e., showing results for only the records and labels that were used in the latest run) or all auto labeler runs (i.e., the most recent results for every label and record for which the auto labeler has been used) by clicking the appropriate box under Report scope. 

Performance by user

This section provides details of true and false positives and negatives for each user, as well as a number of performance metrics:

• Recall: ratio of true positives to the total actual positives
• Precision: ratio of true positives to the total predicted positives
• Specificity: ratio of true negatives to the total actual negatives
• Accuracy: ratio of the sum of true positives and true negatives to the total number of instances

The numbers of true and false positives and negatives are hyperlinks which will open a new window showing the corresponding records. Reviewing these will help you determine what adjustments might be useful to improve auto labeler accuracy (or to improve human reviewing, as the auto labeler can sometimes indicate systematic errors by human reviewers).

Performance by label

This section summarizes the true and false positives and negatives for each label, as well as the performance metrics described above, at the label level for all user answers combined.

Label values by article

This section provide a summary at the article level of user and auto label answers. Click on the arrows in the bottom right corner to scroll through these articles and click on the article ID in the first column to link directly to the article in Sysrev.