Suggested workflow for testing and running auto labels

In general, before running the auto labeller across the entirety of your project documents, you should test, assess and optimize your auto labels on a small random sample of records. Here is a recommended workflow for optimizing and then using the auto labeller for a literature or document review project.

1. Two human reviewers label records in a project. Go to Manage and then the Options section of Settings and select Full for Article Review Priority. This allows you to quickly accumulate a number of double-reviewed records. Consider skipping records without abstracts to avoid running the auto labeller on incomplete metadata.
2. Set Prefill review answers to No: This is found in the Options section of Manage -> Settings. We advise turning this off while refining your label prompts, and turning it on once label prompts are finalized and you are ready to auto label all records.
3. Set article filter to only double-reviewed records: To achieve this, set the following two labels:
   1. Match -> Filter Type: Consensus + Consensus Status: Determined + Inclusion: Any
   2. Exclude -> Filter Type: Consensus + Consensus Status: Single + Inclusion: Any
4. Ensure that your label auto label settings are correct: Review each active label to ensure that only desired labels have the Auto Label Extraction box checked, and that Full Text is set to the desired content (citations only vs. attachments). Note: This is important for ensuring that you don't accidentally overspend your budget. You can also turn on or off "Probability and Reasoning". Turning this on will add a set of reasoning steps to your prompt and may result in changes in auto labeller performance. This will also provide some insight into why the auto labeller chose its answer. This can be helpful in the prompt engineering stage.
5. Set auto label Max Articles to desired number of records for initial assessment: This setting will work through the filtered articles in order up to the max number provided. We recommend auto labelling at least 20 double-reviewed articles at a time while testing and refining auto labelling prompts.
6. Run the auto labeller.
7. Review the Auto Label Report: This will appear at the bottom of the Overview page. More information about reading and using the report is in the box below.
8. Refine label question prompts: See below for tips on improving your label question prompts.
9. Set article filter for a new set of double-labeled records: If you want to test your revised prompts on a new set of records (recommended to avoid over-engineering your prompts to one small set of articles), you can add an additional filter to the already filtered set of double-screened records: Exclude -> Filter Type: Auto label + Label: Any label. Alternatively, you can increase the Max Articles setting in the auto labeller to label both previously auto labelled records and a set of new records.
10. Review the Auto Label Report and repeat steps 8-10 until you have reached a comfortable accuracy score.
11. Auto label the remaining records in the project: At this point, you can set Prefill review answers to Yes (in the Options section of Manage -> Settings). Run the auto labeller for all records by removing filters and increasing the Max Articles setting. Once complete, reviewers can review articles to check the auto label answers to ensure accuracy.