---
title: "Suggested workflow for testing and running the Auto-labeler"
url: "https://help.sysrev.com/en/article/suggested-workflow-for-testing-and-running-auto-labels-copy"
language: "en"
last_updated: "2025-09-17T15:38:30.790Z"
---

# Suggested workflow for testing and running the Auto-labeler

In general, before running the Auto-labeler across the entirety of your project documents, you should test, assess and optimize your Auto-label on a small random sample of records. Here is a recommended workflow for optimizing and then using the Auto-labeler for a literature or document review project.

1.  **Two human reviewers label records in a project.** Go to Manage and then the Review Settings section of Settings and select Full for Article Review Priority. This allows you to quickly accumulate a number of double-reviewed records. Consider skipping records without abstracts to avoid running the Auto-labeler on incomplete metadata.
2.  **Toggle off Prefill review answers:** This is found in the Review Settings section of Manage -> Settings. We advise turning this off while refining your label prompts, and turning it on once label prompts are finalized and you are ready to Auto-label all records.
3.  **Set article filter to only double-reviewed records:** To achieve this, set the following two labels:
    1.  Match -> Filter Type: Consensus + Consensus Status: Determined + Inclusion: Any
    2.  Exclude -> Filter Type: Consensus + Consensus Status: Single + Inclusion: Any
4.  **Ensure that your label Auto-label settings are correct:** Review each active label to ensure that only desired labels have the Auto-label Extraction box checked, and that Full Text is set to the desired content (citations only vs. attachments). **Note:** This is important for ensuring that you don't accidentally overspend your budget. You can also turn on or off "Reasoning with Confidence". Turning this on will add [a set of reasoning steps](https://help.sysrev.com/en/article/probability-reasoning) to your prompt and may result in changes in Auto-labeler performance. This will also provide some insight into why the Auto-labeler chose its answer. This can be helpful in the prompt engineering stage. Decide which model (Gemini or OpenAI) you want to used for each label.
5.  **Set Auto-label Max Articles to desired number of records for initial assessment:** This setting will work through the filtered articles in order up to the max number provided. We recommend auto labeling at least 20 double-reviewed articles at a time while testing and refining Auto-labeling prompts.
6.  **Run the Auto-labeler.**
7.  **Review the Auto-label Report:** This will appear at the bottom of the Overview page. More information about reading and using the report is in the box below.
8.  **Refine label question prompts:** See below for tips on improving your label question prompts.
9.  **Set article filter for a new set of double-labeled records:** If you want to test your revised prompts on a new set of records (recommended to avoid over-engineering your prompts to one small set of articles), you can add an additional filter to the already filtered set of double-screened records: Exclude -> Filter Type: Auto-label + Label: Any label. Alternatively, you can increase the Max Articles setting in the Auto-labeler to label both previously auto-labeled records and a set of new records.
10.  **Review the Auto-label Report** and repeat steps 8-10 until you have reached a comfortable accuracy score.
11.  **Auto-label the remaining records in the project:** At this point, you can set Prefill review answers to Yes (in the Review Settings section of Manage -> Settings). Run the Auto-labeler for all records by removing filters and increasing the Max Articles setting. Once complete, reviewers can review articles to check the Auto-label answers to ensure accuracy.