Reasoning with Confidence

When running the Auto-labeler, you have the option to turn on Reasoning with Confidence. This setting adds a set of pre-configured chain-of-thought prompts to your custom label prompt. The LLM will work through a series of reasoning steps as follows, providing responses to each in it's final answer:

Follow these instructions. Do not repeat the instructions in the steps, just supply your own response.

• Step 1: Given the original label description and your answers, rephrase and expand on your explanations to help you do better answering. Maintain all information in the original label descriptions. Output your thoughts.
• Step 2: Using your knowledge of the world and topic, as well as the information provided, provide a few reasons for each label why the answer might be incorrect. Rate the strength of each reason. Output your thoughts.
• Step 3: Using your knowledge of the world and topic, as well as the information provided, provide a few reasons for each label why the answer might be correct. Rate the strength of each reason. Output your thoughts.
• Step 4: Aggregate your considerations. Think like a research assistant. Output your thoughts.
• Step 5: Output an initial probability (prediction) as a number between 0 and 1 given steps 1-4.
• Step 6: Evaluate whether your calculated probability is excessively confident or not confident enough. If you were only provided a citation or abstract and not the article full text, consider the possibility that the full text could provide relevant information. Also, consider anything else that might affect the prediction that you did not before consider (e.g. base rate of occurrence). Output your thoughts.
• Step 7: Output your final label value with no additional formatting.
• Step 8: Output an explanation for your final answer.
• Step 9: Output the probability for your final answer (a number between 0 and 1).

When using the Reasoning with Confidence setting, you may gain accuracy in your Auto-label answers and you will get a better sense of how the LLM is interpreting and implementing your prompt. Due to the increased output however, there is an additional charge for each record and label when this setting is turned on. 

We recommended testing a few records with and without the Reasoning with Confidence to get a better sense of the value it adds to your particularly review project.