Validator OnFail Actions

OnFail Actions

Validators ship with several out of the box on_fail policies. The OnFailAction specifies the corrective action that should be taken if the quality criteria is not met. The corrective action can be one of the following:

Action	Behavior
OnFailAction.REASK	Reask the LLM to generate an output that meets the correctness criteria specified in the validator. The prompt used for reasking contains information about which quality criteria failed, which is auto-generated by the validator.
OnFailAction.FIX	Programmatically fix the generated output to meet the correctness criteria when possible. E.g. the formatter provenance_llm validator will remove any sentences that are estimated to be hallucinated.
OnFailAction.FILTER	(Only applicable for structured data validation) Filter the incorrect value. This only filters the field that fails, and will return the rest of the generated output.
OnFailAction.REFRAIN	Refrain from returning an output. This is useful when the generated output is not safe to return, in which case a None value is returned instead.
OnFailAction.NOOP	Do nothing. The failure will still be recorded in the logs, but no corrective action will be taken.
OnFailAction.EXCEPTION	Raise an exception when validation fails.
OnFailAction.FIX_REASK	First, fix the generated output deterministically, and then rerun validation with the deterministically fixed output. If validation fails, then perform reasking.
OnFailAction.CUSTOM	This action is set internally when the validator is passed a custom function to handle failures. The function is called with the value that failed validation and the FailResult returned from the Validator. i.e. the custom on fail handler must implement the method signature def on_fail(value: Any, fail_result: FailResult) -> Any

Example

Let's assess the output of a trivial validator in diffent cases of OnFailAction. Take the following Validator a basic implementation of a Toxic Language Validator:

TOXIC_WORDS = ["asshole", "damn"]

class BasicToxicLanguage(Validator):
    def validate(self, value: Any, metadata: Dict) -> ValidationResult:
        is_toxic_language = any(toxic_word in value for toxic_word in TOXIC_WORDS)

        # if a value contains toxic words we return FailResult otherwise PassResult
        if is_toxic_language:
            for toxic_word in TOXIC_WORDS:
                value = value.replace(toxic_word, "")
            return FailResult(
                error_message=f"Value '{value}' contains toxic language including words: {TOXIC_WORDS} which is not allowed.",
                fix_value=value,
            )

        return PassResult()

For more information on how to write Custom Validators refer to our guide here

Now suppose some unhinged LLM returns damn you!, in ths scenario:

• OnFailAction.REASK an LLM will be reasked to correct it's output based on the error_message provided to FailResult. In this example it will reask the LLM with a reask prompt which includes the error message: Value 'damn you!' contains toxic language including words: ["asshole","damn"] which is not allowed.. You can set num_reasks on the guard() call to determine how many times we retry.
• OnFailAction.FIX the value is replaced with fix_value that is provided to FailResult. In this example value from the LLM damn you! will be returned as you!.
• OnFailAction.FILTER if used in structured data generation we do not display the field that fails validation (more on this below). In this example value from the LLM damn you! will return an empty response.
• OnFailAction.REFRAIN we do not return anything as the validation is deemed unsafe for end users. In this example value from the LLM damn you! will return an empty response.
• OnFailAction.NOOP the value is returned as is and failures are logged in the history. In this example value we return the value as is damn you!
• OnFailAction.EXCEPTION during a guard execution or direct validation call we raise an error indicating the validation failed.
• OnFailAction.FIX_REASK during a fix reask, we first perform the same action as OnFailAction.FIX and re-validate the output. If it fails we run a OnFailAction.REASK action, otherwise we return the passed validation.

Structured Data

Using OnFail actions is powerful when also working with structured data as we can determine how to treat each field's validation failure.

prompt = """
Given the following fast food order, please provide a summary of the orders.
${order}
${gr.complete_xml_suffix_v2}
"""

order = """I want a burger with two large fries and a coke zero."""

# MinimumOneRange is a hypothetical custom validator that an integer > 0 is supplied
class Lineitem(BaseModel):
    item: str = Field(description="The name of the item being ordered", validators=[LowerCase()])
    quantity: int = Field(description="The quantity of the item being ordered", validators=[MinimumOneRange(min=1, max=10, on_fail="fix")])

guard = Guard.from_pydantic(output_class=List[Lineitem])

response = guard(
    model="gpt-4o",
    messages=[{
        "role": "system",
        "content": "You are a helpful assistant."
    },{
        "role": "user",
        "content": prompt
    }],
    prompt_params={"order": order},
)

print(response.validated_output)

# [{'item': 'burger', 'quantity': 1},
#  {'item': 'fries', 'quantity': 2},
#  {'item': 'coke zero', 'quantity': 1}]