Generating Bug Free Leetcode Solutions

note

To download this tutorial as a Jupyter notebook, click here.

In this example, we want to solve String Manipulation leetcode problems such that the code is bug free.

We make the assumption that:

1. We don't need any external libraries that are not already installed in the environment.
2. We are able to execute the code in the environment.

Objective

We want to generate bug-free code for solving leetcode problems. In this example, we don't account for semantic bugs, only for syntactic bugs.

In short, we want to make sure that the code can be executed without any errors.

Step 1: Install validators from the hub

First, we install the validators and packages we need to make sure generated python is valid.

guardrails hub install hub://reflex/valid_python --quiet

    Installing hub://reflex/valid_python...
    ✅Successfully installed reflex/valid_python!
    
    

Step 2: Create a Guard object

The Guard object contains the validations we aim to check the generated code against. This object also takes corrective action to fix the code if it doesn't pass the validations. As configured here, it will reask the LLM to correct the code.

from guardrails.hub import ValidPython
from guardrails import Guard

guard = Guard().use(ValidPython(on_fail="reask"))

Step 3: Use the Guard to make and validate the LLM API call

# Add your OPENAI_API_KEY as an environment variable if it's not already set
# import os
# os.environ["OPENAI_API_KEY"] = "YOUR_API_KEY"

prompt = """
Given the following high level leetcode problem description, write a short Python code snippet that solves the problem.
Do not include any markdown in the response.

Problem Description:
${leetcode_problem}
"""

leetcode_problem = """
Given a string s, find the longest palindromic substring in s. You may assume that the maximum length of s is 1000.
"""

response = guard(
    model="gpt-4o",
    messages=[{
        "role": "user",
        "content": prompt
    }],
    prompt_params={"leetcode_problem": leetcode_problem},
    temperature=0
)

print(response)

    ValidationOutcome(
        call_id='14508958944',
        raw_llm_output='def longest_palindrome(s: str) -> str:\n    if len(s) == 0:\n        return ""\n    \n    start, end = 0, 0\n    \n    def expand_around_center(left: int, right: int) -> int:\n        while left >= 0 and right < len(s) and s[left] == s[right]:\n            left -= 1\n            right += 1\n        return right - left - 1\n    \n    for i in range(len(s)):\n        len1 = expand_around_center(i, i)\n        len2 = expand_around_center(i, i + 1)\n        max_len = max(len1, len2)\n        \n        if max_len > end - start:\n            start = i - (max_len - 1) // 2\n            end = i + max_len // 2\n    \n    return s[start:end + 1]',
        validated_output='def longest_palindrome(s: str) -> str:\n    if len(s) == 0:\n        return ""\n    \n    start, end = 0, 0\n    \n    def expand_around_center(left: int, right: int) -> int:\n        while left >= 0 and right < len(s) and s[left] == s[right]:\n            left -= 1\n            right += 1\n        return right - left - 1\n    \n    for i in range(len(s)):\n        len1 = expand_around_center(i, i)\n        len2 = expand_around_center(i, i + 1)\n        max_len = max(len1, len2)\n        \n        if max_len > end - start:\n            start = i - (max_len - 1) // 2\n            end = i + max_len // 2\n    \n    return s[start:end + 1]',
        reask=None,
        validation_passed=True,
        error=None
    )


    /Users/dtam/dev/guardrails/guardrails/validator_service/__init__.py:85: UserWarning: Could not obtain an event loop. Falling back to synchronous validation.
      warnings.warn(

The response above shows a brief summary of what the Guard did. We can destructure it to show the final output:

print(response.validated_output)

    def longest_palindrome(s: str) -> str:
        if len(s) == 0:
            return ""
        
        start, end = 0, 0
        
        def expand_around_center(left: int, right: int) -> int:
            while left >= 0 and right < len(s) and s[left] == s[right]:
                left -= 1
                right += 1
            return right - left - 1
        
        for i in range(len(s)):
            len1 = expand_around_center(i, i)
            len2 = expand_around_center(i, i + 1)
            max_len = max(len1, len2)
            
            if max_len > end - start:
                start = i - (max_len - 1) // 2
                end = i + max_len // 2
        
        return s[start:end + 1]

We can confirm that the code is bug free by executing the code in the environment.

try:
    exec(response.validated_output)
    print("Success!")
except Exception as e:
    print("Failed!")

    Success!