LLM Codebase Optimization Testing Framework

This framework allows you to empirically test how different code organization and documentation strategies impact LLM performance using Anthropic's Sonnet model.

Setup

1. Install dependencies:

pip install anthropic

2. Set up your Anthropic API key:

export ANTHROPIC_API_KEY='your-api-key-here'

3. Configure settings in config.json (optional):

• Adjust model parameters (max tokens, temperature)
• Modify testing parameters (retry attempts, timeouts)
• Configure metrics collection settings

4. Run the test:

python example_test.py

Recommendation for Rewriting Code to be LLM Optimized

Use this prompt to rewrite code to be LLM Optimized:

Please rewrite the following code to be optimally parseable by an LLM while maintaining identical functionality and method signatures. Follow these guidelines:

1. Structure:
   - Use clear, logical code organization
   - Add descriptive but concise docstrings that explain purpose and behavior
   - Break complex operations into well-named helper functions
   - Maintain consistent indentation and formatting

2. Naming and Documentation:
   - Keep all existing method names unchanged
   - Use self-documenting variable names
   - Include type hints for parameters and return values
   - Add brief inline comments only for non-obvious logic

3. Optimization Guidelines:
   - Remove redundant or unnecessary comments
   - Eliminate dead code and unused imports
   - Simplify complex conditionals
   - Use pythonic patterns where they improve readability
   - Avoid unnecessary abstraction layers

4. LLM-Specific Considerations:
   - Structure code to make dependencies and data flow clear
   - Use standard library functions where possible
   - Keep line lengths reasonable (≤100 characters)
   - Group related functionality together
   - Minimize nested code blocks
   - Add clear structure markers
   - Add comments that benefit AI models, if they only benefit humans do not include

5. Performance Balance:
   - Maintain O(n) complexity where possible
   - Don't sacrifice significant performance for readability
   - Keep memory usage efficient

Please provide the optimized code while preserving all existing functionality and external interfaces.

Usage

1. Create a test case:

test_case = {
    "name": "Your Test Case",
    "description": "Description of what you're testing",
    "standard_code": "Your baseline code here",
    "optimized_code": "Your LLM-optimized code here",
    "tasks": [
        "Task 1 description",
        "Task 2 description"
    ]
}

2. Run the test:

from llm_optimization_tester import LLMOptimizationTester

tester = LLMOptimizationTester()
results = tester.run_comparison_test(test_case)
report = tester.generate_report(results)

Metrics Tracked

• Completion Time: Time taken to understand and complete tasks
• Accuracy Score: Quality of solutions provided
• Context Questions: Number of clarifying questions needed
• Token Usage: API token consumption for each version

Example Output

{
    "performance_comparison": {
        "time_difference": -2.5,
        "accuracy_improvement": 0.15,
        "context_questions_reduced": 3,
        "token_usage_difference": -150
    },
    "conclusion": "Detailed analysis of optimization effectiveness"
}

Best Practices

1. Test Case Design

   • Use realistic code samples
   • Include diverse tasks (analysis, modification, bug fixing)
   • Provide clear task descriptions

2. Code Optimization

   • Follow guidelines in llm-codebase-optimization.md
   • Maintain consistency in documentation format
   • Balance human readability with LLM optimization

3. Results Analysis

   • Consider all metrics holistically
   • Account for task complexity
   • Track trends across multiple test runs

Contributing

Feel free to contribute by:

• Adding new test cases
• Improving metrics calculation
• Enhancing documentation standards
• Implementing additional analysis tools

License

MIT License - See LICENSE file for details