Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: remove model_weak configuration and simplify model selection #1392

Merged
merged 1 commit into from
Dec 11, 2024

Conversation

mrT23
Copy link
Collaborator

@mrT23 mrT23 commented Dec 11, 2024

PR Type

enhancement, documentation


Description

  • Removed the model_weak configuration from the codebase and documentation, simplifying model selection.
  • Introduced a new utility function get_weak_model to handle weak model retrieval.
  • Updated the default model type in retry_with_fallback_models to ModelType.REGULAR.
  • Adjusted documentation to reflect the removal of model_weak and updated configuration examples.

Changes walkthrough 📝

Relevant files
Enhancement
pr_processing.py
Simplify model selection and fallback logic                           

pr_agent/algo/pr_processing.py

  • Added get_weak_model function to import statements.
  • Changed default model_type to ModelType.REGULAR in
    retry_with_fallback_models.
  • Simplified logic for obtaining weak models using get_weak_model.
  • +4/-4     
    utils.py
    Add utility function for weak model retrieval                       

    pr_agent/algo/utils.py

  • Added get_weak_model function to retrieve weak model configuration.
  • +6/-0     
    Documentation
    changing_a_model.md
    Update documentation to remove model_weak configuration   

    docs/docs/usage-guide/changing_a_model.md

  • Removed references to model_weak configuration.
  • Updated examples to reflect removal of model_weak.
  • +2/-13   
    Configuration changes
    configuration.toml
    Comment out model_weak in configuration                                   

    pr_agent/settings/configuration.toml

    • Commented out model_weak configuration.
    +1/-1     

    💡 PR-Agent usage: Comment /help "your question" on any pull request to receive relevant information

    Copy link
    Contributor

    PR Reviewer Guide 🔍

    Here are some key observations to aid the review process:

    ⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
    🏅 Score: 95
    🧪 No relevant tests
    🔒 No security concerns identified
    ⚡ Recommended focus areas for review

    Error Handling
    The get_weak_model function should handle the case when config.model_weak is None or invalid

    Code Clarity
    The else clause in _get_all_models function has no matching if condition, making the code flow unclear

    Copy link
    Contributor

    PR Code Suggestions ✨

    Explore these optional code suggestions:

    CategorySuggestion                                                                                                                                    Score
    Possible issue
    Add validation to ensure a model configuration exists before attempting to use it

    Add error handling for the case when both model_weak and model configurations are
    missing. This prevents potential None returns or configuration errors.

    pr_agent/algo/utils.py [30-33]

     def get_weak_model() -> str:
         if get_settings().get("config.model_weak"):
             return get_settings().config.model_weak
    +    if not get_settings().config.model:
    +        raise ValueError("No model configuration found")
         return get_settings().config.model
    • Apply this suggestion
    Suggestion importance[1-10]: 7

    Why: The suggestion adds important error handling to prevent silent failures when model configuration is missing, which could lead to runtime errors downstream.

    7
    Add error handling for model configuration retrieval failures

    Handle potential exceptions from get_weak_model() to prevent crashes when model
    configurations are invalid.

    pr_agent/algo/pr_processing.py [357-360]

    -if model_type == ModelType.WEAK:
    -    model = get_weak_model()
    -else:
    -    model = get_settings().config.model
    +try:
    +    if model_type == ModelType.WEAK:
    +        model = get_weak_model()
    +    else:
    +        model = get_settings().config.model
    +except ValueError as e:
    +    raise Exception(f"Failed to get model configuration: {str(e)}")
    • Apply this suggestion
    Suggestion importance[1-10]: 6

    Why: The suggestion improves error handling by catching and re-raising exceptions from get_weak_model() with more context, making debugging easier.

    6
    • Author self-review: I have reviewed the PR code suggestions, and addressed the relevant ones.

    💡 Need additional feedback ? start a PR chat

    model="gpt-4o-2024-11-20"
    fallback_models=["gpt-4o-2024-08-06"]
    #model_weak="gpt-4o-mini-2024-07-18" # optional, a weaker model to use for some easier tasks
    Copy link
    Collaborator Author

    @mrT23 mrT23 Dec 11, 2024

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    @KennyDizi
    some additional adjustments 1:
    I want the model_weak to be optional. only if the user actively defines it, it will be used. in general, most users prefer quality over cost reduction

    @@ -27,6 +27,12 @@
    from pr_agent.log import get_logger


    def get_weak_model() -> str:
    Copy link
    Collaborator Author

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    @KennyDizi
    some additional adjustments 2:
    fallback to a case where the weak model is not defined

    @@ -333,7 +333,7 @@ def generate_full_patch(convert_hunks_to_line_numbers, file_dict, max_tokens_mod
    return total_tokens, patches, remaining_files_list_new, files_in_patch_list


    async def retry_with_fallback_models(f: Callable, model_type: ModelType = ModelType.WEAK):
    async def retry_with_fallback_models(f: Callable, model_type: ModelType = ModelType.REGULAR):
    Copy link
    Collaborator Author

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    some additional adjustments 3:

    This is a fix. The default should be REGULAR

    Copy link
    Contributor

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    I'm wondering because my commit already set the default model to REGULAR as your suggestion KennyDizi@6352e6e

    @mrT23 mrT23 merged commit f67cc0d into main Dec 11, 2024
    2 checks passed
    @mrT23 mrT23 deleted the tr/model_weak branch December 11, 2024 16:19
    @mrT23
    Copy link
    Collaborator Author

    mrT23 commented Dec 11, 2024

    @KennyDizi if you think additional adjustments are needed, do share.
    I merged because i want 'main' to be "always stable", and the model_weak should be optional

    @KennyDizi
    Copy link
    Contributor

    Thanks for the heads up @mrT23. This is a good PR to make model_weak more sense.

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Projects
    None yet
    Development

    Successfully merging this pull request may close these issues.

    3 participants