refactor: Update Pydantic v2 syntax and improve type hints

[tonykipkemboi]

Pydantic v2 Updates and Type Hint Improvements

Changes Made

1. Updated Pydantic v2 syntax:

   • Replaced Config class with model_config = ConfigDict in multiple tools
   • Updated validator decorator to field_validator for schema validation
   • Fixed deprecated syntax across the codebase

2. Improved type hints:

   • Enhanced custom_embedding_fn type in QdrantVectorSearchTool from callable to Callable[[str], list[float]]
   • Added proper type imports across tools

3. Documentation improvements:

   • Updated README.md links and formatting
   • Improved table of contents structure

Testing

• All tests pass except for known issues with Snowflake integration tests and Spider tool tests (which require API keys)
• No new test failures introduced by these changes

Impact

These changes ensure compatibility with Pydantic v2 and improve type safety across the codebase without introducing breaking changes.

[joaomdmoura]

Disclaimer: This review was made by a crew of AI Agents.

Code Review Comment for PR #217

Overview

This PR makes significant updates to enhance compatibility with Pydantic v2, improving type hints and overall code quality throughout various components of the codebase. Below is a structured analysis highlighting areas for improvement and the reasoning behind these suggestions.

Key Changes and Recommendations

1. Configuration Changes

• Files Affected: patronus_local_evaluator_tool.py, rag_tool.py
• Analysis: The transition from the Config class to ConfigDict is well executed. However, the implementation can be improved for readability and additional safeguards.
• Recommendation:

  # Current:
  model_config = ConfigDict(arbitrary_types_allowed=True)
  
  # Recommended:
  model_config = ConfigDict(
      arbitrary_types_allowed=True,
      validate_assignment=True  # Ensures assignments are validated
  )

2. Validator Updates

• Files Affected: scrapegraph_scrape_tool.py, selenium_scraping_tool.py, vision_tool.py
• Analysis: The update from validator to field_validator adheres to Pydantic v2 best practices, but implementations should include mode specifications and enhanced error handling.
• Recommendation:

  # Current:
  @field_validator("website_url")
  def validate_website_url(cls, v):
      if not v:
          raise ValueError("Website URL cannot be empty")
      return v
  
  # Recommended:
  from pydantic import field_validator, ValidationError
  
  @field_validator("website_url", mode="before")
  def validate_website_url(cls, v: str) -> str:
      if not v:
          raise ValueError("Website URL cannot be empty")
      try:
          result = urlparse(v)
          if not all([result.scheme, result.netloc]):
              raise ValueError("Invalid URL format")
          return v
      except Exception as e:
          raise ValueError(f"URL validation failed: {str(e)}")

3. Type Hints

• Files Affected: qdrant_search_tool.py
• Analysis: While the type hinting for custom_embedding_fn is a step forward, further enhancement could increase clarity.
• Recommendation:

  # Current:
  custom_embedding_fn: Optional[Callable[[str], list[float]]]
  
  # Recommended:
  from typing import Callable, Optional, List, Union
  import numpy as np
  
  custom_embedding_fn: Optional[Callable[[str], Union[List[float], np.ndarray]]] = Field(
      default=None,
      description="Custom embedding function returning vector embeddings as list or numpy array"
  )

4. Documentation Updates

• Files Affected: README.md
• Analysis: The documentation is improved but lacks clarity on version compatibility and migration from previous Pydantic versions.
• Recommendation: Add a section on version compatibility:

  ## Version Compatibility
  
  This package requires:
  - Python 3.8+
  - Pydantic >= 2.0
  - crewAI >= 0.9.0
  
  ### Breaking Changes
  If you're upgrading from an older version using Pydantic v1, please note:
  - Replace all `Config` classes with `model_config`
  - Update validators to use `field_validator`
  - Review type hints for compatibility

5. General Recommendations

• Version Constraints: Update pyproject.toml for proper dependency management.

  [tool.poetry.dependencies]
  pydantic = ">=2.0.0"
  typing-extensions = ">=4.0.0"

• Type Checking: Standardize type checking at the file level.

  from typing import TYPE_CHECKING, cast
  
  if TYPE_CHECKING:
      from pydantic import BaseModel

• Validation Decorators: Consider using validation decorators for cross-field validations.

  from pydantic import field_validator, model_validator
  
  @model_validator(mode='after')
  def validate_model(self) -> 'YourModel':
      # Implement cross-field validation logic
      return self

6. Testing Recommendations

• Add tests to ensure new features and validations work as intended:

  import pytest
  from pydantic import ValidationError
  
  def test_config_validation():
      with pytest.raises(ValidationError):
          YourModel(invalid_field="test")
  
  def test_field_validator():
      model = YourModel(website_url="https://valid-url.com")
      assert model.website_url == "https://valid-url.com"

Summary of Critical Changes Needed

1. Implement explicit error handling in all field validators.
2. Enhance type hints with clearer union types.
3. Introduce comprehensive version compatibility documentation.
4. Implement cross-field validation to prevent inconsistent model states.
5. Strengthen test coverage around Pydantic v2 features.

The updates in this PR contribute positively to improving the codebase's compatibility and maintainability. Addressing the above recommendations will further elevate the code quality, ensuring robustness and clarity in future developments.