Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Integration][bitbucket] Added bitbucket ocean integration #1448

Open
wants to merge 13 commits into
base: main
Choose a base branch
from

Conversation

oiadebayo
Copy link
Member

@oiadebayo oiadebayo commented Feb 28, 2025

User description

Description

What - Created a new Ocean integration for Bitbucket that enables ingestion of Bitbucket resources into Port's catalog. The integration supports:

  • Projects kind
  • Repository kind
  • Pull Request kind
  • Folder kind

Why - To enable users using Bitbucket cloud version to automatically sync their resources to Port

How -

Type of change

Please leave one option from the following and delete the rest:

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • New Integration (non-breaking change which adds a new integration)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Non-breaking change (fix of existing functionality that will not change current behavior)
  • Documentation (added/updated documentation)

All tests should be run against the port production environment(using a testing org).

Core testing checklist

  • Integration able to create all default resources from scratch
  • Resync finishes successfully
  • Resync able to create entities
  • Resync able to update entities
  • Resync able to detect and delete entities
  • Scheduled resync able to abort existing resync and start a new one
  • Tested with at least 2 integrations from scratch
  • Tested with Kafka and Polling event listeners
  • Tested deletion of entities that don't pass the selector

Integration testing checklist

  • Integration able to create all default resources from scratch
  • Resync able to create entities
  • Resync able to update entities
  • Resync able to detect and delete entities
  • Resync finishes successfully
  • If new resource kind is added or updated in the integration, add example raw data, mapping and expected result to the examples folder in the integration directory.
  • If resource kind is updated, run the integration with the example data and check if the expected result is achieved
  • If new resource kind is added or updated, validate that live-events for that resource are working as expected
  • Docs PR link here

Preflight checklist

  • Handled rate limiting
  • Handled pagination
  • Implemented the code in async
  • Support Multi account

Screenshots

Include screenshots from your environment showing how the resources of the integration will look.

API Documentation

Provide links to the API documentation used for this integration.


PR Type

Enhancement, Tests, Documentation


Description

  • Added a new Bitbucket integration for syncing resources.

  • Implemented support for Projects, Repositories, Folders, and Pull Requests.

  • Included tests for client functionality and API interactions.

  • Added configuration files, documentation, and examples for integration setup.


Changes walkthrough 📝

Relevant files
Enhancement
4 files
client.py
Implement Bitbucket API client for resource fetching         
+131/-0 
integration.py
Define integration configuration and resource selectors   
+52/-0   
main.py
Implement resync logic for Bitbucket resources                     
+121/-0 
blueprints.json
Add blueprints for Bitbucket resource types                           
+120/-0 
Miscellaneous
1 files
debug.py
Add debug entry point for integration                                       
+4/-0     
Tests
2 files
conftest.py
Add test fixtures for mocking Bitbucket context                   
+30/-0   
test_client.py
Add tests for Bitbucket client functionality                         
+225/-0 
Configuration changes
5 files
.env.example
Provide example environment variables for integration       
+10/-0   
port-app-config.yml
Define mappings for Bitbucket resources in Port                   
+46/-0   
spec.yaml
Add integration specification for Bitbucket                           
+27/-0   
Makefile
Add Makefile for integration build and testing                     
+1/-0     
sonar-project.properties
Add SonarQube project configuration                                           
+2/-0     
Documentation
3 files
CHANGELOG.md
Add changelog for Bitbucket integration                                   
+15/-0   
CONTRIBUTING.md
Add contributing guidelines for Bitbucket integration       
+7/-0     
README.md
Add README for Bitbucket integration usage and development
+7/-0     
Dependencies
2 files
poetry.toml
Configure virtual environments for integration development
+3/-0     
pyproject.toml
Define project dependencies and settings                                 
+113/-0 
Additional files
1 files
__init__.py [link]   

Need help?
  • Type /help how to ... in the comments thread for any questions about Qodo Merge usage.
  • Check out the documentation for more information.
  • Copy link
    Contributor

    qodo-merge-pro bot commented Feb 28, 2025

    CI Feedback 🧐

    (Feedback updated until commit ec7261a)

    A test triggered by this PR failed. Here is an AI-generated analysis of the failure:

    Action: 🚢 bitbucket

    Failed stage: Lint [❌]

    Failure summary:

    The action failed during the linting process due to code formatting issues:

  • The black formatter identified 2 files that need reformatting:
    - /integrations/bitbucket/main.py

    - /integrations/bitbucket/client.py
  • The code needs to be formatted according to Black's style guidelines before it can pass the checks
  • Additionally, there's a warning about deprecated top-level linter settings in pyproject.toml that
    should be moved to the lint section

  • Relevant error logs:
    1:  ##[group]Operating System
    2:  Ubuntu
    ...
    
    1040:  - Installing pytest-httpx (0.34.0)
    1041:  - Installing pytest-xdist (3.6.1)
    1042:  - Installing ruff (0.6.9)
    1043:  - Installing towncrier (23.11.0)
    1044:  Installing the current project: bitbucket (0.1.0-beta)
    1045:  Warning: The current project could not be installed: No file/folder found for package bitbucket
    1046:  If you do not want to install the current project use --no-root.
    1047:  If you want to use Poetry only for dependency management but not for packaging, you can disable package mode by setting package-mode = false in your pyproject.toml file.
    1048:  In a future version of Poetry this warning will become an error!
    ...
    
    1065:  warning: The top-level linter settings are deprecated in favour of their counterparts in the `lint` section. Please update the following options in `pyproject.toml`:
    1066:  - 'ignore' -> 'lint.ignore'
    1067:  All checks passed!
    1068:  Running black
    1069:  would reformat /home/runner/work/ocean/ocean/integrations/bitbucket/main.py
    1070:  would reformat /home/runner/work/ocean/ocean/integrations/bitbucket/client.py
    1071:  Oh no! 💥 💔 💥
    1072:  2 files would be reformatted, 5 files would be left unchanged.
    1073:  �[0;31mOne or more checks failed with exit code 1�[0m
    1074:  make: *** [../_infra/Makefile:60: lint] Error 1
    1075:  ##[error]Process completed with exit code 2.
    

    Copy link

    This pull request is automatically being deployed by Amplify Hosting (learn more).

    Access this pull request here: https://pr-1448.d1ftd8v2gowp8w.amplifyapp.com

    @oiadebayo oiadebayo marked this pull request as ready for review March 3, 2025 10:38
    Copy link
    Contributor

    qodo-merge-pro bot commented Mar 3, 2025

    Qodo Merge was enabled for this repository. To continue using it, please link your Git account with your Qodo account here.

    PR Reviewer Guide 🔍

    Here are some key observations to aid the review process:

    ⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
    🧪 PR contains tests
    🔒 Security concerns

    Sensitive information exposure:
    The client implementation in client.py stores API credentials (username:app_password combination) in plaintext memory before base64 encoding. While the credentials are eventually encoded, they remain accessible in memory. Consider using secure credential storage or environment variables.

    ⚡ Recommended focus areas for review

    Security Concern

    The client stores sensitive credentials (username:app_password) in memory as plaintext before base64 encoding. Consider using secure credential storage.

    self.credentials = f"{username}:{app_password}"
    self.encoded_credentials = base64.b64encode(
        self.credentials.encode()
    ).decode()
    Performance Issue

    The folder sync implementation may not scale well with large repositories due to recursive directory traversal with max_depth=3. Consider implementing pagination or limiting depth.

    async def resync_pull_requests(kind: str) -> ASYNC_GENERATOR_RESYNC_TYPE:
        """Resync all pull requests from all repositories."""
        client = await init_client()
        async for repositories in client.get_repositories():
    Error Handling

    The error handling in _send_api_request could be improved to handle non-JSON responses and network errors more gracefully.

        response.raise_for_status()
        return response.json()
    except HTTPStatusError as e:
        error_data = e.response.json()
        error_message = error_data.get("error", {}).get("message", str(e))
        raise HTTPStatusError(error_message, request=e.request, response=e.response)

    Copy link
    Contributor

    qodo-merge-pro bot commented Mar 3, 2025

    Qodo Merge was enabled for this repository. To continue using it, please link your Git account with your Qodo account here.

    PR Code Suggestions ✨

    Explore these optional code suggestions:

    CategorySuggestion                                                                                                                                    Impact
    General
    Use defined constant instead of hardcoded value
    Suggestion Impact:The commit replaced the hardcoded timeout value 30.0 with the CLIENT_TIMEOUT constant

    code diff:

    -        self.client.timeout = Timeout(30.0)
    +        self.client.timeout = Timeout(CLIENT_TIMEOUT)

    The CLIENT_TIMEOUT constant is defined but never used. Instead, a hardcoded
    timeout value of 30.0 is used in the init method. Use the constant for
    consistency.

    integrations/bitbucket/client.py [9-25]

     CLIENT_TIMEOUT = 30
     
     
     class BitbucketClient:
         def __init__(
             self,
             workspace: str,
             username: Optional[str] = None,
             app_password: Optional[str] = None,
             workspace_token: Optional[str] = None,
         ) -> None:
             self.base_url = "https://api.bitbucket.org/2.0"
             self.workspace = workspace
             self.client = http_async_client
    -        self.client.timeout = Timeout(30.0)
    +        self.client.timeout = Timeout(CLIENT_TIMEOUT)

    [Suggestion has been applied]

    Suggestion importance[1-10]: 5

    __

    Why: The suggestion improves code maintainability by using the defined constant instead of a hardcoded value, making the code more consistent and easier to maintain. This is a valid improvement for code quality.

    Low
    Fix inconsistent documentation
    Suggestion Impact:The commit changed the max_depth parameter to be configurable with a default value of 2, removing the inconsistent documentation comment

    code diff:

    +        self, repo_slug: str, branch: str, path: str, max_depth: int = 2
         ) -> AsyncGenerator[list[dict[str, Any]], None]:
             """Get contents of a directory."""
             params = {
    -            "max_depth": 3,  # Get nested directories up to depth 20
    -            "pagelen": 100    # Maximum items per page
    +            "max_depth": max_depth,
    +            "pagelen": PAGE_SIZE,

    The max_depth parameter in get_directory_contents is set to 3 in the comment but
    20 in the actual code. This inconsistency should be fixed to match the intended
    behavior.

    integrations/bitbucket/client.py [114-117]

     params = {
    -    "max_depth": 3,  # Get nested directories up to depth 20
    +    "max_depth": 3,  # Get nested directories up to depth 3
         "pagelen": 100    # Maximum items per page
     }

    [Suggestion has been applied]

    Suggestion importance[1-10]: 4

    __

    Why: The suggestion correctly identifies a documentation inconsistency between the comment and the actual code value, which could cause confusion. While not a functional issue, fixing this improves code clarity and maintainability.

    Low
    • Update

    Copy link
    Member

    @mk-armah mk-armah left a comment

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    Left some comments

    Comment on lines +64 to +68
    except HTTPStatusError as e:
    error_data = e.response.json()
    error_message = error_data.get("error", {}).get("message", str(e))
    logger.error(f"Bitbucket API error: {error_message}")

    Copy link
    Member

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    You've suppressed error propagation entirely here, it won't be possible to catch errors anywhere else

    Comment on lines +83 to +89
    values = response.get("values", [])
    if values:
    yield values
    next_page = response.get("next")
    if not next_page:
    break
    endpoint = next_page.replace(self.base_url + "/", "")
    Copy link
    Member

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    Is values not consistent in all the api responses, if it is, lets be explicit..

    Copy link
    Member

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    Are there are situation where values is empty yet the next page has resources ?

    Copy link
    Member

    @mk-armah mk-armah left a comment

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    Left more comments

    break
    endpoint = next_page.replace(self.base_url + "/", "")

    # Project endpoints
    Copy link
    Member

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    remove

    ):
    yield projects

    # Repository endpoints
    Copy link
    Member

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    remove comment

    Comment on lines +19 to +23
    class ObjectKind(StrEnum):
    PROJECT = "project"
    FOLDER = "folder"
    REPOSITORY = "repository"
    PULL_REQUEST = "pull-request"
    Copy link
    Member

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    move to client or some utility

    Comment on lines +9 to +23
    class AsyncIteratorMock:
    """Helper class to mock async iterators."""

    def __init__(self, items: Iterator[Any]) -> None:
    self.items = items

    def __aiter__(self) -> "AsyncIteratorMock":
    return self

    async def __anext__(self) -> Any:
    try:
    return next(self.items)
    except StopIteration:
    raise StopAsyncIteration

    Copy link
    Member

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    whats the need for this ?

    Comment on lines +101 to +114
    @pytest.mark.asyncio
    async def test_send_api_request_error(mock_client: BitbucketClient) -> None:
    """Test API request with error response."""
    error_response = Response(400, json={"error": {"message": "Test error"}})
    mock_error = HTTPStatusError(
    "Test error", request=MagicMock(), response=error_response
    )
    with patch.object(
    mock_client.client, "request", new_callable=AsyncMock
    ) as mock_request:
    mock_request.side_effect = mock_error
    with pytest.raises(HTTPStatusError) as exc_info:
    await mock_client._send_api_request("test/endpoint")
    assert "Test error" in str(exc_info.value)
    Copy link
    Member

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    I'm not sure I understand the goal of this test

    The test patches the request method on mock_client.client to immediately raise an HTTPStatusError (via side_effect). This means that the error is raised at the line:

    response = await self.client.request(...)

    So, the function never even gets a chance to enter the try block where it would normally call response.raise_for_status().

    As a result, the error handling code inside the except block (e.g., extracting and logging the error message) isn’t exercised.

    Please Reiterate

    Copy link
    Member

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    Its concerning because your _send_api_request doesn't raise an error at all from its design.

    Comment on lines +118 to +134
    async def test_send_paginated_api_request(mock_client: BitbucketClient) -> None:
    """Test paginated API request."""
    page1 = {
    "values": [{"id": 1}, {"id": 2}],
    "next": "https://api.bitbucket.org/2.0/test/endpoint?page=2",
    }
    page2 = {"values": [{"id": 3}], "next": None}
    with patch.object(
    mock_client, "_send_api_request", new_callable=AsyncMock
    ) as mock_request:
    mock_request.side_effect = [page1, page2]
    results = []
    async for batch in mock_client._send_paginated_api_request("test/endpoint"):
    results.extend(batch)
    assert len(results) == 3
    assert [item["id"] for item in results] == [1, 2, 3]
    assert mock_request.call_count == 2
    Copy link
    Member

    @mk-armah mk-armah Mar 5, 2025

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    also assert that its capable of yielding one page at a time

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Projects
    None yet
    Development

    Successfully merging this pull request may close these issues.

    2 participants