Add permissions fetcher

ComplianceAsCode · Mar 1, 2021 · 02ded94 · 02ded94
1 parent 41874ad
commit 02ded94
Show file tree

Hide file tree

Showing 5 changed files with 65 additions and 145 deletions.
diff --git a/CHANGES.md b/CHANGES.md
@@ -1,3 +1,7 @@
+# [0.11.0](https://github.com/ComplianceAsCode/auditree-arboretum/releases/tag/v0.11.0)
+
+- [ADDED] Organization repository permissions fetcher added to `permissions`.
+
 # [0.10.0](https://github.com/ComplianceAsCode/auditree-arboretum/releases/tag/v0.10.0)
 
 - [ADDED] Organization repository direct collaborators check added to `permissions`.

diff --git a/arboretum/__init__.py b/arboretum/__init__.py
@@ -14,4 +14,4 @@
 # limitations under the License.
 """Arboretum - Checking your compliance & security posture, continuously."""
 
-__version__ = '0.10.0'
+__version__ = '0.11.0'
diff --git a/arboretum/permissions/README.md b/arboretum/permissions/README.md
@@ -16,14 +16,11 @@ how to include the fetchers and checks from this library in your downstream proj
 
 ## Fetchers
 
-### Organization Integrity (Repository Collaborators)
+### Organization Integrity Permissions
 
-* Class: [GithubOrgCollaboratorsFetcher][gh-org-fetcher]
-* Purpose: Writes the details of collaborators in Github organizations to the evidence locker. This fetcher class is only meant for use with Github or Github Enterprise organizations.
-* Behavior: For each Github organization specified, an evidence file per collaborator type (affiliation) is stored in
-the locker containing collaborator details for the specified repositories in the organization. The default is to
-retrieve all collaborators by affiliation from all repositories in each specified Github organization.  TTL is set to 1
-day.
+* Class: [GithubOrgPermissionFetcher][fetch-org-permissions]
+* Purpose: Writes the details of collaborators and repository forks in Github organizations to the evidence locker. This fetcher class is only meant for use with Github or Github Enterprise organizations.
+* Behavior: For each Github organization specified, Github collaborator and Github fork evidence files per collaborator type (affiliation) are stored in the locker containing details for the specified repositories in the organization. The default is to retrieve all collaborators and all forks by affiliation from all repositories in each specified Github organization.  TTL is set to 1 day.
 * Configuration elements:
   * `org.permissions.org_integrity.orgs`
      * Required
@@ -36,7 +33,7 @@ day.
            * Optional
            * List of strings in the form of `["my-repo", "my-other-repo"]`.
            * Defaults to all repositories in the organization.
-           * Use if looking to filter collaborator evidence to a subset of repositories in the organization otherwise do not include.
+           * Use if looking to filter permissions evidence to a subset of repositories in the organization otherwise do not include.
         * `collaborator_types`
            * Optional
            * List of strings in the form of `["all", "direct", "outside"]`.
@@ -92,79 +89,7 @@ day.
 * Import statement:
 
   ```python
-  from arboretum.permissions.fetchers.github.fetch_org_collaborators import GithubOrgCollaboratorsFetcher
-  ```
-
-### Organization Integrity (Repository Forks)
-
-* Class: [GithubRepoForksFetcher][gh-org-fetcher]
-* Purpose: Writes the repository forks in Github organizations to the evidence locker. This fetcher class is only meant for use with Github or Github Enterprise organizations.
-* Behavior: For each Github organization specified, an evidence file  is stored in
-the locker containing the forks for the specified repositories in the organization. The default is to
-retrieve all forks from all repositories in each specified Github organization.  TTL is set to 1
-day.
-* Configuration elements:
-  * `org.permissions.org_integrity.orgs`
-     * Required
-     * List of dictionaries each containing Github organization retrieval configuration.
-        * `url`
-           * Required
-           * String in the form of `"https://github.com/my-org"` or `"https://github.<company>.com/my-org"`.
-           * Use to define the Github organization url to use.
-        * `repos`
-           * Optional
-           * List of strings in the form of `["my-repo", "my-other-repo"]`.
-           * Defaults to all repositories in the organization.
-           * Use if looking to filter collaborator evidence to a subset of repositories in the organization otherwise do not include.
-* Example configuration:
-
-  ```json
-  {
-    "org": {
-      "permissions": {
-        "org_integrity": {
-          "orgs": [
-            {
-              "url": "https://github.com/my-org-1"
-            },
-            {
-              "url": "https://github.my-company.com/my-org-2",
-              "repos": ["repo1", "repo2"]
-            }
-          ]
-        }
-      }
-    }
-  }
-  ```
-
-* Required credentials:
-  * `github` or `github_enterprise` credentials with [appropriate permissions to the repositories][repository-permissions] are required for this fetcher to successfully retrieve evidence.
-     * `username`: The Github user used to run the fetcher.
-     * `token`: The Github user access token used to run the fetcher.
-  * Example credentials file entry:
-
-     ```ini
-     [github]
-     username=gh-user-name
-     token=gh-access-token
-     ```
-
-     or
-
-     ```ini
-     [github_enterprise]
-     username=ghe-user-name
-     token=ghe-access-token
-     ```
-
-  * NOTE: These credentials are also needed for basic configuration of the
-  Auditree framework. The expectation is that the same credentials are used for all Github interactions.
-
-* Import statement:
-
-  ```python
-  from arboretum.permissions.fetchers.github.fetch_repo_forks import GithubRepoForksFetcher
+  from arboretum.permissions.fetchers.github.fetch_org_permissions import GithubOrgPermissionFetcher
   ```
 
 ## Checks
@@ -236,6 +161,6 @@ direct collaborators matching the exceptions are found.
 [auditree-framework]: https://github.com/ComplianceAsCode/auditree-framework
 [auditree-framework documentation]: https://complianceascode.github.io/auditree-framework/
 [usage]: https://github.com/ComplianceAsCode/auditree-arboretum#usage
-[gh-org-fetcher]: https://github.com/ComplianceAsCode/auditree-arboretum/blob/main/arboretum/permissions/fetchers/github/fetch_org_collaborators.py
+[fetch-org-permissions]: https://github.com/ComplianceAsCode/auditree-arboretum/blob/main/arboretum/permissions/fetchers/github/fetch_org_permissions.py
 [repository-permissions]: https://docs.github.com/en/free-pro-team@latest/github/setting-up-and-managing-organizations-and-teams/repository-permission-levels-for-an-organization
-[org-collaborators-check]: https://github.com/ComplianceAsCode/auditree-arboretum/blob/main/arboretum/permissions/checks/test_org_collaborators.py
+[org-collaborators-check]: https://github.com/ComplianceAsCode/auditree-arboretum/blob/main/arboretum/permissions/checks/test_org_collaborators.py
diff --git a/arboretum/permissions/fetchers/github/fetch_org_permissions.py b/arboretum/permissions/fetchers/github/fetch_org_permissions.py
@@ -0,0 +1,52 @@
+# -*- mode:python; coding:utf-8 -*-
+# Copyright (c) 2021 IBM Corp. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Github organization permissions fetcher."""
+import json
+
+import arboretum.permissions.fetchers.github.fetch_org_collaborators as collabs
+
+
+class GithubOrgPermissionsFetcher(collabs.GithubOrgCollaboratorsFetcher):
+    """Fetch Github permissions evidence."""
+
+    def fetch_repo_forks(self):
+        """Fetch Github repository forks."""
+        for config in self.config.get('org.permissions.org_integrity.orgs'):
+            host, org = config['url'].rsplit('/', 1)
+            url_hash = collabs.get_sha256_hash([config['url']], 10)
+            path = ['permissions', f'gh_forks_{url_hash}.json']
+            description = (f'Fork of the {org} GH org')
+            self.config.add_evidences(
+                [
+                    collabs.RawEvidence(
+                        path[1], path[0], collabs.DAY, description
+                    )
+                ]
+            )
+            with collabs.raw_evidence(self.locker, '/'.join(path)) as evidence:
+                if evidence:
+                    if host not in self.gh_pool:
+                        self.gh_pool[host] = collabs.Github(base_url=host)
+                    if not config.get('repos'):
+                        repos = self.gh_pool[host].paginate_api(
+                            f'orgs/{org}/repos'
+                        )
+                        config['repos'] = [repo['name'] for repo in repos]
+                    forks = {}
+                    for repo in config['repos']:
+                        forks[repo] = self.gh_pool[host].paginate_api(
+                            f'repos/{org}/{repo}/forks'
+                        )
+                    evidence.set_content(json.dumps(forks))
diff --git a/arboretum/permissions/fetchers/github/fetch_repo_forks.py b/arboretum/permissions/fetchers/github/fetch_repo_forks.py