v0.4.57 (#759)

* update release refs to rc-0.4.57 * avoid codeql log injection complaint (#746) * comment out log injection; explain it * Update java/impl/aws/src/main/java/co/worklytics/psoxy/aws/VaultAwsIamAuth.java Co-authored-by: Jose Lorenzo <jose@worklytics.co> --------- Co-authored-by: Jose Lorenzo <jose@worklytics.co> * document scopes per-connector for MSFT sources (#745) * document scopes, in one-pager per connector for MSFT * msft-teams * Update docs/sources/microsoft-365/msft-teams/README.md Co-authored-by: aperez-worklytics <75276364+aperez-worklytics@users.noreply.github.com> * Update docs/sources/microsoft-365/msft-teams/README.md Co-authored-by: aperez-worklytics <75276364+aperez-worklytics@users.noreply.github.com> * Update docs/sources/microsoft-365/msft-teams/README.md Co-authored-by: aperez-worklytics <75276364+aperez-worklytics@users.noreply.github.com> * Update docs/sources/microsoft-365/entra-id/README.md Co-authored-by: Jose Lorenzo <jose@worklytics.co> --------- Co-authored-by: aperez-worklytics <75276364+aperez-worklytics@users.noreply.github.com> Co-authored-by: Jose Lorenzo <jose@worklytics.co> * docs fixes (#747) * Fix AWS/GCP doc indexes * Format table using HTML Try to leverage GitBook's full width table feature * Create docs branch and display manual instructions on release * update gcp deps (#751) * update BOM * ARTIFACT_REGISTRY in bulk case (already done for API case) * update functions fw API * functions fw as provided * comment on functions FW via bom * update functions plugin * some cleanup * use top-level pom variable for gcp BOM version * fix bom unification * fix gcp build with latest maven (#750) * script to help check for the issue * pom that pins good version of maven-assembly-plugin (3.6.0) * more correct uber-jar assembly pom * revert assembly plugin version to latest, although leave pinned * add Implementation Guide to docs (#753) * implementation guide * Update docs/guides/implementation.md Co-authored-by: Jose Lorenzo <jose@worklytics.co> --------- Co-authored-by: Jose Lorenzo <jose@worklytics.co> * Support mailboxsettings for directory (#755) * Updated permission * Updated documentation * Updated changelog * fix MSFT owners data resource (#754) * Teams cleanup (#756) * Format * Drop not required permission * Using v1.0 in path * Move salesforce tests under rules directory * Drop from docs * Changelog update * Missing perm * java17 runtimes by default, in both AWS and GCP (#757) * GCP container scanning tooling (#758) * GCP container scanning tooling * Update docs/development/releases.md * prep release v0.4.57 (#752) * update release refs to v0.4.57 * improve CHANGELOG * port correct building to AWS case; improve check tool * tooling fixes (#760) * improve rc-to-main next steps, for clarity * fix prompt * expose param to set permissions boundary on roles (#761) --------- Co-authored-by: Jose Lorenzo <jose@worklytics.co> Co-authored-by: aperez-worklytics <75276364+aperez-worklytics@users.noreply.github.com> Co-authored-by: David <1151427+davidfq@users.noreply.github.com>
Worklytics · Jul 11, 2024 · 1e13417 · 1e13417
1 parent b7102fb
commit 1e13417
Show file tree

Hide file tree

Showing 65 changed files with 691 additions and 251 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -7,6 +7,18 @@ Changes to be including in future/planned release notes will be added here.
 
 ## Next
 
+## [0.4.57](https://github.com/Worklytics/psoxy/release/tag/v0.4.57)
+Several changes in this version will result in visible changes during `terraform plan`/`apply`:
+- Permission changes on Microsoft 365:
+  - `MailboxSettings.Read` permission has been added for directory connectors (`azure-ad`, `entra-id`)
+    This will have NO impact until your Admin grants this permission in the Microsoft 365 Admin
+    Center; until that happens, Worklytics will continue to NOT retrieve mailbox settings.
+  - `OnlineMeetings.Read.All` and `OnlineMeetingArtifact.Read.All` permissions have been removed from `outlook-cal` connector
+- `java17` runtime by default (previously, was `java11`); `java11` is still supported by AWS, but
+   will be deprecated by GCP in Sept 2024. As of 0.4, proxy code is still compiled for java 11 - so
+   if you wish to keep using `java11` runtime, it will work; if you require this, let us know asnd
+   we'll expose option to select runtime version in the Terraform module.
+
 ## [0.4.56](https://github.com/Worklytics/psoxy/release/tag/v0.4.56)
  - due to refactoring, users of Microsoft connectors may see some moves of resources in Terraform
    plan; these will be no-ops.

diff --git a/docs/README.md b/docs/README.md
diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md
@@ -22,14 +22,15 @@
   * [Getting Started](gcp/getting-started.md)
   * [Authentication and Authorization](gcp/authentication-authorization.md)
   * [Getting Started with Google Cloud Shell](gcp/cloud-shell.md)
-  * [GCP Development](gcp/development.md)
   * [GCP Troubleshooting](gcp/troubleshooting.md)
+  * [GCP Development](gcp/development.md)
 * [General Guides](guides/README.md)
-  * [Cleaning Up](guides/cleaning-up.md)
-  * [Deployment Migration](guides/deployment-migration.md)
+  * [Implementation](guides/implementation.md)
   * [Terraform Cloud / Enterprise](guides/terraform-cloud.md)
   * [Testing](guides/testing.md)
+  * [Deployment Migration](guides/deployment-migration.md)
   * [Upgrade Proxy Versions](guides/upgrading-versions.md)
+  * [Cleaning Up](guides/cleaning-up.md)
 * [General Troubleshooting](troubleshooting.md)
 * [Configuration](configuration/README.md)
   * [API Data Sanitization](configuration/api-data-sanitization.md)

diff --git a/docs/aws/README.md b/docs/aws/README.md
@@ -0,0 +1 @@
+# AWS
diff --git a/docs/development/releases.md b/docs/development/releases.md
@@ -22,8 +22,16 @@ On `rc-`:
 
 QA aws, gcp dev examples by running `terraform apply` for each, and testing various connectors.
 
+Scan a GCP container image for vulnerabilities:
+
+```shell
+./tools/gcp/container-scan.sh psoxy-dev-erik psoxy-dev-erik-gcal
+```
+
+Create PR to merge `rc-` to `main`.
+
 ```shell
-./tools/release/rc-to-release.sh v0.4.16
+./tools/release/rc-to-main.sh v0.4.16
 ```
 
 After merged to `main`:

diff --git a/docs/gcp/README.md b/docs/gcp/README.md
diff --git a/docs/gcp/README.md b/docs/gcp/README.md
@@ -0,0 +1 @@
+# GCP
diff --git a/docs/guides/implementation.md b/docs/guides/implementation.md
@@ -0,0 +1,65 @@
+# Implementation Guide
+
+This guide provides a roadmap of a typical implementation with Worklytics-provided support.
+
+## 1 Kick-off/Scoping meeting
+
+*30-60 min video call to get overview of process, responsibilities*
+
+Attendees:
+  - Product Stakeholder(s)
+  - Data Source Administrator(s), if identified
+  - IT Admin(s), if identified
+
+Agenda:
+ - determine data sources, and who can authorize access to each
+ - determine host platform (GCP or AWS)
+ - identify who has the permissions to manage infra, will be able to run Terraform, and how
+   they'll run it (where, authenticated how)
+ - scope desired data interval, approximate headcount, etc.
+ - identify any potential integration issues or infrastructure constraints
+
+## 2 Initial Walk through
+
+*1-2 hr video call, to walk-through customization and initial terraform runs via screenshare*
+
+Attendees:
+  - IT Admin(s) who will be running Terraform
+  - Worklytics technical contact
+
+Prior to this call, please follow the initial steps in the `Getting Started` section for your host
+platform and ensure you have all Prereqs
+
+Goals:
+  1. get example customized and a terraform plan working.
+  2. run `terraform apply`. Obtain the `TODO 1` files you can send to your data source
+     administrators to complete, as needed.
+
+Tips:
+  - Works best if we screenshare
+
+## 3 Testing / Validation
+
+*can be completed without call; but Worklytics can assist if desired*
+
+  - follow `TODO 2` files / use test.sh shell scripts produced by `terraform apply`
+  - validate that authentication/authorization is correct for all connections, and that you're
+    satisfied with proxy behavior
+
+## 4 Authorize Worklytics to Access Sanitized Data
+
+*can be completed without call; but Worklytics can assist if desired*
+
+Authorize Worklytics to invoke API connectors / access sanitized bulk data:
+  - obtain service account ID of your tenant from Worklytics; configure it in you terraform.tfvars file
+  - run `terraform apply` again to update IAM policy to reflect the change
+
+## 5 Connect Sanitized Data Sources to Worklytics
+
+*can be completed without call; but Worklytics can assist if desired*
+
+  - follow `TODO 3` files (or terraform output values) generated by the `terraform apply` command
+  - if you do not have access to [Worklytics](https://app.worklytics.co),  or you do, but do not have `Data Connection Admin` role, send
+    these files to the appropriate person
+
+
diff --git a/docs/sources/microsoft-365/README.md b/docs/sources/microsoft-365/README.md
@@ -68,11 +68,11 @@ proxy host platform)
 
 The following Scopes are required for each connector. Note that they are all READ-only scopes.
 
-| Source&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | Examples &nbsp;&nbsp;                                                                                                        | Application Scopes                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
-|--------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| Entra ID                                                                                               | [data](https://github.com/Worklytics/psoxy/tree/main/docs/sources/microsoft-365/directory/example-api-responses) - [rules](https://github.com/Worklytics/psoxy/tree/main/docs/sources/microsoft-365/directory/directory.yaml)          | [`User.Read.All`](https://learn.microsoft.com/en-us/graph/permissions-reference#userreadall) [`Group.Read.All`](https://learn.microsoft.com/en-us/graph/permissions-reference#groupreadall)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
-| Calendar                                                                                               | [data](https://github.com/Worklytics/psoxy/tree/main/docs/sources/microsoft-365/outlook-cal/example-api-responses) - [rules](https://github.com/Worklytics/psoxy/tree/main/docs/sources/microsoft-365/outlook-cal/outlook-cal.yaml)    | [`User.Read.All`](https://learn.microsoft.com/en-us/graph/permissions-reference#userreadall) [`Group.Read.All`](https://learn.microsoft.com/en-us/graph/permissions-reference#groupreadall) [`OnlineMeetings.Read.All`](https://learn.microsoft.com/en-us/graph/permissions-reference#onlinemeetingsreadall) [`Calendars.Read`](https://learn.microsoft.com/en-us/graph/permissions-reference#calendarsread) [`MailboxSettings.Read`](https://learn.microsoft.com/en-us/graph/permissions-reference#mailboxsettingsread)                                                                                                                                                                                                                                     |
-| Mail                                                                                                   | [data](https://github.com/Worklytics/psoxy/tree/main/docs/sources/microsoft-365/outlook-mail/example-api-responses) - [rules](https://github.com/Worklytics/psoxy/tree/main/docs/sources/microsoft-365/outlook-mail/outlook-mail.yaml) | [`User.Read.All`](https://learn.microsoft.com/en-us/graph/permissions-reference#userreadall) [`Group.Read.All`](https://learn.microsoft.com/en-us/graph/permissions-reference#groupreadall) [`Mail.ReadBasic.All`](https://learn.microsoft.com/en-us/graph/permissions-reference#mailreadbasicall) [`MailboxSettings.Read`](https://learn.microsoft.com/en-us/graph/permissions-reference#mailboxsettingsread)                                                                                                                                                                                                                                                                                                                                               |
+| Source&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; | Examples &nbsp;&nbsp;                                                                                                        | Application Scopes                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
+|--------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Entra ID                                                                                               | [data](https://github.com/Worklytics/psoxy/tree/main/docs/sources/microsoft-365/directory/example-api-responses) - [rules](https://github.com/Worklytics/psoxy/tree/main/docs/sources/microsoft-365/directory/directory.yaml)          | [`User.Read.All`](https://learn.microsoft.com/en-us/graph/permissions-reference#userreadall) [`Group.Read.All`](https://learn.microsoft.com/en-us/graph/permissions-reference#groupreadall) [`MailboxSettings.Read`](https://learn.microsoft.com/en-us/graph/permissions-reference#mailboxsettingsread)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
+| Calendar                                                                                               | [data](https://github.com/Worklytics/psoxy/tree/main/docs/sources/microsoft-365/outlook-cal/example-api-responses) - [rules](https://github.com/Worklytics/psoxy/tree/main/docs/sources/microsoft-365/outlook-cal/outlook-cal.yaml)    | [`User.Read.All`](https://learn.microsoft.com/en-us/graph/permissions-reference#userreadall) [`Group.Read.All`](https://learn.microsoft.com/en-us/graph/permissions-reference#groupreadall) [`Calendars.Read`](https://learn.microsoft.com/en-us/graph/permissions-reference#calendarsread) [`MailboxSettings.Read`](https://learn.microsoft.com/en-us/graph/permissions-reference#mailboxsettingsread)                                                                                                                                                                                                                                     |
+| Mail                                                                                                   | [data](https://github.com/Worklytics/psoxy/tree/main/docs/sources/microsoft-365/outlook-mail/example-api-responses) - [rules](https://github.com/Worklytics/psoxy/tree/main/docs/sources/microsoft-365/outlook-mail/outlook-mail.yaml) | [`User.Read.All`](https://learn.microsoft.com/en-us/graph/permissions-reference#userreadall) [`Group.Read.All`](https://learn.microsoft.com/en-us/graph/permissions-reference#groupreadall) [`Mail.ReadBasic.All`](https://learn.microsoft.com/en-us/graph/permissions-reference#mailreadbasicall) [`MailboxSettings.Read`](https://learn.microsoft.com/en-us/graph/permissions-reference#mailboxsettingsread)                                                                                                                                                                                                                                                                                                                                              |
 | Teams (**__beta__**)                                                                                   | [data](https://github.com/Worklytics/psoxy/tree/main/docs/sources/microsoft-365/msft-teams/example-api-responses) - [rules](https://github.com/Worklytics/psoxy/tree/main/docs/sources/microsoft-365/msft-teams/msft-teams.yaml)| [`User.Read.All`](https://learn.microsoft.com/en-us/graph/permissions-reference#userreadall) [`Team.ReadBasic.All`](https://learn.microsoft.com/en-us/graph/permissions-reference#teamreadbasicall) [`Channel.ReadBasic.All`](https://learn.microsoft.com/en-us/graph/permissions-reference#channelreadbasicall) [`Chat.Read.All`](https://learn.microsoft.com/en-us/graph/permissions-reference#chatreadall) [`ChannelMessage.Read.All`](https://learn.microsoft.com/en-us/graph/permissions-reference#channelmessagereadall) [`CallRecords.Read.All`](https://learn.microsoft.com/en-us/graph/permissions-reference#channelmessagereadall) [`OnlineMeetings.Read.All`](https://learn.microsoft.com/en-us/graph/permissions-reference#onlinemeetingsreadall) |
 
 NOTE: the above scopes are copied from

diff --git a/docs/sources/microsoft-365/entra-id/README.md b/docs/sources/microsoft-365/entra-id/README.md
@@ -1,15 +1,39 @@
 # Entra ID
 
-## Examples
+Connect to Directory data in Microsoft 365. This allows enumeration of all users, groups, and group
+members in your organization, to provide additional segmentation, timezone/workday information, etc.
 
-- [Example Rules](entra-id.yaml)
-- [Example Rules: no App IDs](entra-id_no-app-ids.yaml)
-- [Example Rules: no App IDs, no orig](entra-id_no-app-ids_no-orig.yaml)
-- Example Data:
-  - [original/group-members.json](example-api-responses/original/group-members.json) |
-    [sanitized/group-members.json](example-api-responses/sanitized/group-members.json)
-  - [original/users.json](example-api-responses/original/users.json) |
-    [sanitized/users.json](example-api-responses/sanitized/users.json)
+## Required Scopes
+- [`User.Read.All`](https://learn.microsoft.com/en-us/graph/permissions-reference#userreadall)
+- [`Group.Read.All`](https://learn.microsoft.com/en-us/graph/permissions-reference#userreadall)
+
+## Authentication
+
+See the [Microsoft 365 Authentication](../README.md#authentication) section of the main README.
+
+## Authorization
+
+See the [Microsoft 365 Authorization](../README.md#authorization) section of the main README.
+
+## Example Data
+
+| API Endpoint | Example Response                                                             | Sanitized Example Response |
+| --- |------------------------------------------------------------------------------| --- |
+| `/v1.0/groups/{group-id}/members` | [original/group-members.json](example-api-responses/original/group-members.json) | [sanitized/group-members.json](example-api-responses/sanitized/group-members.json) |
+| `/v1.0/users` | [original/users.json](example-api-responses/original/users.json)             | [sanitized/users.json](example-api-responses/sanitized/users.json) |
+| `/v1.0/users/me` | [original/user.json](example-api-responses/original/user.json)              | [sanitized/user.json](example-api-responses/sanitized/user.json) |
+| `/v1.0/groups` | [original/groups.json](example-api-responses/original/groups.json)          | [sanitized/groups.json](example-api-responses/sanitized/groups.json) |
+
+
+Assuming proxy is auth'd as an application, you'll have to replace `me` with your MSFT ID or
+`UserPrincipalName` (often your email address).
 
 See more examples in the `docs/sources/microsoft-365/entra-id/example-api-responses` folder
 of the [Psoxy repository](https://github.com/Worklytics/psoxy).
+
+## Sanitization Rule Examples
+
+- [Default Rules](entra-id.yaml)
+- [Rules, pseudonymizing MSFT account IDs](entra-id_no-app-ids.yaml)
+- [Rules, pseudonymizing MSFT account IDs](entra-id_no-app-ids_no-orig.yaml)
+