Audit Log Refactor #345

juggernot325 · 2024-01-24T16:11:32Z

Description

This PR refactors the generation of audit log records to use a combination of new middleware logic and context tracking to store the data needed for the log. The main goal of this change was to easily be able to track failure scenarios on endpoints that generate audit logs as we previously only tracked successful auditable actions. It also let's us capture the error messages to provide information as to why the failure occurred.

With this refactor, audit log creation will now mostly occur at the database layer. A new wrapper function called AuditableTransaction was introduced that can be used to ensure audit log entries are created around sensitive changes to the DB. This changes audit log functionality a bit.

When an AuditableTransaction first kicks off an initial audit log entry will be created with a status of intent. This indicates that a user has attempted to make a sensitive change to the data that should be audited. The audit log entry will be given a commit_id UUID that will help associate it with additional audit logs that may be created for the same transaction.
The transaction occurs as normal.
A second audit log entry is created showing the result of the transaction. The status of this entry will be either success or failure depending on if the transaction was successful. If an error occurs, that will be captured in the fields section of the audit log entry.

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Chore (a change that does not modify the application functionality)
Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

Documentation updates are needed, and have been made accordingly.
I have added and/or updated tests to cover my changes.
All new and existing tests passed.
My changes include a database migration.

…of a generic response

… on the presence of specific context This commit mostly contains the logic to make this method for audit logging work. The only audit log event that works in this commit is CreateUser

…ure cases.

* updated IP address parsing * added tests for IP parsing, changed IP delimiter to comma

…elds

* chore: clean up old approach to reduce noise * chore: cleanup audit logging interface for new usages * chore: fix existing tests (audit integration test needs new target) * chore: improve integration config defaults * feat: add audit logging to UpdateUser This adds a new `AuditableContext` method to the Database, which wraps the Gorm Transaction method with some auditing context and ensures the two phase commit of the audit log is run * feat: add audit logging to DeleteSAMLProvider fix: use actual AuditData instead of the direct model * feat: CreateAssetGroup and DeleteAssetGroup are now back to auditable fix: broken tests * feat: plumb commitID for real this time fix: incorrect formatting of audit log action names fix: additional test fixes * fix: additional test fixes * review comments --------- Co-authored-by: Irshad Ahmed <iahmed@specterops.io>

* BED-3858 - Fix not ingesting tenant count during azure analysis (#328) Co-authored-by: Irshad Ahmed <iahmed@specterops.io> * ESC9a Edge Composition (#354) * feat: esc9a post * test: add esc9 test * chore: add harness files * fix: regen schema after merge * chore: fix small nits * chore: cleanup cert template new function * chore: add missing props * wip: 9a composition * fix: treat failure to grab properties as true * wip: esc9a composition * wip: esc9a composition * feat+chore: add depth controls to dawgs patterns * wip: esc9a composition * fix: do not drop the current segment if the next pattern is optional * wip: esc9a composition * fix: update other continuations to respect depth correctly * wip: edge comp * fix: swap * chore: remove unnecessary logs * feat: esc9a post * test: add esc9 test * chore: fix small nits * wip: 9a composition * wip: esc9a composition * wip: esc9a composition * feat+chore: add depth controls to dawgs patterns * wip: esc9a composition * fix: do not drop the current segment if the next pattern is optional * wip: esc9a composition * fix: update other continuations to respect depth correctly * wip: edge comp * fix: swap * chore: remove unnecessary logs * test: add test covering esc9a edge comp * chore: revert random re-ordering * chore: handle negative min/max depth on continuations --------- Co-authored-by: John Hopper <jhopper@specterops.io> * docs: Add to ESC3 abuse info (#350) * docs: add note in ESC6 abuse info (#356) * feat: Add ADCS pre-built queries (#342) Co-authored-by: Rohan Vazarkar <rvazarkar@users.noreply.github.com> * feat: esc6a edge composition (#359) * feat: esc6a edge composition * chore: allow composition accordion to show for 6a * fix: add trustedby rel to path4 pattern, use outboundwithdepth for optional memberof traversal * chore: update dcfor pattern to use outboundwithdepth for optional group membership * ESC10a Post Processing (#360) * wip: initial ESC10a post * test: all the tests for esc10a * chore: add edges to post processed * chore: add harnessgen script * test: remove edges from harness * chore: don't exit loop if we hit an error, continue instead * chore: log and continue * feat: filter out ESC3 false positives (#351) * feat: filter out ESC3 false positives * fix: handle esc3 filtering without retraversal * fix: handle esc3 filtering without retraversal * fix: handle esc3 filtering without retraversal * chore: rename function for re-use * chore: log and continue --------- Co-authored-by: rvazarkar <rvazarkar@specterops.io> Co-authored-by: Rohan Vazarkar <rvazarkar@users.noreply.github.com> * chore: patch EULAAcceptance bypass to only run if the current user is set to false * fix: incorrect usage of RemoteAddr fix: unnecessary AuditData() calls fix: use pointers for AuditEntry.Model assignments so successful actions can record updated fields like ID --------- Co-authored-by: mistahj67 <26472282+mistahj67@users.noreply.github.com> Co-authored-by: Irshad Ahmed <iahmed@specterops.io> Co-authored-by: Rohan Vazarkar <rvazarkar@users.noreply.github.com> Co-authored-by: John Hopper <jhopper@specterops.io> Co-authored-by: Jonas Bülow Knudsen <12843299+JonasBK@users.noreply.github.com> Co-authored-by: Ulises Rangel <urangel@specterops.io> Co-authored-by: rvazarkar <rvazarkar@specterops.io>

* audit logs for auth tokens and secrets * fix bad copy

…elds

…ector, CreateSAMLIdentityProvider, and UpdateSAMLIdentityProvider have audit log support (#374) chore: remove unused RemoveAssetGroupSelector method chore: update tests and mocks to account for interface changes

* more auth handlers for audit log * minor optimization

…status

* feat: create an audit log record when unauthorized access is attempted * chore: Only log unauthorized write access * chore: add audit logging to remaining auth middleware * test: Fix middleware/auth_test to support new audit log changes

It was previously sized to only account for max 1 IPv6 address, but our design changes mean it could now be any length

cmd/api/src/api/middleware/middleware.go

This reverts commit 2a2c839.

…ostgres

* Refactored unauthorized access audit logging: - moved audit logging responsibility to Authorizer interface - cleaned up injections of *db through several layers of code (no longer necessary) - minor cleanup and support work

zinic

Some changes requested. Rest of the changeset looks good.

cmd/api/src/database/migration/migrations/v5.6.0.sql

cmd/api/src/api/middleware/middleware.go

cmd/api/src/api/router/router.go

cmd/api/src/api/middleware/middleware.go

zinic

Changes look good from my end.

juggernot325 added 14 commits January 25, 2024 16:40

feat: Initial migration and model updates for AuditLog improvements

0e1d356

feat: populate actor_email for existing audit_logs

d0c6173

chore: Fix audit log docs to show the actual response object instead …

1d08960

…of a generic response

feat: Log Requester IP for source and actor email with audit logs

6019c99

wip: createUser audit logging via middleware

0f0d55a

chore: refactor audit logging to trigger from LoggingMiddleware based…

3d65a85

… on the presence of specific context This commit mostly contains the logic to make this method for audit logging work. The only audit log event that works in this commit is CreateUser

wip: Successfully creating audit logs from middleware. Capturing fail…

86b77a3

…ure cases.

feat: Capture DB errors in audit logs

22c3e8c

chore: remove debug logging

cf4b854

chore: Refactor setting of status to receiver function

985d580

wip: All "simple" audit context endpoints updated for middeware

29673a9

fix: Fix a few bugs found during testing

a05ba51

fix: Remove doc for non-existent POST /api/v2/saml endpoint

cd78c04

wip: Refactor to simplify setting of AuditCtx in endpoints

9cb8a3d

irshadaj force-pushed the populate-audit-log-fields branch from 9b8a441 to 9cb8a3d Compare January 25, 2024 23:18

superlinkx and others added 15 commits January 25, 2024 16:28

fix: registration issue after rebase

e1d81bd

tasks 6 and 7 for #345 (#357)

3ac13c1

* updated IP address parsing * added tests for IP parsing, changed IP delimiter to comma

Merge remote-tracking branch 'origin/main' into populate-audit-log-fi…

fd32507

…elds

redo IP parsing for changed requirements (#362)

2a14ff5

add eula_accepted to user audit data (#371)

6810bd3

audit logs for auth tokens and secrets (#372)

0482b61

* audit logs for auth tokens and secrets * fix bad copy

Merge branch 'main' into populate-audit-log-fields

af01314

Merge remote-tracking branch 'origin/main' into populate-audit-log-fi…

9ff9831

…elds

more auth handlers for audit log (#373)

d744c31

* more auth handlers for audit log * minor optimization

chore: update documentation for our new commit_id field and intent …

061dc91

…status

chore: change source column to TEXT

13f4160

It was previously sized to only account for max 1 IPv6 address, but our design changes mean it could now be any length

mistahj67 reviewed Feb 1, 2024

View reviewed changes

cmd/api/src/api/middleware/middleware.go Outdated Show resolved Hide resolved

irshadaj and others added 7 commits February 1, 2024 11:19

rename audit log source column, add indices

2a2c839

Revert "rename audit log source column, add indices"

cc16a11

This reverts commit 2a2c839.

chore: rename audit_log.source as 'source' is a reserved keyword in p…

2624969

…ostgres

chore: add additional fields to user.AuditData

c36c609

Refactored unauthorized access audit logging: (#381)

eb5e741

* Refactored unauthorized access audit logging: - moved audit logging responsibility to Authorizer interface - cleaned up injections of *db through several layers of code (no longer necessary) - minor cleanup and support work

chore: cleanup before review

0719913

Merge branch 'main' into populate-audit-log-fields

847c92c

juggernot325 marked this pull request as ready for review February 2, 2024 20:25

zinic requested changes Feb 2, 2024

View reviewed changes

cmd/api/src/database/migration/migrations/v5.6.0.sql Show resolved Hide resolved

cmd/api/src/api/middleware/middleware.go Outdated Show resolved Hide resolved

cmd/api/src/api/router/router.go Outdated Show resolved Hide resolved

cmd/api/src/api/middleware/middleware.go Show resolved Hide resolved

review comments

37726e5

zinic approved these changes Feb 5, 2024

View reviewed changes

irshadaj added 2 commits February 5, 2024 12:01

Merge branch 'main' into populate-audit-log-fields

9e3cbe7

undo testing change

ba742f5

irshadaj merged commit f14442a into main Feb 5, 2024
3 checks passed

irshadaj deleted the populate-audit-log-fields branch February 5, 2024 18:31

github-actions bot locked and limited conversation to collaborators Feb 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Audit Log Refactor #345

Audit Log Refactor #345

juggernot325 commented Jan 24, 2024 •

edited

Loading

zinic left a comment

zinic left a comment

Audit Log Refactor #345

Audit Log Refactor #345

Conversation

juggernot325 commented Jan 24, 2024 • edited Loading

Description

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Checklist:

zinic left a comment

Choose a reason for hiding this comment

zinic left a comment

Choose a reason for hiding this comment

juggernot325 commented Jan 24, 2024 •

edited

Loading