Merge pull request #224 from idaholab/v23.07.0_merge_idaholab

Malcolm v23.07.0 is a feature release with a number of improvements, bux fixes and component updates.

v23.05.1...v23.07.0

* New features
    - scan docker images built via GitHub actions for vulnerabilities using Trivy (#218)
    - document building and deplolying Malcolm with an AWS AMI image (#205)
    - handle Arkime field actions (#200)
    - kubernetes: document how to get running on Amazon EKS (#194)
    - Populate NetBox inventory via passively-gathered network traffic metadata (basic functionality, work in progress) (#135)

* Enhancements
    - use .tar.xz instead of .tar.gz for packaging Malcolm docker images for better compression (and smaller ISO file size)
    - Malcolm documentation edits (#204)
    - add option to enable SSH via password in hedgehog's configure-interfaces.py script (#158)
    - updated "Network Traffic Analysis with Malcolm" slides
    - use an init container in Kubernetes container startup to ensure necessary directories get created under PersistentVolume objects before startup
    - improvements to identifying source of third-party logs sent via fluent bit
    - don't do unnecessary clone of Zeek plugins, just install using URL
    - parse [bacnet_device_control.log](https://github.com/cisagov/icsnpp-bacnet/#device-control-log-bacnet_device_controllog) produced by the icsnpp-bacnet parser for Zeek

* Bug fixes
    - maxlogins value includes tmux sessions, can lock user out of SSH (#214)
    - curl rc file for connecting to external OpenSearch without auth enabled causes logstash startup to fail (#209)
    - failure to parse some suricata alerts due to integer type which should be indexed as long (#206)
    - netbox-restore doesn't work in Kubernetes (#202)
    - PCAP File with no `-` in pcapng Fails to Upload (cisagov#265)
    - disable NetBox telemetry

* Component version updates
    - Alpine (docker container image base) to [v3.18.0](https://www.alpinelinux.org/posts/Alpine-3.18.0-released.html)
    - Arkime to [v4.3.2](https://github.com/arkime/arkime/blob/8bd9d1ccaf3214eeb07da910c45d6172f9ff4ca8/CHANGELOG#L40-L55)
    - capa to [v6.0.0](https://github.com/mandiant/capa/releases/tag/v6.0.0)
    - filebeat to [v8.8.2](https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-8.8.2.html)
    - NetBox to [v3.5.4](https://github.com/netbox-community/netbox/releases/tag/v3.5.4)
    - OpenSearch and OpenSearch Dashboards to [v2.8.0](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.8.0.md)
    - Supercronic to [v0.2.25](https://github.com/aptible/supercronic/releases/tag/v0.2.25)
    - YARA to [v4.3.2](https://github.com/VirusTotal/yara/releases/tag/v4.3.2)
    - Zeek to [v5.2.2](https://github.com/zeek/zeek/releases/tag/v5.2.2)

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from [https://malcolm.fyi/](https://malcolm.fyi/docs/download.html).

.dockerignore

@@ -10,6 +10,7 @@
 **/.ldap_config_defaults
 **/htpasswd
 **/malcolm_*images.tar.gz
+**/malcolm_*images.tar.xz
 **/malcolm_netbox_backup_*.psql.gz
 **/.envrc
 **/.direnv

.github/workflows/api-build-and-push-ghcr.yml

@@ -12,6 +12,7 @@ on:
       - '!shared/bin/agg-init.sh'
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
+      - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'
       - '!shared/bin/zeek*'
@@ -26,6 +27,7 @@ jobs:
       actions: write
       packages: write
       contents: read
+      security-events: write
     steps:
       -
         name: Cancel previous run in progress
@@ -66,3 +68,24 @@ jobs:
           file: ./Dockerfiles/api.Dockerfile
           push: true
           tags: ghcr.io/${{ github.repository_owner }}/malcolm/api:${{ steps.extract_branch.outputs.branch }}
+      -
+        name: Run Trivy vulnerability scanner
+        id: trivy-scan
+        uses: aquasecurity/trivy-action@master
+        with:
+          scan-type: 'image'
+          scanners: 'vuln'
+          image-ref: ghcr.io/${{ github.repository_owner }}/malcolm/api:${{ steps.extract_branch.outputs.branch }}
+          format: 'sarif'
+          output: 'trivy-results.sarif'
+          severity: 'HIGH,CRITICAL'
+          vuln-type: 'os,library'
+          hide-progress: true
+          ignore-unfixed: true
+          exit-code: '0'
+      -
+        name: Upload Trivy scan results to GitHub Security tab
+        uses: github/codeql-action/upload-sarif@v2
+        if: always()
+        with:
+          sarif_file: 'trivy-results.sarif'

.github/workflows/arkime-build-and-push-ghcr.yml

@@ -12,6 +12,7 @@ on:
       - '!shared/bin/agg-init.sh'
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
+      - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'
       - '!shared/bin/zeek*'
@@ -26,6 +27,7 @@ jobs:
       actions: write
       packages: write
       contents: read
+      security-events: write
     steps:
       -
         name: Cancel previous run in progress
@@ -68,3 +70,24 @@ jobs:
             MAXMIND_GEOIP_DB_LICENSE_KEY=${{ secrets.MAXMIND_GEOIP_DB_LICENSE_KEY }}
           push: true
           tags: ghcr.io/${{ github.repository_owner }}/malcolm/arkime:${{ steps.extract_branch.outputs.branch }}
+      -
+        name: Run Trivy vulnerability scanner
+        id: trivy-scan
+        uses: aquasecurity/trivy-action@master
+        with:
+          scan-type: 'image'
+          scanners: 'vuln'
+          image-ref: ghcr.io/${{ github.repository_owner }}/malcolm/arkime:${{ steps.extract_branch.outputs.branch }}
+          format: 'sarif'
+          output: 'trivy-results.sarif'
+          severity: 'HIGH,CRITICAL'
+          vuln-type: 'os,library'
+          hide-progress: true
+          ignore-unfixed: true
+          exit-code: '0'
+      -
+        name: Upload Trivy scan results to GitHub Security tab
+        uses: github/codeql-action/upload-sarif@v2
+        if: always()
+        with:
+          sarif_file: 'trivy-results.sarif'

.github/workflows/dashboards-build-and-push-ghcr.yml

@@ -12,6 +12,7 @@ on:
       - '!shared/bin/agg-init.sh'
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
+      - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'
       - '!shared/bin/zeek*'
@@ -26,6 +27,7 @@ jobs:
       actions: write
       packages: write
       contents: read
+      security-events: write
     steps:
       -
         name: Cancel previous run in progress
@@ -66,3 +68,24 @@ jobs:
           file: ./Dockerfiles/dashboards.Dockerfile
           push: true
           tags: ghcr.io/${{ github.repository_owner }}/malcolm/dashboards:${{ steps.extract_branch.outputs.branch }}
+      -
+        name: Run Trivy vulnerability scanner
+        id: trivy-scan
+        uses: aquasecurity/trivy-action@master
+        with:
+          scan-type: 'image'
+          scanners: 'vuln'
+          image-ref: ghcr.io/${{ github.repository_owner }}/malcolm/dashboards:${{ steps.extract_branch.outputs.branch }}
+          format: 'sarif'
+          output: 'trivy-results.sarif'
+          severity: 'HIGH,CRITICAL'
+          vuln-type: 'os,library'
+          hide-progress: true
+          ignore-unfixed: true
+          exit-code: '0'
+      -
+        name: Upload Trivy scan results to GitHub Security tab
+        uses: github/codeql-action/upload-sarif@v2
+        if: always()
+        with:
+          sarif_file: 'trivy-results.sarif'

.github/workflows/dashboards-helper-build-and-push-ghcr.yml

@@ -12,6 +12,7 @@ on:
       - '!shared/bin/agg-init.sh'
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
+      - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'
       - '!shared/bin/zeek*'
@@ -26,6 +27,7 @@ jobs:
       actions: write
       packages: write
       contents: read
+      security-events: write
     steps:
       -
         name: Cancel previous run in progress
@@ -66,3 +68,24 @@ jobs:
           file: ./Dockerfiles/dashboards-helper.Dockerfile
           push: true
           tags: ghcr.io/${{ github.repository_owner }}/malcolm/dashboards-helper:${{ steps.extract_branch.outputs.branch }}
+      -
+        name: Run Trivy vulnerability scanner
+        id: trivy-scan
+        uses: aquasecurity/trivy-action@master
+        with:
+          scan-type: 'image'
+          scanners: 'vuln'
+          image-ref: ghcr.io/${{ github.repository_owner }}/malcolm/dashboards-helper:${{ steps.extract_branch.outputs.branch }}
+          format: 'sarif'
+          output: 'trivy-results.sarif'
+          severity: 'HIGH,CRITICAL'
+          vuln-type: 'os,library'
+          hide-progress: true
+          ignore-unfixed: true
+          exit-code: '0'
+      -
+        name: Upload Trivy scan results to GitHub Security tab
+        uses: github/codeql-action/upload-sarif@v2
+        if: always()
+        with:
+          sarif_file: 'trivy-results.sarif'

.github/workflows/dirinit-build-and-push-ghcr.yml

@@ -0,0 +1,83 @@
+name: dirinit-build-and-push-ghcr
+
+on:
+  push:
+    branches:
+      - main
+      - development
+    paths:
+      - 'Dockerfiles/dirinit.Dockerfile'
+      - 'shared/bin/docker-uid-gid-setup.sh'
+      - '.trigger_workflow_build'
+  workflow_dispatch:
+  repository_dispatch:
+
+jobs:
+  docker:
+    runs-on: ubuntu-22.04
+    permissions:
+      actions: write
+      packages: write
+      contents: read
+      security-events: write
+    steps:
+      -
+        name: Cancel previous run in progress
+        uses: styfle/cancel-workflow-action@0.11.0
+        with:
+          ignore_sha: true
+          all_but_latest: true
+          access_token: ${{ secrets.GITHUB_TOKEN }}
+      -
+        name: Checkout
+        uses: actions/checkout@v3
+      -
+        name: Extract branch name
+        shell: bash
+        run: echo "branch=$(echo ${GITHUB_REF#refs/heads/})" >> $GITHUB_OUTPUT
+        id: extract_branch
+      -
+        name: Set up QEMU
+        uses: docker/setup-qemu-action@v2
+      -
+        name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v2
+        with:
+          driver-opts: |
+            image=moby/buildkit:master
+      -
+        name: Log in to registry
+        uses: docker/login-action@v2
+        with:
+          registry: ghcr.io
+          username: ${{ github.repository_owner }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+      -
+        name: Build and push
+        uses: docker/build-push-action@v3
+        with:
+          context: .
+          file: ./Dockerfiles/dirinit.Dockerfile
+          push: true
+          tags: ghcr.io/${{ github.repository_owner }}/malcolm/dirinit:${{ steps.extract_branch.outputs.branch }}
+      -
+        name: Run Trivy vulnerability scanner
+        id: trivy-scan
+        uses: aquasecurity/trivy-action@master
+        with:
+          scan-type: 'image'
+          scanners: 'vuln'
+          image-ref: ghcr.io/${{ github.repository_owner }}/malcolm/dirinit:${{ steps.extract_branch.outputs.branch }}
+          format: 'sarif'
+          output: 'trivy-results.sarif'
+          severity: 'HIGH,CRITICAL'
+          vuln-type: 'os,library'
+          hide-progress: true
+          ignore-unfixed: true
+          exit-code: '0'
+      -
+        name: Upload Trivy scan results to GitHub Security tab
+        uses: github/codeql-action/upload-sarif@v2
+        if: always()
+        with:
+          sarif_file: 'trivy-results.sarif'

.github/workflows/file-monitor-build-and-push-ghcr.yml

@@ -12,6 +12,7 @@ on:
       - '!shared/bin/agg-init.sh'
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
+      - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'
       - '!shared/bin/zeek*.sh'
@@ -26,6 +27,7 @@ jobs:
       actions: write
       packages: write
       contents: read
+      security-events: write
     steps:
       -
         name: Cancel previous run in progress
@@ -66,3 +68,24 @@ jobs:
           file: ./Dockerfiles/file-monitor.Dockerfile
           push: true
           tags: ghcr.io/${{ github.repository_owner }}/malcolm/file-monitor:${{ steps.extract_branch.outputs.branch }}
+      -
+        name: Run Trivy vulnerability scanner
+        id: trivy-scan
+        uses: aquasecurity/trivy-action@master
+        with:
+          scan-type: 'image'
+          scanners: 'vuln'
+          image-ref: ghcr.io/${{ github.repository_owner }}/malcolm/file-monitor:${{ steps.extract_branch.outputs.branch }}
+          format: 'sarif'
+          output: 'trivy-results.sarif'
+          severity: 'HIGH,CRITICAL'
+          vuln-type: 'os,library'
+          hide-progress: true
+          ignore-unfixed: true
+          exit-code: '0'
+      -
+        name: Upload Trivy scan results to GitHub Security tab
+        uses: github/codeql-action/upload-sarif@v2
+        if: always()
+        with:
+          sarif_file: 'trivy-results.sarif'

.github/workflows/file-upload-build-and-push-ghcr.yml

@@ -12,6 +12,7 @@ on:
       - '!shared/bin/agg-init.sh'
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
+      - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'
       - '!shared/bin/zeek*'
@@ -26,6 +27,7 @@ jobs:
       actions: write
       packages: write
       contents: read
+      security-events: write
     steps:
       -
         name: Cancel previous run in progress
@@ -66,3 +68,24 @@ jobs:
           file: ./Dockerfiles/file-upload.Dockerfile
           push: true
           tags: ghcr.io/${{ github.repository_owner }}/malcolm/file-upload:${{ steps.extract_branch.outputs.branch }}
+      -
+        name: Run Trivy vulnerability scanner
+        id: trivy-scan
+        uses: aquasecurity/trivy-action@master
+        with:
+          scan-type: 'image'
+          scanners: 'vuln'
+          image-ref: ghcr.io/${{ github.repository_owner }}/malcolm/file-upload:${{ steps.extract_branch.outputs.branch }}
+          format: 'sarif'
+          output: 'trivy-results.sarif'
+          severity: 'HIGH,CRITICAL'
+          vuln-type: 'os,library'
+          hide-progress: true
+          ignore-unfixed: true
+          exit-code: '0'
+      -
+        name: Upload Trivy scan results to GitHub Security tab
+        uses: github/codeql-action/upload-sarif@v2
+        if: always()
+        with:
+          sarif_file: 'trivy-results.sarif'

.github/workflows/filebeat-build-and-push-ghcr.yml

@@ -12,6 +12,7 @@ on:
       - '!shared/bin/agg-init.sh'
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
+      - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'
       - '!shared/bin/zeek*'
@@ -26,6 +27,7 @@ jobs:
       actions: write
       packages: write
       contents: read
+      security-events: write
     steps:
       -
         name: Cancel previous run in progress
@@ -66,3 +68,24 @@ jobs:
           file: ./Dockerfiles/filebeat.Dockerfile
           push: true
           tags: ghcr.io/${{ github.repository_owner }}/malcolm/filebeat-oss:${{ steps.extract_branch.outputs.branch }}
+      -
+        name: Run Trivy vulnerability scanner
+        id: trivy-scan
+        uses: aquasecurity/trivy-action@master
+        with:
+          scan-type: 'image'
+          scanners: 'vuln'
+          image-ref: ghcr.io/${{ github.repository_owner }}/malcolm/filebeat-oss:${{ steps.extract_branch.outputs.branch }}
+          format: 'sarif'
+          output: 'trivy-results.sarif'
+          severity: 'HIGH,CRITICAL'
+          vuln-type: 'os,library'
+          hide-progress: true
+          ignore-unfixed: true
+          exit-code: '0'
+      -
+        name: Upload Trivy scan results to GitHub Security tab
+        uses: github/codeql-action/upload-sarif@v2
+        if: always()
+        with:
+          sarif_file: 'trivy-results.sarif'