Skip to content

HADOOP-18257. Analyze S3A Audit Logs #6901

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

HADOOP-18257. Analyze S3A Audit Logs #6901

wants to merge 1 commit into from

Conversation

steveloughran
Copy link
Contributor

Description of PR

HADOOP-18257. Analyze S3A Audit Logs

This is #6000 rebased and squashed with a couple of changes on top.

  • tool is invoked through hadoop s3guard command
  • which can now also be invoked as "hadoop s3a"! (not documented, maybe we should isolate?)
  • tests are improved.
  • OperationDuration implements DurationTracker for bit more completeness

TODO

  • split out the record parsing into a hadoop MR record read/write
  • so support large scale parsing
  • cli tool just glues that together either for small parallelised extraction
    or for aggregation to one file.
  • but a bulk job would work with a larger dataset

How was this patch tested?

new unit test cases.

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 6m 38s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 xmllint 0m 0s xmllint was not available.
+0 🆗 shelldocs 0m 0s Shelldocs was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 4 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 14m 21s Maven dependency ordering for branch
+1 💚 mvninstall 20m 18s trunk passed
+1 💚 compile 8m 38s trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚 compile 8m 2s trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚 checkstyle 2m 3s trunk passed
+1 💚 mvnsite 14m 36s trunk passed
+1 💚 javadoc 4m 51s trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚 javadoc 4m 53s trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
-1 ❌ spotbugs 17m 6s /branch-spotbugs-root-warnings.html root in trunk has 2 extant spotbugs warnings.
+1 💚 shadedclient 36m 36s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 30s Maven dependency ordering for patch
+1 💚 mvninstall 18m 35s the patch passed
+1 💚 compile 8m 24s the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚 javac 8m 24s the patch passed
+1 💚 compile 8m 6s the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚 javac 8m 6s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 2m 0s /results-checkstyle-root.txt root: The patch generated 7 new + 2 unchanged - 0 fixed = 9 total (was 2)
+1 💚 mvnsite 8m 41s the patch passed
+1 💚 shellcheck 0m 1s No new issues.
+1 💚 javadoc 4m 45s the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚 javadoc 4m 54s the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
-1 ❌ spotbugs 0m 46s /new-spotbugs-hadoop-tools_hadoop-aws.html hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
-1 ❌ spotbugs 17m 27s /new-spotbugs-root.html root generated 1 new + 2 unchanged - 0 fixed = 3 total (was 2)
+1 💚 shadedclient 37m 11s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 645m 36s /patch-unit-root.txt root in the patch passed.
+1 💚 asflicense 1m 4s The patch does not generate ASF License warnings.
868m 4s
Reason Tests
SpotBugs module:hadoop-tools/hadoop-aws
Found reliance on default encoding in org.apache.hadoop.fs.s3a.audit.AuditTool.run(String[], PrintStream):in org.apache.hadoop.fs.s3a.audit.AuditTool.run(String[], PrintStream): new java.io.PrintWriter(OutputStream) At AuditTool.java:[line 122]
SpotBugs module:root
Found reliance on default encoding in org.apache.hadoop.fs.s3a.audit.AuditTool.run(String[], PrintStream):in org.apache.hadoop.fs.s3a.audit.AuditTool.run(String[], PrintStream): new java.io.PrintWriter(OutputStream) At AuditTool.java:[line 122]
Failed junit tests hadoop.hdfs.server.blockmanagement.TestDatanodeManager
Subsystem Report/Notes
Docker ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6901/1/artifact/out/Dockerfile
GITHUB PR #6901
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint shellcheck shelldocs
uname Linux 346db37e4260 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / f3e1bd5
Default Java Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6901/1/testReport/
Max. process+thread count 4268 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws . U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6901/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@steveloughran steveloughran force-pushed the s3/HADOOP-18257-audit-log-parse branch from f3e1bd5 to fe0322a Compare August 7, 2024 19:01
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 19s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 xmllint 0m 0s xmllint was not available.
+0 🆗 shelldocs 0m 0s Shelldocs was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 4 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 15m 25s Maven dependency ordering for branch
+1 💚 mvninstall 20m 22s trunk passed
+1 💚 compile 9m 4s trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚 compile 8m 22s trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚 checkstyle 2m 17s trunk passed
+1 💚 mvnsite 18m 35s trunk passed
+1 💚 javadoc 5m 3s trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚 javadoc 5m 0s trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
-1 ❌ spotbugs 17m 35s /branch-spotbugs-root-warnings.html root in trunk has 2 extant spotbugs warnings.
+1 💚 shadedclient 37m 33s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 30s Maven dependency ordering for patch
+1 💚 mvninstall 18m 53s the patch passed
+1 💚 compile 8m 41s the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚 javac 8m 41s the patch passed
+1 💚 compile 8m 24s the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚 javac 8m 24s the patch passed
-1 ❌ blanks 0m 0s /blanks-eol.txt The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️ checkstyle 2m 2s /results-checkstyle-root.txt root: The patch generated 2 new + 2 unchanged - 0 fixed = 4 total (was 2)
+1 💚 mvnsite 11m 36s the patch passed
+1 💚 shellcheck 0m 0s No new issues.
+1 💚 javadoc 4m 54s the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚 javadoc 5m 5s the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚 spotbugs 19m 48s the patch passed
+1 💚 shadedclient 37m 41s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 639m 13s /patch-unit-root.txt root in the patch passed.
+1 💚 asflicense 0m 56s The patch does not generate ASF License warnings.
867m 41s
Reason Tests
Failed junit tests hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2
Subsystem Report/Notes
Docker ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6901/2/artifact/out/Dockerfile
GITHUB PR #6901
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint shellcheck shelldocs
uname Linux 913d0845eac3 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / fe0322a
Default Java Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6901/2/testReport/
Max. process+thread count 4010 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws . U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6901/2/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 21s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 xmllint 0m 0s xmllint was not available.
+0 🆗 shelldocs 0m 0s Shelldocs was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 4 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 15m 53s Maven dependency ordering for branch
+1 💚 mvninstall 21m 24s trunk passed
+1 💚 compile 10m 1s trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚 compile 9m 13s trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚 checkstyle 2m 25s trunk passed
+1 💚 mvnsite 16m 47s trunk passed
+1 💚 javadoc 5m 19s trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚 javadoc 5m 3s trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
-1 ❌ spotbugs 18m 8s /branch-spotbugs-root-warnings.html root in trunk has 2 extant spotbugs warnings.
+1 💚 shadedclient 40m 7s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 33s Maven dependency ordering for patch
+1 💚 mvninstall 21m 50s the patch passed
+1 💚 compile 9m 56s the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚 javac 9m 56s the patch passed
+1 💚 compile 9m 27s the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚 javac 9m 27s the patch passed
-1 ❌ blanks 0m 0s /blanks-eol.txt The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️ checkstyle 2m 15s /results-checkstyle-root.txt root: The patch generated 8 new + 2 unchanged - 0 fixed = 10 total (was 2)
+1 💚 mvnsite 9m 49s the patch passed
+1 💚 shellcheck 0m 0s No new issues.
+1 💚 javadoc 4m 24s the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
-1 ❌ javadoc 4m 33s /patch-javadoc-root-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt root in the patch failed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05.
-1 ❌ spotbugs 0m 49s /new-spotbugs-hadoop-tools_hadoop-aws.html hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
-1 ❌ spotbugs 17m 55s /new-spotbugs-root.html root generated 1 new + 2 unchanged - 0 fixed = 3 total (was 2)
+1 💚 shadedclient 37m 57s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 639m 16s /patch-unit-root.txt root in the patch passed.
+1 💚 asflicense 1m 8s The patch does not generate ASF License warnings.
875m 29s
Reason Tests
SpotBugs module:hadoop-tools/hadoop-aws
org.apache.hadoop.fs.s3a.audit.mapreduce.S3AAuditLogMergerAndParser.FIELDS_OF_TYPE_LONG is a mutable collection which should be package protected At S3AAuditLogMergerAndParser.java:which should be package protected At S3AAuditLogMergerAndParser.java:[line 76]
SpotBugs module:root
org.apache.hadoop.fs.s3a.audit.mapreduce.S3AAuditLogMergerAndParser.FIELDS_OF_TYPE_LONG is a mutable collection which should be package protected At S3AAuditLogMergerAndParser.java:which should be package protected At S3AAuditLogMergerAndParser.java:[line 76]
Failed junit tests hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2
hadoop.fs.s3a.audit.TestAuditTool
Subsystem Report/Notes
Docker ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6901/3/artifact/out/Dockerfile
GITHUB PR #6901
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint shellcheck shelldocs
uname Linux 48468eb1eff9 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 7303563
Default Java Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6901/3/testReport/
Max. process+thread count 4787 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws . U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6901/3/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@steveloughran steveloughran force-pushed the s3/HADOOP-18257-audit-log-parse branch from 7303563 to fb2a10e Compare November 18, 2024 21:36
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 20s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 xmllint 0m 0s xmllint was not available.
+0 🆗 shelldocs 0m 0s Shelldocs was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 4 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 15m 34s Maven dependency ordering for branch
+1 💚 mvninstall 19m 19s trunk passed
+1 💚 compile 8m 57s trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 compile 8m 22s trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 checkstyle 2m 4s trunk passed
+1 💚 mvnsite 14m 58s trunk passed
+1 💚 javadoc 5m 20s trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 4m 58s trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 spotbugs 19m 17s trunk passed
+1 💚 shadedclient 37m 30s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 30s Maven dependency ordering for patch
-1 ❌ mvninstall 0m 14s /patch-mvninstall-hadoop-tools_hadoop-aws.txt hadoop-aws in the patch failed.
-1 ❌ mvninstall 18m 5s /patch-mvninstall-root.txt root in the patch failed.
-1 ❌ compile 8m 23s /patch-compile-root-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt root in the patch failed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.
-1 ❌ javac 8m 23s /patch-compile-root-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt root in the patch failed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.
-1 ❌ compile 8m 10s /patch-compile-root-jdkPrivateBuild-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga.txt root in the patch failed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga.
-1 ❌ javac 8m 10s /patch-compile-root-jdkPrivateBuild-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga.txt root in the patch failed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga.
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 2m 31s /results-checkstyle-root.txt root: The patch generated 6 new + 2 unchanged - 0 fixed = 8 total (was 2)
-1 ❌ mvnsite 2m 56s /patch-mvnsite-root.txt root in the patch failed.
+1 💚 shellcheck 0m 0s No new issues.
-1 ❌ javadoc 4m 55s /patch-javadoc-root-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt root in the patch failed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.
-1 ❌ javadoc 4m 33s /patch-javadoc-root-jdkPrivateBuild-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga.txt root in the patch failed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga.
-1 ❌ spotbugs 0m 21s /patch-spotbugs-hadoop-tools_hadoop-aws.txt hadoop-aws in the patch failed.
-1 ❌ spotbugs 15m 50s /patch-spotbugs-root.txt root in the patch failed.
+1 💚 shadedclient 21m 25s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 647m 48s /patch-unit-root.txt root in the patch failed.
+1 💚 asflicense 1m 6s The patch does not generate ASF License warnings.
860m 43s
Reason Tests
Failed junit tests hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy
hadoop.hdfs.TestDecommissionWithBackoffMonitor
hadoop.hdfs.TestDecommission
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6901/4/artifact/out/Dockerfile
GITHUB PR #6901
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint shellcheck shelldocs
uname Linux a5a51b1cb4b9 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / fb2a10e
Default Java Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6901/4/testReport/
Max. process+thread count 4350 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws . U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6901/4/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 18s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 xmllint 0m 0s xmllint was not available.
+0 🆗 shelldocs 0m 0s Shelldocs was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 4 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 14m 38s Maven dependency ordering for branch
+1 💚 mvninstall 20m 40s trunk passed
+1 💚 compile 10m 36s trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 compile 8m 49s trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 checkstyle 2m 20s trunk passed
+1 💚 mvnsite 14m 54s trunk passed
+1 💚 javadoc 5m 24s trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 4m 53s trunk passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 spotbugs 19m 18s trunk passed
+1 💚 shadedclient 37m 29s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 30s Maven dependency ordering for patch
+1 💚 mvninstall 18m 46s the patch passed
+1 💚 compile 8m 43s the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚 javac 8m 43s the patch passed
+1 💚 compile 8m 30s the patch passed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
+1 💚 javac 8m 30s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 2m 4s /results-checkstyle-root.txt root: The patch generated 6 new + 2 unchanged - 0 fixed = 8 total (was 2)
+1 💚 mvnsite 9m 58s the patch passed
+1 💚 shellcheck 0m 1s No new issues.
-1 ❌ javadoc 4m 48s /patch-javadoc-root-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt root in the patch failed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.
-1 ❌ javadoc 4m 32s /patch-javadoc-root-jdkPrivateBuild-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga.txt root in the patch failed with JDK Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga.
-1 ❌ spotbugs 0m 49s /new-spotbugs-hadoop-tools_hadoop-aws.html hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
-1 ❌ spotbugs 17m 32s /new-spotbugs-root.html root generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚 shadedclient 37m 33s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 647m 39s /patch-unit-root.txt root in the patch passed.
+1 💚 asflicense 0m 52s The patch does not generate ASF License warnings.
871m 52s
Reason Tests
SpotBugs module:hadoop-tools/hadoop-aws
org.apache.hadoop.fs.s3a.audit.mapreduce.S3AAuditLogMergerAndParser.FIELDS_OF_TYPE_LONG is a mutable collection which should be package protected At S3AAuditLogMergerAndParser.java:which should be package protected At S3AAuditLogMergerAndParser.java:[line 76]
SpotBugs module:root
org.apache.hadoop.fs.s3a.audit.mapreduce.S3AAuditLogMergerAndParser.FIELDS_OF_TYPE_LONG is a mutable collection which should be package protected At S3AAuditLogMergerAndParser.java:which should be package protected At S3AAuditLogMergerAndParser.java:[line 76]
Failed junit tests hadoop.hdfs.TestDecommissionWithBackoffMonitor
hadoop.hdfs.TestDecommission
hadoop.fs.s3a.audit.TestAuditTool
Subsystem Report/Notes
Docker ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6901/5/artifact/out/Dockerfile
GITHUB PR #6901
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint shellcheck shelldocs
uname Linux 81b7addc1e9b 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / e012694
Default Java Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-gaus1-0ubuntu220.04-ga
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6901/5/testReport/
Max. process+thread count 4598 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws . U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6901/5/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@steveloughran
HADOOP-18257. avro log stuff
8cd6bec 
Updated for the avro upgrade to compile, but
doesn't deal with the new avro tightening which
needs to be addressed in production.

Change-Id: Iab54a5fc151f7c447b1b84fb14dc8afb2a7d5573

HADOOP-18257. S3AAuditLogMergerAndParser changes

* dest is a path to the avro file to create containing all
  of the files parsed.
* tests updated with changed API

Change-Id: I810b165a4c6d2354bc91523594c7a24a4c031aee

HADOOP-18257. audit log: ParsedAuditLogEntry

Change-Id: Iba0d57357cc04aad571f9f0f4d69ee7f1dd11712

HADOOP-18257. audit log

* moved avro source
* WiP of a serializable/writable object for easy use in MR, spark

Change-Id: I56bbff0ad2d71b64984f4cc59e1f2c83ea25597a

HADOOP-18257. spotbugs

Change-Id: Ia0ab414bcca8a898eca95fe9ddd89b5bfcef3028

HADOOP-18257 audit log parser

* tool is invoked through hadoop s3guard command
* which can now also be invoked as "hadoop s3a"!
* tests are improved.
* OperationDuration implements DurationTracker for bit more completeness

TODO
* split out the record parsing into a hadoop MR record read/write
* so support large scale parsing
* cli tool just glues that together either for small parallelised extraction
  or for aggregation to one file.
* but a bulk job would work with a larger dataset

Change-Id: I25e333592d1058b460b0bfda5313a20de13c2e35

HADOOP-18257 audit log parser

Initial round of changes
* only log at debug during parsing
* cli to take <path to source> <path of output file>
  where the output is a filename not a dir.

Change-Id: Ibdba37e42a36b1933979d2a1e7265d07069ea0bd

HADOOP-18257. Merging and Parsing S3A audit logs into Avro format for analysis.
@steveloughran steveloughran force-pushed the s3/HADOOP-18257-audit-log-parse branch from e012694 to 8cd6bec Compare April 14, 2025 16:24
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 20s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 xmllint 0m 0s xmllint was not available.
+0 🆗 shelldocs 0m 0s Shelldocs was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 4 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 7m 8s Maven dependency ordering for branch
+1 💚 mvninstall 18m 51s trunk passed
+1 💚 compile 8m 15s trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 compile 7m 21s trunk passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 checkstyle 1m 55s trunk passed
+1 💚 mvnsite 14m 5s trunk passed
+1 💚 javadoc 6m 5s trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javadoc 4m 46s trunk passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 spotbugs 19m 27s trunk passed
+1 💚 shadedclient 42m 11s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 28s Maven dependency ordering for patch
+1 💚 mvninstall 20m 38s the patch passed
+1 💚 compile 8m 57s the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚 javac 8m 57s the patch passed
+1 💚 compile 7m 37s the patch passed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
+1 💚 javac 7m 37s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 1m 54s /results-checkstyle-root.txt root: The patch generated 6 new + 2 unchanged - 0 fixed = 8 total (was 2)
+1 💚 mvnsite 12m 42s the patch passed
+1 💚 shellcheck 0m 0s No new issues.
-1 ❌ javadoc 4m 33s /patch-javadoc-root-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt root in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
-1 ❌ javadoc 4m 16s /patch-javadoc-root-jdkPrivateBuild-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06.txt root in the patch failed with JDK Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06.
-1 ❌ spotbugs 0m 45s /new-spotbugs-hadoop-tools_hadoop-aws.html hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
-1 ❌ spotbugs 18m 34s /new-spotbugs-root.html root generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚 shadedclient 39m 38s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 493m 24s /patch-unit-root.txt root in the patch passed.
+1 💚 asflicense 1m 0s The patch does not generate ASF License warnings.
713m 36s
Reason Tests
SpotBugs module:hadoop-tools/hadoop-aws
org.apache.hadoop.fs.s3a.audit.mapreduce.S3AAuditLogMergerAndParser.FIELDS_OF_TYPE_LONG is a mutable collection which should be package protected At S3AAuditLogMergerAndParser.java:which should be package protected At S3AAuditLogMergerAndParser.java:[line 76]
SpotBugs module:root
org.apache.hadoop.fs.s3a.audit.mapreduce.S3AAuditLogMergerAndParser.FIELDS_OF_TYPE_LONG is a mutable collection which should be package protected At S3AAuditLogMergerAndParser.java:which should be package protected At S3AAuditLogMergerAndParser.java:[line 76]
Failed junit tests hadoop.fs.s3a.audit.TestAuditTool
Subsystem Report/Notes
Docker ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6901/6/artifact/out/Dockerfile
GITHUB PR #6901
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint shellcheck shelldocs
uname Linux 12bcbce6ac71 5.15.0-136-generic #147-Ubuntu SMP Sat Mar 15 15:53:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 8cd6bec
Default Java Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06us1-0ubuntu120.04-b06
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6901/6/testReport/
Max. process+thread count 2975 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws . U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6901/6/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@steveloughran steveloughran marked this pull request as draft April 16, 2025 12:25
@steveloughran
Copy link
Contributor Author

hasn't adaped to the 11.4 upgrade as it was using avro ser/deser as part of the workflow, and that can't be done with the class restrictions.

@github-actions GitHub Actions
Copy link
Contributor

We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you feel like this was a mistake, or you would like to continue working on it, please feel free to re-open it and ask for a committer to remove the stale tag and review again.
Thanks all for your contribution.

@github-actions github-actions bot added the Stale label Sep 25, 2025
@github-actions github-actions bot closed this Sep 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
AWS build Common Stale TOOLS trunk
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@steveloughran @hadoop-yetus