Feature/113 info permissions config #114
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…file.permissions` config value - tests use MiniDfsCluster to assert controlled correct behavior
…missions, buildfix)
dk1844
commented
Oct 1, 2021
benedeki
reviewed
Oct 1, 2021
| import org.apache.hadoop.hdfs.MiniDFSCluster | ||
| import org.scalatest.{BeforeAndAfterAll, Suite} | ||
|
|
||
| trait MiniDfsClusterBase extends BeforeAndAfterAll { this: Suite => |
…licly exposed; HdfsFileUtils.getInfoFilePermissionsFromConfig does not use the HdfsFileUtils.DefaultFilePermissions in any way, the user is supposed to compose it on his own by e.g.: `HdfsFileUtils.getInfoFilePermissionsFromConfig().getOrElse(HdfsFileUtils.DefaultFilePermissions)`
benedeki
approved these changes
Oct 6, 2021
|
Tested withing Enceladus on local HDFS. Works as expected. |
Zejnilovic
added a commit
that referenced
this pull request
Apr 13, 2022
* #88: Add some files to configure the project (#89) * #88: Add some files configure the project * Git configuration * Scalastyle support * Ensured no Scalastyle errors * Added CRLF for Windows *.bat and *.cmd files to .editorconfig * Added Spark 3.1 build into the `build-all.sh` script * Created Windows `build-all.cmd` script * Upgrade from Spark 3.1.1 to 3.1.2 (fixes several issues of the previous version) * `--no-transfer-progress` added to build.yml * #97 Aggregate control type strategy (#107) * #97 AggregateControlTypeStrategy suggested API for ControlMeasureBuilder usage * #97 ControlMeasureBuilder.withAggregateColumn(s) implementations. Todo needs testing, cleanup & documentation update MeasurementProcessor split into object/class to offer generic processing methods to be reusable. * #97 ControlMeasureBuilder.withAggregateColumn(s) unit tests added (regression guard) * #97 ControlMeasureBuilder.withAggregateColumn(s) in README.md, original only-default `cmBuilder.calculateMeasurement` removed * [maven-release-plugin] prepare release v3.6.0 * [maven-release-plugin] prepare for next development iteration * #97 readme update - ControlMeasureBuilder API (#110) * #97 readme update (related to #97, too) * #97 maven central version badge added * Feature/113 info permissions config (#114) * #113 atum info file permissions for hdfs loaded from `atum.hdfs.info.file.permissions` config value - tests use MiniDfsCluster to assert controlled correct behavior - test update (custom MiniDfsCluster with umask 000 allows max permissions) - HdfsFileUtils.DefaultFilePermissions is now publicly exposed; the user is expected to call compose the default and configured value it on his own by e.g.: `HdfsFileUtils.getInfoFilePermissionsFromConfig().getOrElse(HdfsFileUtils.DefaultFilePermissions)` * #77 Fix parameter handling bug in CreateInfoFileToolCSV (#78) * #121 sbt cross comptilation * #121 multiversion build (scala, spark, json4s). * #121 hadoop3 used for Spark3/Scala2.12 * #121 sbt github autobuild * #125 publish, pgp/gpg plugin added; sbt-sonatype howto referenced; model, parent prefixed by `atum-` to conform to the mvn publish, cleanup * Upgrade dependencies and remove MiniDFSCluster * GH Action fix * Remove pom.xml files * Add licence header and header check * Fix examples Co-authored-by: David Benedeki <14905969+benedeki@users.noreply.github.com> Co-authored-by: Daniel K <dk1844@gmail.com> Co-authored-by: Jan Scherbaum <kmoj02@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Inspired by Spline, Atum now also writes allows setting file permissions to write
_INFOfiles to HDFS. If no value is given, the default is derived from local umask, just like in Spline.The preferred permissions can be given in typesafe config under
atum.hdfs.info.file.permissions, so e.g.atum.hdfs.info.file.permissions=711(However, please be aware that the resulting permission is limited by local
umask. (if u mask is066and you desire777to be written, you only get711))000to demonstrate preferred permission ranging from000to777.Test run:
Tested in Enceladus: AbsaOSS/enceladus#1934