diff --git a/README.md b/README.md index 0a13a26034f..c5f3fde1f98 100644 --- a/README.md +++ b/README.md @@ -265,6 +265,9 @@ Example usage: The super-linter allows you to pass the following `ENV` variables to be able to trigger different functionality. +
+ The behavior in this section has changed with 23andMe's custom logic (see next section). Expand to see original. + _Note:_ All the `VALIDATE_[LANGUAGE]` variables behave in a very specific way: - If none of them are passed, then they all default to true. @@ -274,6 +277,7 @@ _Note:_ All the `VALIDATE_[LANGUAGE]` variables behave in a very specific way: This means that if you run the linter "out of the box", all languages will be checked. But if you wish to select or exclude specific linters, we give you full control to choose which linters are run, and won't run anything unexpected. +
| **ENV VAR** | **Default Value** | **Notes** | | ---------------------------------- | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | @@ -383,17 +387,34 @@ But if you wish to select or exclude specific linters, we give you full control | **VALIDATE_YAML** | `true` | Flag to enable or disable the linting process of the YAML language. | | **YAML_CONFIG_FILE** | `.yaml-lint.yml` | Filename for [Yamllint configuration](https://yamllint.readthedocs.io/en/stable/configuration.html) (ex: `.yaml-lint.yml`, `.yamllint.yml`) | -### 23andMe Custom Environment Variables +### 23andMe Custom Behavior and Environment Variables + +23andMe's custom super-linter introduced the ability to group linters into language packs, which can then be toggled on or off as a group. +As a result, the `VALIDATE_[LANGUAGE]` variables behave differently from those in the base super-linter: + +- **All variables default to `false`, unless explicitly toggled to true.** +- A single linter/language can appear in multiple linter packs. Its `VALIDATE_[LANGUAGE]` variable will be set to `true` if *any* of the language packs containing it are marked `true`, *except*... +- If there is an explicit `VALIDATE_[LANGUAGE]` variable for that specific linter/language, the variable will override all language pack settings. + | **ENV VAR** | **Default Value** | **Notes** | | -------------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **CLOUDFORMATION_CFN_NAG_CONFIG_FILE** | `.cfnnag` | Filename for [cfn-nag configuration](https://github.com/stelligent/cfn_nag#profiles) | | **GITLEAKS_CONFIG_FILE** | `none` | Filename for optional [gitleaks configuration](https://github.com/zricethezav/gitleaks#configuration) (ex: `simple_regex_config.toml`). Uses the [gitleaks default](https://github.com/zricethezav/gitleaks/blob/master/config/default.go) if unspecified | | **OUTPUT_MODE** | `none` | Additional means for outputting findings, other than writing to stdout. Supported modes: lintly | | **PYTHON_BANDIT_CONFIG_FILE** | `.bandit` | Filename for [bandit configuration](https://bandit.readthedocs.io/en/latest/config.html) (ex: `.bandit`, `tox.ini`) | -| **VALIDATE_CLOUDFORMATION_CFN_NAG** | `true` | Flag to enable or disable the security linting process of the AWS CloudFormation language. | +| **RUN_CODE_QUALITY_TOOLS** | `false ` | Flag to enable or disable the entire suite of code-quality-related linters. See below for a list of linters in this pack. | +| **RUN_SECURITY_TOOLS** | `true ` | Flag to enable or disable the entire suite of security tools. See below for a list of tools in this pack. | +| **VALIDATE_CLOUDFORMATION_CFN_NAG** | `true` | Flag to enable or disable the security linting process of the AWS CloudFormation language. | | **VALIDATE_GITLEAKS** | `true` | Flag to enable or disable the linting process of potentially leaked secrets across all files and languages. | | **VALIDATE_PYTHON_BANDIT** | `true` | Flag to enable or disable the linting process of the Python language. (Utilizing: bandit) | +The following is a list of supported language packs. + +| **Language pack** | **Included Tools** | +| -------------------------- | ------------------------------------------------------------------------------------------------------------------- | +| **RUN_SECURITY_TOOLS** | `CLOUDFORMATION_CFN_NAG`
`DOCKERFILE_HADOLINT`
`GITLEAKS`
`PYTHON_BANDIT`
`TERRAFORM_TERRASCAN` | +| **RUN_CODE_QUALITY_TOOLS** | _(All other tools that are currently not security tools)_ | + ### Template rules files You can use the **GitHub** **Super-Linter** _with_ or _without_ your own personal rules sets. This allows for greater flexibility for each individual code base. The Template rules all try to follow the standards we believe should be enabled at the basic level. diff --git a/lib/functions/validation.sh b/lib/functions/validation.sh index d97597078d0..45c64f52595 100755 --- a/lib/functions/validation.sh +++ b/lib/functions/validation.sh @@ -53,53 +53,39 @@ function GetValidationInfo() { eval "${VALIDATE_LANGUAGE}=${!VALIDATE_LANGUAGE,,}" done - ################################################ - # Determine if any linters were explicitly set # - ################################################ - ANY_SET="false" - ANY_TRUE="false" - ANY_FALSE="false" - # Loop through all languages - for LANGUAGE in "${LANGUAGE_ARRAY[@]}"; do - # build the variable - VALIDATE_LANGUAGE="VALIDATE_${LANGUAGE}" - # Check to see if the variable was set - if [ -n "${!VALIDATE_LANGUAGE}" ]; then - # It was set, need to set flag - ANY_SET="true" - if [ "${!VALIDATE_LANGUAGE}" == "true" ]; then - ANY_TRUE="true" - elif [ "${!VALIDATE_LANGUAGE}" == "false" ]; then - ANY_FALSE="true" + # Loop through language packs to turn on any linters in those packs. + for PACK in "${LANGUAGE_PACKS[@]}"; do + LANGUAGES="${PACK}[@]" + # Build the RUN_*_TOOLS variable + RUN_LANGUAGE_PACK="RUN_${PACK}" + + for LANGUAGE in "${!LANGUAGES}"; do + # Build the VALIDATE_* variable + VALIDATE_LANGUAGE="VALIDATE_${LANGUAGE}" + SHOULD_VALIDATE_LANGUAGE="SHOULD_${VALIDATE_LANGUAGE}" + if [[ -n "${!VALIDATE_LANGUAGE}" ]]; then + # If variable is explicitly set, that will override any language pack settings. + continue fi - fi + # Should run current linter if *any* language pack flips it to true. + if [[ "${!SHOULD_VALIDATE_LANGUAGE}" != "true" ]]; then + eval "${SHOULD_VALIDATE_LANGUAGE}='${!RUN_LANGUAGE_PACK}'" + fi + done done - if [ $ANY_TRUE == "true" ] && [ $ANY_FALSE == "true" ]; then - fatal "Behavior not supported, please either only include (VALIDATE=true) or exclude (VALIDATE=false) linters, but not both" - fi - - ######################################################### - # Validate if we should check/omit individual languages # - ######################################################### - # Loop through all languages + # Loop through all languages. for LANGUAGE in "${LANGUAGE_ARRAY[@]}"; do # build the variable VALIDATE_LANGUAGE="VALIDATE_${LANGUAGE}" - # Check if ANY_SET was set - if [[ ${ANY_SET} == "true" ]]; then - # Check to see if the variable was set - if [ -z "${!VALIDATE_LANGUAGE}" ]; then - # Flag was not set, default to: - # if ANY_TRUE then set to false - # if ANY_FALSE then set to true - eval "${VALIDATE_LANGUAGE}='$ANY_FALSE'" - fi + SHOULD_VALIDATE_LANGUAGE="SHOULD_${VALIDATE_LANGUAGE}" + if [ -n "${!VALIDATE_LANGUAGE}" ]; then + # If variable was explicitly set, honor that. + continue else - # No linter flags were set - default all to true - eval "${VALIDATE_LANGUAGE}='true'" + # Else use language pack settings to determine whether it should run. + eval "${VALIDATE_LANGUAGE}='${!SHOULD_VALIDATE_LANGUAGE:-false}'; export ${VALIDATE_LANGUAGE}" fi - eval "export ${VALIDATE_LANGUAGE}" done ####################################### diff --git a/lib/linter.sh b/lib/linter.sh index 1f739a785b7..2da79ab279f 100755 --- a/lib/linter.sh +++ b/lib/linter.sh @@ -203,6 +203,14 @@ else JAVASCRIPT_STYLE='standard' fi +######################### +# Language pack toggles # +######################### +# shellcheck disable=SC2034 # Variable is referenced indirectly +RUN_SECURITY_TOOLS="${RUN_SECURITY_TOOLS:-true}" +# shellcheck disable=SC2034 # Variable is referenced indirectly +RUN_CODE_QUALITY_TOOLS="${RUN_CODE_QUALITY_TOOLS:-false}" + ################## # Language array # ################## @@ -292,6 +300,17 @@ LINTER_NAMES_ARRAY['YAML']="yamllint" ############################################ LINTED_LANGUAGES_ARRAY=() # Will be filled at run time with all languages that were linted +################## +# Language packs # +################## +LANGUAGE_PACKS=('SECURITY_TOOLS' 'CODE_QUALITY_TOOLS') +# shellcheck disable=SC2034 # Variable is referenced indirectly +SECURITY_TOOLS=('CLOUDFORMATION_CFN_NAG' 'DOCKERFILE_HADOLINT' 'GITLEAKS' 'PYTHON_BANDIT' 'TERRAFORM_TERRASCAN') +# CODE_QUALITY_TOOLS are all tools (i.e., items in LANGUAGE_ARRAY) that are not in SECURITY_TOOLS +# shellcheck disable=SC2034 # Variable is referenced indirectly +mapfile -t CODE_QUALITY_TOOLS < <(printf '%s\n' "${LANGUAGE_ARRAY[@]}" "${SECURITY_TOOLS[@]}" | sort | uniq -u) +export LANGUAGE_PACKS + ################### # GitHub ENV Vars # ###################