Skip to content

Commit

Permalink
Merge pull request #111 from 23andMe/SEC-1493-linter-packs
Browse files Browse the repository at this point in the history 
SEC-1493 Added the ability to group linters via linter packs.
  • Loading branch information
@sarahc23
sarahc23 authored Jul 13, 2021
2 parents ab621f2 + c4ad61f commit a7a2a4f
Show file tree
Hide file tree
Showing 3 changed files with 67 additions and 41 deletions.
25 changes: 23 additions & 2 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -265,6 +265,9 @@ Example usage:

The super-linter allows you to pass the following `ENV` variables to be able to trigger different functionality.

<details>
<summary>The behavior in this section has changed with 23andMe's custom logic (see next section). Expand to see original.</summary>

_Note:_ All the `VALIDATE_[LANGUAGE]` variables behave in a very specific way:

- If none of them are passed, then they all default to true.
Expand All @@ -274,6 +277,7 @@ _Note:_ All the `VALIDATE_[LANGUAGE]` variables behave in a very specific way:

This means that if you run the linter "out of the box", all languages will be checked.
But if you wish to select or exclude specific linters, we give you full control to choose which linters are run, and won't run anything unexpected.
</details>

| **ENV VAR** | **Default Value** | **Notes** |
| ---------------------------------- | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
Expand Down Expand Up @@ -383,17 +387,34 @@ But if you wish to select or exclude specific linters, we give you full control
| **VALIDATE_YAML** | `true` | Flag to enable or disable the linting process of the YAML language. |
| **YAML_CONFIG_FILE** | `.yaml-lint.yml` | Filename for [Yamllint configuration](https://yamllint.readthedocs.io/en/stable/configuration.html) (ex: `.yaml-lint.yml`, `.yamllint.yml`) |

### 23andMe Custom Environment Variables
### 23andMe Custom Behavior and Environment Variables

23andMe's custom super-linter introduced the ability to group linters into language packs, which can then be toggled on or off as a group.
As a result, the `VALIDATE_[LANGUAGE]` variables behave differently from those in the base super-linter:

- **All variables default to `false`, unless explicitly toggled to true.**
- A single linter/language can appear in multiple linter packs. Its `VALIDATE_[LANGUAGE]` variable will be set to `true` if *any* of the language packs containing it are marked `true`, *except*...
- If there is an explicit `VALIDATE_[LANGUAGE]` variable for that specific linter/language, the variable will override all language pack settings.

| **ENV VAR** | **Default Value** | **Notes** |
| -------------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **CLOUDFORMATION_CFN_NAG_CONFIG_FILE** | `.cfnnag` | Filename for [cfn-nag configuration](https://github.com/stelligent/cfn_nag#profiles) |
| **GITLEAKS_CONFIG_FILE** | `none` | Filename for optional [gitleaks configuration](https://github.com/zricethezav/gitleaks#configuration) (ex: `simple_regex_config.toml`). Uses the [gitleaks default](https://github.com/zricethezav/gitleaks/blob/master/config/default.go) if unspecified |
| **OUTPUT_MODE** | `none` | Additional means for outputting findings, other than writing to stdout. Supported modes: lintly |
| **PYTHON_BANDIT_CONFIG_FILE** | `.bandit` | Filename for [bandit configuration](https://bandit.readthedocs.io/en/latest/config.html) (ex: `.bandit`, `tox.ini`) |
| **VALIDATE_CLOUDFORMATION_CFN_NAG** | `true` | Flag to enable or disable the security linting process of the AWS CloudFormation language. |
| **RUN_CODE_QUALITY_TOOLS** | `false ` | Flag to enable or disable the entire suite of code-quality-related linters. See below for a list of linters in this pack. |
| **RUN_SECURITY_TOOLS** | `true ` | Flag to enable or disable the entire suite of security tools. See below for a list of tools in this pack. |
| **VALIDATE_CLOUDFORMATION_CFN_NAG** | `true` | Flag to enable or disable the security linting process of the AWS CloudFormation language. |
| **VALIDATE_GITLEAKS** | `true` | Flag to enable or disable the linting process of potentially leaked secrets across all files and languages. |
| **VALIDATE_PYTHON_BANDIT** | `true` | Flag to enable or disable the linting process of the Python language. (Utilizing: bandit) |

The following is a list of supported language packs.

| **Language pack** | **Included Tools** |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| **RUN_SECURITY_TOOLS** | `CLOUDFORMATION_CFN_NAG`<br />`DOCKERFILE_HADOLINT`<br />`GITLEAKS`<br />`PYTHON_BANDIT`<br />`TERRAFORM_TERRASCAN` |
| **RUN_CODE_QUALITY_TOOLS** | _(All other tools that are currently not security tools)_ |

### Template rules files

You can use the **GitHub** **Super-Linter** _with_ or _without_ your own personal rules sets. This allows for greater flexibility for each individual code base. The Template rules all try to follow the standards we believe should be enabled at the basic level.
Expand Down
64 changes: 25 additions & 39 deletions lib/functions/validation.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -53,53 +53,39 @@ function GetValidationInfo() {
eval "${VALIDATE_LANGUAGE}=${!VALIDATE_LANGUAGE,,}"
done

################################################
# Determine if any linters were explicitly set #
################################################
ANY_SET="false"
ANY_TRUE="false"
ANY_FALSE="false"
# Loop through all languages
for LANGUAGE in "${LANGUAGE_ARRAY[@]}"; do
# build the variable
VALIDATE_LANGUAGE="VALIDATE_${LANGUAGE}"
# Check to see if the variable was set
if [ -n "${!VALIDATE_LANGUAGE}" ]; then
# It was set, need to set flag
ANY_SET="true"
if [ "${!VALIDATE_LANGUAGE}" == "true" ]; then
ANY_TRUE="true"
elif [ "${!VALIDATE_LANGUAGE}" == "false" ]; then
ANY_FALSE="true"
# Loop through language packs to turn on any linters in those packs.
for PACK in "${LANGUAGE_PACKS[@]}"; do
LANGUAGES="${PACK}[@]"
# Build the RUN_*_TOOLS variable
RUN_LANGUAGE_PACK="RUN_${PACK}"

for LANGUAGE in "${!LANGUAGES}"; do
# Build the VALIDATE_* variable
VALIDATE_LANGUAGE="VALIDATE_${LANGUAGE}"
SHOULD_VALIDATE_LANGUAGE="SHOULD_${VALIDATE_LANGUAGE}"
if [[ -n "${!VALIDATE_LANGUAGE}" ]]; then
# If variable is explicitly set, that will override any language pack settings.
continue
fi
fi
# Should run current linter if *any* language pack flips it to true.
if [[ "${!SHOULD_VALIDATE_LANGUAGE}" != "true" ]]; then
eval "${SHOULD_VALIDATE_LANGUAGE}='${!RUN_LANGUAGE_PACK}'"
fi
done
done

if [ $ANY_TRUE == "true" ] && [ $ANY_FALSE == "true" ]; then
fatal "Behavior not supported, please either only include (VALIDATE=true) or exclude (VALIDATE=false) linters, but not both"
fi

#########################################################
# Validate if we should check/omit individual languages #
#########################################################
# Loop through all languages
# Loop through all languages.
for LANGUAGE in "${LANGUAGE_ARRAY[@]}"; do
# build the variable
VALIDATE_LANGUAGE="VALIDATE_${LANGUAGE}"
# Check if ANY_SET was set
if [[ ${ANY_SET} == "true" ]]; then
# Check to see if the variable was set
if [ -z "${!VALIDATE_LANGUAGE}" ]; then
# Flag was not set, default to:
# if ANY_TRUE then set to false
# if ANY_FALSE then set to true
eval "${VALIDATE_LANGUAGE}='$ANY_FALSE'"
fi
SHOULD_VALIDATE_LANGUAGE="SHOULD_${VALIDATE_LANGUAGE}"
if [ -n "${!VALIDATE_LANGUAGE}" ]; then
# If variable was explicitly set, honor that.
continue
else
# No linter flags were set - default all to true
eval "${VALIDATE_LANGUAGE}='true'"
# Else use language pack settings to determine whether it should run.
eval "${VALIDATE_LANGUAGE}='${!SHOULD_VALIDATE_LANGUAGE:-false}'; export ${VALIDATE_LANGUAGE}"
fi
eval "export ${VALIDATE_LANGUAGE}"
done

#######################################
Expand Down
19 changes: 19 additions & 0 deletions lib/linter.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -203,6 +203,14 @@ else
JAVASCRIPT_STYLE='standard'
fi

#########################
# Language pack toggles #
#########################
# shellcheck disable=SC2034 # Variable is referenced indirectly
RUN_SECURITY_TOOLS="${RUN_SECURITY_TOOLS:-true}"
# shellcheck disable=SC2034 # Variable is referenced indirectly
RUN_CODE_QUALITY_TOOLS="${RUN_CODE_QUALITY_TOOLS:-false}"

##################
# Language array #
##################
Expand Down Expand Up @@ -292,6 +300,17 @@ LINTER_NAMES_ARRAY['YAML']="yamllint"
############################################
LINTED_LANGUAGES_ARRAY=() # Will be filled at run time with all languages that were linted

##################
# Language packs #
##################
LANGUAGE_PACKS=('SECURITY_TOOLS' 'CODE_QUALITY_TOOLS')
# shellcheck disable=SC2034 # Variable is referenced indirectly
SECURITY_TOOLS=('CLOUDFORMATION_CFN_NAG' 'DOCKERFILE_HADOLINT' 'GITLEAKS' 'PYTHON_BANDIT' 'TERRAFORM_TERRASCAN')
# CODE_QUALITY_TOOLS are all tools (i.e., items in LANGUAGE_ARRAY) that are not in SECURITY_TOOLS
# shellcheck disable=SC2034 # Variable is referenced indirectly
mapfile -t CODE_QUALITY_TOOLS < <(printf '%s\n' "${LANGUAGE_ARRAY[@]}" "${SECURITY_TOOLS[@]}" | sort | uniq -u)
export LANGUAGE_PACKS

###################
# GitHub ENV Vars #
###################
Expand Down

0 comments on commit a7a2a4f

Please sign in to comment.