Implement Anchore pass/fail in CI/CD

[lewisdaly]

Goal:

As an "OSS Developer", I want to "define a security policy for pass/fails on Anchore-cli docker image scans" so I can "address container security concerns as part of the CI/CD process"

In the last PI, we added anchore image scanning to most of the containers' CI/CD process. This scanning produces a vulnerability file, which lists a number of potential security issues with the given container. We currently ignore this file in the CI/CD process.

In this task, we need to understand that vulnerability file, and develop a security policy for setting a PASS/FAIL action in the CI/CD process so we can start to reduce the number of potential vulnerabilities.

Tasks:

• analyze existing vulnerability files (these can be found in CircleCI artifacts for release builds)
• Implement a PASS/FAIL gate on a given container, say ml-api-adapter, and resolve potential issues.
• Apply for the other containers:
  • account-lookup-service - PR
  • bulk-api-adapter - PR
  • central-event-processor - PR
  • central-ledger - PR
  • central-settlement - PR
  • email-notifier - PR
  • ml-api-adapter - PR
  • quoting-service - PR
• Communicate these changes with the rest of the OSS team somewhere in mojaloop/documentation and link in the readme.md files of the projects.
• maybe: Should we set up a Slack notification to say that a release has failed because of the image scans?

Acceptance Criteria:

• Designs are up-to date
• Unit Tests pass
• Integration Tests pass
• Code Style & Coverage meets standards
• Changes made to config (default.json) are broadcast to team and follow-up tasks added to update helm charts and other deployment config.
• TBD

Pull Requests:

• Feature/1223 Add custom mojaloop policy for evaluating anchore-cli scans quoting-service#192
• Feature/1223 anchore policy email-notifier#150
• Add custom mojaloop policy for evaluating anchore-cli scans ml-api-adapter#397
• Feature/1223 anchore policy central-settlement#292
• Feature/1223 anchore policy central-ledger#672
• Add custom mojaloop policy for evaluating anchore-cli scans central-event-processor#174
• feature/1223-anchore-policy bulk-api-adapter#24
• Add custom mojaloop policy for evaluating anchore-cli scans account-lookup-service#300

Follow-up:

• Security Vulnerabilities in central-settlement + central-ledger #1292

Dependencies:

• N/A

Accountability:

• Owner: @lewisdaly
• QA/Review: TBC

[lewisdaly]

@godfreykutumela, I'm happy to own this, and work on the development side, but I need someone with more experience in the security side to help with analyzing the vulnerability files and set the baseline for passing vs failing.

[lewisdaly]

@pedrosousabarreto we spoke about turning this story into a more generic "define baseline requirements for code security", but I think I would like to keep this specifically about Anchore image scanning and our CI/CD process.

Perhaps we can open another ticket about setting goals in Mojaloop around application security (as we talked about), or roll it into #1222, which talks about a tool summary.

[godfreykutumela]

How are doing on this one @lewisdaly

[lewisdaly]

I've started by comparing the anchore vuln.json files against the base docker image of node:12.16.0-alpine

[lewisdaly]

I think we're going to spend a little more time evaluating the business goals on this one and how it can fit into our developer workflow with minimal overhead.

[lewisdaly]

From what we've discussed in meeting and on Slack, I'm going to go ahead with the following:

Base Image Scanning

• daily cron job to scan the node:12.16.0-alpine base image (we can always add other images in addition if we need)
• notifications on the slack channel daily if a new vulnerability is found. Something like: 'Anchore-cli has found a new vulnerability...'

Scan every release

• notify on success/failure (we currently only notify on success)
• configure the strictness of how we evaluate the vuln.json files via env var or remotely
• worry about whitelisting certain vulnerabilities later on
• figure out how to diff: inform developers that their build failed, and include information on if the vulnerabilities were also present in the base image

[godfreykutumela]

Yes that was basically the agreement in principle @lewisdaly please go ahead with the implementation. I have put more issues into this sprint so please review and let me know if feasible to archive them all.

[lewisdaly]

I've started working on a global anchore policy, based on the docker CIS 1.14 guidelines.

I'm also working on a tool which will 'diff' anchore reports, so we can determine if an issue has been added to a docker image that isn't present in the base image. You can see these works in progress here:

https://github.com/vessels-tech/anchore

[lewisdaly]

The Anchore CircleCI orb fails to load our custom policies. I'm trying to debug it, but it looks like an issue with Anchore itself. If I don't make any progress, I'll follow up with the anchore devs.

Noted @lewisdaly thanks for the update

[lewisdaly]

It appears that I'm not the only one with this issue: anchore/ci-tools#27

It could be that the error message is a red herring, and the actual policy file is invalid. This wouldn't be a crazy situation because there's not much documentation about creating policy files.

[lewisdaly]

It's working: https://app.circleci.com/pipelines/github/mojaloop/quoting-service/700/workflows/338409e3-f679-4bce-b75c-84c15f420c46

[lewisdaly]

I'll work on opening some the proper PRs across the repos over the next few days.

[godfreykutumela]

Great news and well done for the persistence on this one @lewisdaly

[lewisdaly]

All builds passing, so ready for reviews now.

[rasputtintin]

This is awesome news.

…

On Tue, Apr 14, 2020 at 5:19 AM Lewis Daly ***@***.***> wrote: All builds passing, so ready for reviews now. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#1223 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMQFXH35MENSD3ZCEWTW6Z3RMPB2TANCNFSM4KTW6KAA> .

-- Know thy enemy and know thyself, and you shall survive the outcome of a thousand battles.