Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nancy uses Nexus IQ Server, the journey #68

Merged
merged 59 commits into from
Feb 27, 2020
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
882af3e
WIP for Nexus IQ Server use
DarthHater Jan 2, 2020
1126c52
Feedback from @fitzoh and use the config URL
DarthHater Jan 2, 2020
540319d
Server URL
DarthHater Jan 2, 2020
3803263
Fall into the gap
DarthHater Jan 2, 2020
f0d0a00
Various and Sundry
DarthHater Jan 2, 2020
ea49c69
Mas various and sundry
DarthHater Jan 2, 2020
a6df6e0
Newlines for EMPHASIS
DarthHater Jan 2, 2020
3613341
README
DarthHater Jan 2, 2020
f564124
MORE SPACE
DarthHater Jan 2, 2020
57a2fd8
FromString because much nicer
DarthHater Jan 2, 2020
6839696
Feedback from @zendern
DarthHater Jan 3, 2020
772d557
More from @zendern
DarthHater Jan 3, 2020
e46bd59
Mas from @zendern
DarthHater Jan 3, 2020
93eceaf
Check with customerrors.Check and exit
DarthHater Jan 3, 2020
8c81fa8
go mod tidy
DarthHater Jan 3, 2020
4adf87f
Move some types over, comments
DarthHater Jan 4, 2020
4de41cc
Fix tests and make @zendern happy
DarthHater Jan 4, 2020
b1e3bcf
tests pass
DarthHater Jan 4, 2020
706eedc
IQ Config type
DarthHater Jan 4, 2020
123b17f
Ok hybridize Nancy with OSS Index and IQ, and start writing some test…
DarthHater Feb 6, 2020
2bce2bf
TESTED
DarthHater Feb 6, 2020
0cbe2aa
remove this
DarthHater Feb 6, 2020
2f28bf5
Happy path cyclonedx test
DarthHater Feb 6, 2020
74d58e4
Add in a vuln for parsing
DarthHater Feb 6, 2020
976f992
Config
DarthHater Feb 6, 2020
b68c112
For the honor
DarthHater Feb 6, 2020
0bcc6ff
Wipe cache
DarthHater Feb 6, 2020
05671b3
Clever
DarthHater Feb 6, 2020
fbe2d53
Clean up and FUN
DarthHater Feb 6, 2020
ccfac65
Ok try this
DarthHater Feb 6, 2020
4a87b82
Add to IQ block
DarthHater Feb 6, 2020
e7094a2
Try and defer db close
DarthHater Feb 6, 2020
b5ab28b
Errrm
DarthHater Feb 6, 2020
e120525
Hmmm
DarthHater Feb 7, 2020
1a5b80e
Kill test DB dir
DarthHater Feb 7, 2020
ecd0ee6
Wierdness warshed over the dude
DarthHater Feb 7, 2020
323fde6
Something is happening and it is weird so limit to one proc for tests
DarthHater Feb 7, 2020
89de1e1
Back to this
DarthHater Feb 7, 2020
988416c
Test that XML
DarthHater Feb 7, 2020
ce572ea
After error check
DarthHater Feb 7, 2020
2e0f0fc
Bye bye for now
DarthHater Feb 7, 2020
180312c
Config for max retries
DarthHater Feb 7, 2020
ca93f03
Drop verbose flag
DarthHater Feb 7, 2020
3e0da2e
Test case with server unreachable, add panic
DarthHater Feb 7, 2020
1bb2446
Couple more test cases
DarthHater Feb 7, 2020
1889f40
Remove dead code, use customerrors
DarthHater Feb 7, 2020
1dca5e9
Test ParseIQ
DarthHater Feb 7, 2020
44025fb
Tiny clean ups
DarthHater Feb 7, 2020
7dda206
Listen to go
DarthHater Feb 7, 2020
84ecaf9
Okie dokie, move back to errs and exits for now
DarthHater Feb 20, 2020
cf478b5
Merge master
DarthHater Feb 21, 2020
2b58404
Merge master
DarthHater Feb 27, 2020
b977b5a
An audit for YOU
DarthHater Feb 27, 2020
431b1a9
Test with single proc
DarthHater Feb 27, 2020
55c529c
Spit out failure in CI
DarthHater Feb 27, 2020
aa1fe64
Why not both
DarthHater Feb 27, 2020
34d5e5b
Try this on for size
DarthHater Feb 27, 2020
07ab7df
Just this one, for now
DarthHater Feb 27, 2020
2461c57
Kill this for now
DarthHater Feb 27, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 46 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,22 +7,24 @@

# Nancy

`nancy` is a tool to check for vulnerabilities in your Golang dependencies, powered by [Sonatype OSS Index](https://ossindex.sonatype.org/).
`nancy` is a tool to check for vulnerabilities in your Golang dependencies, powered by [Sonatype OSS Index](https://ossindex.sonatype.org/), and as well, works with Nexus IQ Server, allowing you a smooth experience as a Golang developer, using the best tools in the market!

### Usage

```
~ > nancy
Usage:
go list -m all | nancy [options]
nancy [options] </path/to/Gopkg.lock>
nancy [options] </path/to/go.sum>
nancy [options] </path/to/Gopkg.lock>
DarthHater marked this conversation as resolved.
Show resolved Hide resolved
nancy [options] </path/to/go.sum>
nancy iq [options]

Options:
-exclude-vulnerability value
Comma separated list of CVEs to exclude
-exclude-vulnerability-file string
Path to a file containing newline separated CVEs to be excluded (default "./.nancy-ignore")
-help
provides help text on how to use nancy
-no-color
indicate output should not be colorized
-noColor
Expand All @@ -31,11 +33,23 @@ Options:
indicate output should contain only packages with vulnerabilities
-version
prints current nancy version

IQ Options:
-application string
Specify application ID for request
-server-url string
Specify Nexus IQ Server URL/port (default "http://localhost:8070")
-stage string
Specify stage for application (default "build")
-token string
Specify token/password for request (default "admin123")
-user string
Specify username for request (default "admin")
```

`nancy` currently works for projects that use `dep` or `go mod` for dependencies.

### Options
### OSS Index Options

#### Quiet mode

Expand Down Expand Up @@ -73,6 +87,33 @@ CVN-123 # Mitigated the risk of this since we only use one method in this packag
CVN-543
```

### Nexus IQ Server Options

By default, assuming you have an out of the box Nexus IQ Server running, you can run `nancy` like so:

`go list -m all | ./nancy iq -application public-application-id`

It is STRONGLY suggested that you do not do this, and we will warn you on output if you are.

A more logical use of `nancy` against Nexus IQ Server will look like so:

`go list -m all | ./nancy iq -application public-application-id -user nondefaultuser -token yourtoken -server-url http://adifferentserverurl:port -stage develop`

Options for stage are as follows:

`build, develop, stage-release, release`

By default `-stage` will be `develop`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why would the default be develop?? Im going to guess 99% of the time this will be running on CI so shouldn't it be build. Keep in mind i have no real idea what this stages mean in IQ (only briefly read some of the docs :) ) so I'll defer to you of course.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have build > develop > release-stage > release. I went with develop as a default as Nancy can be used by developers locally pretty easy. We could set it to build for sure, but it kinda depends on how you setup your policy TBH with Nexus IQ. I originally had it as build ftr. I can go either way!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool cool.... that makes sense then.

#scopeCreepAndMaybeNotWorthIt
But would it make sense to detect if on CI (looking for TRAVIS, CIRCLECI, CI, etc etc) environment variables and if one of those is set you change the stage automagically to build/release-stage?? Maybe that is still back into the "how you setup your policy in Nexus IQ" again and it doesn't matter. ¯_(ツ)_/¯

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I LIKE THAT. We'd still need to allow someone to override it, though, because people use Travis, CircleCI etc... to do CD too.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah for sure...if you pass it in then that is the value to be used. But nancy will attempt to "smart default" it for you.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nancy is so clever!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should create a ticket for this, it was a fantastic suggestion, however I dunno if I want to do it in this PR.


Successful submissions to Nexus IQ Server will result in either an OS exit of 0, meaning all is clear and a response akin to:

```
Wonderbar! No policy violations reported for this audit!
Report URL: http://reportURL
```

Failed submissions will either indicate failure because of an issue with processing the request, or a policy violation. Both will exit with a code of 1, allowing you to fail your build in CI. Policy Violation failures will include a report URL where you can learn more about why you encountered a failure.

DarthHater marked this conversation as resolved.
Show resolved Hide resolved
### Usage in CI

You can see an example of using `nancy` in Travis-CI at [this intentionally vulnerable repo we made](https://github.com/sonatype-nexus-community/intentionally-vulnerable-golang-project).
Expand Down
62 changes: 49 additions & 13 deletions configuration/parse.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,20 +4,27 @@ import (
"bufio"
"flag"
"fmt"
"github.com/sonatype-nexus-community/nancy/types"
"os"
"regexp"
"strings"

"github.com/sonatype-nexus-community/nancy/types"
)

type Configuration struct {
UseStdIn bool
Help bool
NoColor bool
Quiet bool
Version bool
CveList types.CveListFlag
Path string
UseStdIn bool
Help bool
NoColor bool
Quiet bool
Version bool
CveList types.CveListFlag
IQ bool
Path string
User string
Token string
Stage string
Application string
Server string
DarthHater marked this conversation as resolved.
Show resolved Hide resolved
}

var unixComments = regexp.MustCompile(`#.*$`)
Expand All @@ -35,24 +42,53 @@ func Parse(args []string) (Configuration, error) {
flag.Var(&config.CveList, "exclude-vulnerability", "Comma separated list of CVEs to exclude")
flag.StringVar(&excludeVulnerabilityFilePath, "exclude-vulnerability-file", "./.nancy-ignore", "Path to a file containing newline separated CVEs to be excluded")

iqCommand := flag.NewFlagSet("iq", flag.ExitOnError)
iqCommand.StringVar(&config.User, "user", "admin", "Specify username for request")
iqCommand.StringVar(&config.Token, "token", "admin123", "Specify token/password for request")
DarthHater marked this conversation as resolved.
Show resolved Hide resolved
iqCommand.StringVar(&config.Server, "server-url", "http://localhost:8070", "Specify Nexus IQ Server URL/port")
iqCommand.StringVar(&config.Application, "application", "", "Specify application ID for request")
iqCommand.StringVar(&config.Stage, "stage", "develop", "Specify stage for application")

flag.Usage = func() {
_, _ = fmt.Fprintf(os.Stderr, "Usage: \nnancy [options] </path/to/Gopkg.lock>\nnancy [options] </path/to/go.sum>\n\nOptions:\n")
_, _ = fmt.Fprintf(os.Stderr, `Usage:
nancy [options] </path/to/Gopkg.lock>
nancy [options] </path/to/go.sum>
nancy iq [options]
DarthHater marked this conversation as resolved.
Show resolved Hide resolved

Options:
`)
flag.PrintDefaults()
_, _ = fmt.Fprintf(os.Stderr, `
IQ Options:
`)
iqCommand.PrintDefaults()
os.Exit(2)
}

// Parse config from the command line output
err := flag.CommandLine.Parse(args)
if err != nil {
return config, err
if len(os.Args) > 1 {
if os.Args[1] == "iq" {
err := iqCommand.Parse(os.Args[2:])
if err != nil {
return config, err
}
config.IQ = true
config.UseStdIn = true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this might answer some of my other questions on why you did it the way you did....you are forcing the users to only use the | option by doing this. Personally i feel like making the api consistent so that if im using OSS or IQ server it should work the same. Whether by pipe or passing in the file to be scanned.

Really defining IQ is just saying i want you to use a different underlying audit infrastructure. Nancy is just a facilitator and shouldn't behave differently b/c of that switch out.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I want to make it work for dep but not parsing go.sum (since go list -m all is the preferred way to do this, more authoritative)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's cool. Im going to open an issue to remove support for go.sum if we want to move away. No sense in supporting both.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#69

return config, nil
}
}

if len(flag.Args()) == 0 {
config.UseStdIn = true
}else{
} else {
config.Path = args[len(args)-1]
}

err := flag.CommandLine.Parse(args)
if err != nil {
return config, err
}

if noColorDeprecated == true {
fmt.Println("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!")
fmt.Println("!!!! DEPRECATION WARNING : Please change 'noColor' param to be 'no-color'. This one will be removed in a future release. !!!!")
Expand Down
71 changes: 71 additions & 0 deletions cyclonedx/cyclonedx.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
// Copyright 2020 Sonatype Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

// Package cyclonedx has definitions and functions for processing golang purls into a minimal CycloneDX 1.1 Sbom
package cyclonedx

import (
"encoding/xml"
"fmt"

"github.com/package-url/packageurl-go"
)

type sbom struct {
XMLName xml.Name `xml:"bom"`
Xmlns string `xml:"xmlns,attr"`
Version string `xml:"version,attr"`
Components components `xml:"components"`
}

type components struct {
Component []component `xml:"component"`
}

type component struct {
Type string `xml:"type,attr"`
BomRef string `xml:"bom-ref,attr"`
Name string `xml:"name"`
Version string `xml:"version"`
Purl string `xml:"purl"`
DarthHater marked this conversation as resolved.
Show resolved Hide resolved
}

const cycloneDXBomXmlns1_1 = "http://cyclonedx.org/schema/bom/1.1"

const version = "1"

// ProcessPurlsIntoSBOM will take a slice of packageurl.PackageURL and convert them
// into a minimal 1.1 CycloneDX sbom
func ProcessPurlsIntoSBOM(purls []packageurl.PackageURL) string {
return processPurlsIntoSBOMSchema1_1(purls)
}

func processPurlsIntoSBOMSchema1_1(purls []packageurl.PackageURL) string {
sbom := sbom{}
sbom.Xmlns = cycloneDXBomXmlns1_1
sbom.Version = version
for _, v := range purls {
component := component{Type: "library", BomRef: v.String(), Purl: v.String(), Name: v.Name, Version: v.Version}
sbom.Components.Component = append(sbom.Components.Component, component)
}

output, err := xml.MarshalIndent(sbom, " ", " ")
if err != nil {
fmt.Print(err)
}

output = []byte(xml.Header + string(output))

return string(output)
}
2 changes: 2 additions & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ require (
github.com/Masterminds/vcs v1.13.1 // indirect
github.com/armon/go-radix v1.0.0 // indirect
github.com/boltdb/bolt v1.3.1 // indirect
github.com/coreos/etcd v3.3.18+incompatible // indirect
DarthHater marked this conversation as resolved.
Show resolved Hide resolved
github.com/dgraph-io/badger v1.5.5-0.20181004181505-439fd464b155
github.com/dgryski/go-farm v0.0.0-20180109070241-2de33835d102 // indirect
github.com/dustin/go-humanize v1.0.0 // indirect
Expand All @@ -16,6 +17,7 @@ require (
github.com/jmank88/nuts v0.3.0 // indirect
github.com/logrusorgru/aurora v0.0.0-20190803045625-94edacc10f9b
github.com/nightlyone/lockfile v0.0.0-20180618180623-0ad87eef1443 // indirect
github.com/package-url/packageurl-go v0.1.0
github.com/pelletier/go-toml v1.4.0 // indirect
github.com/pkg/errors v0.8.0 // indirect
github.com/sdboyer/constext v0.0.0-20170321163424-836a14457353 // indirect
Expand Down
4 changes: 4 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ github.com/armon/go-radix v1.0.0 h1:F4z6KzEeeQIMeLFa97iZU6vupzoecKdU5TX24SNppXI=
github.com/armon/go-radix v1.0.0/go.mod h1:ufUuZ+zHj4x4TnLV4JWEpy2hxWSpsRywHrMgIH9cCH8=
github.com/boltdb/bolt v1.3.1 h1:JQmyP4ZBrce+ZQu0dY660FMfatumYDLun9hBCUVIkF4=
github.com/boltdb/bolt v1.3.1/go.mod h1:clJnj/oiGkjum5o1McbSZDSLxVThjynRyGBgiAx27Ps=
github.com/coreos/etcd v3.3.18+incompatible h1:Zz1aXgDrFFi1nadh58tA9ktt06cmPTwNNP3dXwIq1lE=
github.com/coreos/etcd v3.3.18+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
Expand All @@ -34,6 +36,8 @@ github.com/logrusorgru/aurora v0.0.0-20190803045625-94edacc10f9b h1:PMbSa9CgaiQR
github.com/logrusorgru/aurora v0.0.0-20190803045625-94edacc10f9b/go.mod h1:7rIyQOR62GCctdiQpZ/zOJlFyk6y+94wXzv6RNZgaR4=
github.com/nightlyone/lockfile v0.0.0-20180618180623-0ad87eef1443 h1:+2OJrU8cmOstEoh0uQvYemRGVH1O6xtO2oANUWHFnP0=
github.com/nightlyone/lockfile v0.0.0-20180618180623-0ad87eef1443/go.mod h1:JbxfV1Iifij2yhRjXai0oFrbpxszXHRx1E5RuM26o4Y=
github.com/package-url/packageurl-go v0.1.0 h1:efWBc98O/dBZRg1pw2xiDzovnlMjCa9NPnfaiBduh8I=
github.com/package-url/packageurl-go v0.1.0/go.mod h1:C/ApiuWpmbpni4DIOECf6WCjFUZV7O1Fx7VAzrZHgBw=
github.com/pelletier/go-toml v1.4.0 h1:u3Z1r+oOXJIkxqw34zVhyPgjBsm6X2wn21NWs/HfSeg=
github.com/pelletier/go-toml v1.4.0/go.mod h1:PN7xzY2wHTK0K9p34ErDQMlFxa51Fk0OUruD3k1mMwo=
github.com/pkg/errors v0.8.0 h1:WdK/asTD0HN+q6hsWO3/vpuAkAr+tw6aNJNDFFf0+qw=
Expand Down
Loading