Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test(e2e): add E2E testing utilities and sample specs #639

Merged
merged 126 commits into from
Nov 9, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
126 commits
Select commit Hold shift + click to select a range
7844882
test(e2e): add ginkgo and start distribution
qweeah Sep 13, 2022
f76783b
add a spec
qweeah Sep 13, 2022
978219a
rename package
qweeah Sep 14, 2022
72dd79d
add _test suffix
qweeah Sep 14, 2022
dccb967
correct package name
qweeah Sep 14, 2022
dc2c8d6
clean module file
qweeah Sep 14, 2022
8876a01
ignore certain dep
qweeah Sep 15, 2022
1b86ea3
Merge remote-tracking branch 'origin_src/main' into e2e-infra
qweeah Sep 15, 2022
e9aa7ad
Merge remote-tracking branch 'origin_src/main' into e2e-infra
qweeah Sep 26, 2022
b3c5206
rename util package
qweeah Sep 28, 2022
8fe7427
Merge remote-tracking branch 'origin_src/main' into e2e-infra
qweeah Sep 28, 2022
66661ab
remove test delay
qweeah Sep 28, 2022
88d2623
temp: add utility code in
qweeah Sep 28, 2022
463cc60
refactor with exec option
qweeah Sep 28, 2022
4cd813f
fix write error
qweeah Sep 28, 2022
bbf3fc0
fix scenario tests
qweeah Sep 29, 2022
3f32cd1
add mount files for test initialization
qweeah Sep 29, 2022
c2dd82a
revert mount file
qweeah Sep 29, 2022
14c0e64
add mount tarball
qweeah Sep 29, 2022
d6007c0
add document for mounting data
qweeah Sep 29, 2022
e3a0305
omit decompress logs
qweeah Sep 29, 2022
41e9e26
Merge branch 'e2e-infra' into e2e-utility
qweeah Sep 29, 2022
8ab1b59
fix git ignore
qweeah Sep 29, 2022
87d0d53
Merge branch 'e2e-infra' into e2e-utility
qweeah Sep 29, 2022
635215b
use registry auth config for logout testing
qweeah Sep 29, 2022
0a0ef4a
fix: use cred config file when logout
qweeah Sep 29, 2022
83f02a0
Merge branch 'main' of https://github.com/oras-project/oras into main
qweeah Sep 29, 2022
0c2bd61
Merge branch 'fix-logout' into e2e-utility
qweeah Sep 29, 2022
479ca54
Merge remote-tracking branch 'origin_src/main' into e2e-infra
qweeah Sep 29, 2022
e6b1ba6
temp: refactoring state machine
qweeah Sep 30, 2022
bed64a9
done refactoring state machine
qweeah Sep 30, 2022
cf8ff9e
code clean
qweeah Sep 30, 2022
3c14379
more code clean
qweeah Sep 30, 2022
4806422
refactor with gbytes
qweeah Sep 30, 2022
c269f12
bug: cannot detect err out
qweeah Sep 30, 2022
c2fdbd8
Merge remote-tracking branch 'origin_src/main' into e2e-utility
qweeah Oct 6, 2022
74ccab3
Merge branch 'e2e-utility' of https://github.com/qweeah/oras into e2e…
qweeah Oct 6, 2022
d242da2
delay matching until execution is done
qweeah Oct 7, 2022
06fe51f
rephrase auth failure description
qweeah Oct 7, 2022
0992729
remove example test
qweeah Oct 8, 2022
c27309b
add submodule
qweeah Oct 8, 2022
0f85baa
Merge remote-tracking branch 'origin_src/main' into e2e-infra
qweeah Oct 8, 2022
cb93b2e
add cli module in
qweeah Oct 8, 2022
2c24798
try build without module mode
qweeah Oct 8, 2022
7fa6838
skip checking license for go.work
qweeah Oct 8, 2022
248c4e5
update registry mount file
qweeah Oct 8, 2022
6b0848d
Merge branch 'e2e-infra' of https://github.com/qweeah/oras into e2e-i…
qweeah Oct 8, 2022
8170b50
fix go modules
qweeah Oct 8, 2022
37651db
remove go workspace config
qweeah Oct 8, 2022
a3e595f
modify root folder for e2e
qweeah Oct 8, 2022
747f8fd
force module mode while installing ginkgo
qweeah Oct 8, 2022
f991754
use abs path for docker mount
qweeah Oct 8, 2022
1aab431
update readme
qweeah Oct 8, 2022
c3a92f7
replace distribution configure with docker envs
qweeah Oct 8, 2022
2324232
add newline
qweeah Oct 8, 2022
b069bbd
revert go mod changes
qweeah Oct 8, 2022
9cbd970
use gzip for mount file
qweeah Oct 8, 2022
34ddf67
remove unnecessary steps
qweeah Oct 8, 2022
4af75dc
doc clean
qweeah Oct 8, 2022
6087fb2
add doc for global variables
qweeah Oct 8, 2022
09068f5
code clean
qweeah Oct 8, 2022
f2c62ab
code clean
qweeah Oct 8, 2022
6ac4417
Merge remote-tracking branch 'origin_src/main' into e2e-infra
qweeah Oct 8, 2022
1eab9e6
Merge branch 'e2e-infra' into e2e-utility
qweeah Oct 8, 2022
95aed09
fix error
qweeah Oct 8, 2022
01def2a
Merge branch 'e2e-utility' of https://github.com/qweeah/oras into e2e…
qweeah Oct 8, 2022
6757c23
Merge branch 'e2e-utility' of https://github.com/qweeah/oras into e2e…
qweeah Oct 10, 2022
ce2b308
Merge remote-tracking branch 'origin_src/main' into e2e-utility
qweeah Oct 10, 2022
6bc755b
remove obsolete files
qweeah Oct 10, 2022
737f34f
add timeout and password
qweeah Oct 10, 2022
db69f49
longer timeout for login command test
qweeah Oct 10, 2022
7984b40
Merge remote-tracking branch 'origin_src/main' into e2e-utility
qweeah Oct 10, 2022
1be77aa
correct config
qweeah Oct 10, 2022
b29535f
replace it with by
qweeah Oct 10, 2022
1e26b37
add test ci
qweeah Oct 10, 2022
cb3e9fa
change content to string
qweeah Oct 11, 2022
4745024
fix content matching failure
qweeah Oct 11, 2022
9faf62f
try succinct mode
qweeah Oct 11, 2022
cbc8890
correct ginkgo param
qweeah Oct 11, 2022
30ae078
refactor execution options
qweeah Oct 11, 2022
3fc5876
refactor version command test
qweeah Oct 11, 2022
315708c
add document
qweeah Oct 11, 2022
da1d5fb
revert build yaml file
qweeah Oct 11, 2022
141a491
return execution result
qweeah Oct 11, 2022
81a8c1d
change work directory type
qweeah Oct 11, 2022
cdffa27
fix compilation error
qweeah Oct 11, 2022
d979fba
fix bug for working directory
qweeah Oct 11, 2022
cf84ed1
rephrase version test text
qweeah Oct 11, 2022
a753585
doc clean
qweeah Oct 11, 2022
4769e8d
Merge remote-tracking branch 'origin_src/main' into e2e-utility
qweeah Oct 17, 2022
176411e
pull always
qweeah Oct 19, 2022
2dc6684
Merge remote-tracking branch 'origin_src/main' into e2e-utility
qweeah Oct 19, 2022
705ecb4
move auth tests to a separate suite
qweeah Oct 21, 2022
e5560eb
Merge branch 'main' into e2e-utility
qweeah Oct 21, 2022
fab5699
Merge branch 'main' into e2e-utility
qweeah Oct 21, 2022
9fe2b3d
add test for manifest fetch
qweeah Oct 24, 2022
ffd8c3f
run all tests
qweeah Oct 24, 2022
6f26be5
run before all
qweeah Oct 24, 2022
177725b
auth with once flag
qweeah Oct 24, 2022
6e9e971
fix failed cases
qweeah Oct 24, 2022
7365e58
add lock for auth
qweeah Oct 24, 2022
35be2dc
add media type assertion
qweeah Oct 24, 2022
432da50
add doc
qweeah Oct 24, 2022
6acbe3f
add bad path
qweeah Oct 24, 2022
f439e1b
add manifest fetch to file and desc to stdout
qweeah Oct 24, 2022
a0d7559
more strict logout order
qweeah Oct 26, 2022
6f650cc
update test file structure
qweeah Nov 3, 2022
1a4cb3a
fix init copying
qweeah Nov 3, 2022
a2dc628
Merge remote-tracking branch 'origin_src/main' into e2e-utility
qweeah Nov 3, 2022
6c30fd3
create dir only
qweeah Nov 3, 2022
2255a02
run auth before suite
qweeah Nov 3, 2022
47ee168
check readme in
qweeah Nov 7, 2022
eba0431
Merge remote-tracking branch 'origin_src/main' into e2e-utility
qweeah Nov 8, 2022
fcc3828
add auth before testing
qweeah Nov 8, 2022
a2f9ff5
logout before login
qweeah Nov 8, 2022
e02cc86
Merge remote-tracking branch 'origin_src/main' into e2e-utility
qweeah Nov 8, 2022
98f5e03
add doc & remove unused file
qweeah Nov 8, 2022
9ba0a17
resolve comments
qweeah Nov 8, 2022
456a3ba
try string type
qweeah Nov 9, 2022
eea1fb3
rename matcher
qweeah Nov 9, 2022
ed9af5b
add newline
qweeah Nov 9, 2022
c0a09bb
code clean
qweeah Nov 9, 2022
1b0fa11
use unit32 instead
qweeah Nov 9, 2022
0d24bac
use uint ptr
qweeah Nov 9, 2022
29c24b3
rename status
qweeah Nov 9, 2022
19be1f8
add limitation in readme
qweeah Nov 9, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -44,18 +44,21 @@ jobs:
- name: Run E2E Tests
run: |
cd $GITHUB_WORKSPACE/test/e2e
mnt_root="$GITHUB_WORKSPACE/test/e2e/testdata/distribution/mount"
go install github.com/onsi/ginkgo/v2/ginkgo
mnt_root="$GITHUB_WORKSPACE/test/e2e/testdata/distribution/mount"
rm -rf $mnt_root/docker
for layer in $(ls $mnt_root/*.tar.gz); do
tar -xvzf $layer -C $mnt_root
done
trap 'docker kill oras-e2e || true' ERR
docker run -d -p 5000:5000 --rm --name oras-e2e \
--env STORAGE_DELETE_ENABLED=true \
docker run --pull always -d -p 5000:5000 --rm --name oras-e2e \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to --pull always?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default behavior is missing, which can be wrong if github provided an uncleaned VM. always is more defensive.

--env REGISTRY_STORAGE_DELETE_ENABLED=true \
--env REGISTRY_AUTH_HTPASSWD_REALM=test-basic \
--env REGISTRY_AUTH_HTPASSWD_PATH=/etc/docker/registry/passwd \
--mount type=bind,source=$mnt_root/docker,target=/opt/data/registry-root-dir/docker \
--mount type=bind,source=$mnt_root/passwd_bcrypt,target=/etc/docker/registry/passwd \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we generate passwd file on the fly? I have concerns that security checkers will scan the repo and find we are storing credentials.

Copy link
Contributor Author

@qweeah qweeah Nov 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's doable but will complicate the setup. Can we set this aside and consider the design in another issue #680 with configurable username/password/token?

ghcr.io/oras-project/registry:v1.0.0-rc.2
ginkgo -r -p suite
ginkgo -r -p --succinct suite
docker kill oras-e2e || true
env:
ORAS_PATH: bin/linux/amd64/oras
Expand Down
13 changes: 8 additions & 5 deletions test/e2e/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# ORAS End-to-End Testing Dev Guide

**KNOWN LIMITATION**: E2E tests are designed to run in the CI and currently only support running on linux platform.
## Setting up
Minimal setup: Run the script in **step 3**

Expand Down Expand Up @@ -83,9 +83,9 @@ Two suites will be maintained for E2E testing:
Inside a suite, please follow below model when building the hierarchical collections of specs:
```
Describe: <Role>
Context: Scenario or command specific description
When: <Action>
It: <Result> (per-command execution)
When: Scenario or command specific description
It: <Action>
By: <Result> (per-command execution)
Expect: <Result> (detailed checks for execution results)
```

Expand Down Expand Up @@ -116,4 +116,7 @@ graph TD;
B1-- foo2 -->B2[blob1]
B1-- bar -->B3[blob2]
end
```
```

#### 5.2 Scenario Suite
Test files used by scenario-based specs are placed in `$REPO_ROOT/test/e2e/testdata/files`.
2 changes: 1 addition & 1 deletion test/e2e/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,12 @@ go 1.19
require (
github.com/onsi/ginkgo/v2 v2.1.6
github.com/onsi/gomega v1.20.2
github.com/opencontainers/go-digest v1.0.0
oras.land/oras-go/v2 v2.0.0-rc.3.0.20220922092058-3f9653f7bf69
)

require (
github.com/google/go-cmp v0.5.8 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.0.2 // indirect
github.com/oras-project/artifacts-spec v1.0.0-rc.2 // indirect
golang.org/x/net v0.0.0-20220722155237-a158d28d115b // indirect
Expand Down
171 changes: 171 additions & 0 deletions test/e2e/internal/utils/exec.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,171 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package utils

import (
"fmt"
"io"
"os"
"os/exec"
"strings"
"time"

ginkgo "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
"github.com/onsi/gomega/gexec"
"oras.land/oras/test/e2e/internal/utils/match"
)

const (
orasBinary = "oras"

// customize your own basic auth file via `htpasswd -cBb <file_name> <user_name> <password>`
Username = "hello"
Password = "oras-test"
AuthConfigPath = "test.config"
)

// ExecOption provides option used to execute a command.
type ExecOption struct {
binary string
args []string
workDir string
timeout time.Duration

stdin io.Reader
stdout []match.Matcher
stderr []match.Matcher
shouldFail bool
qweeah marked this conversation as resolved.
Show resolved Hide resolved

text string
}

// ORAS returns default execution option for oras binary.
func ORAS(args ...string) *ExecOption {
return Binary(orasBinary, args...)
}

// Binary returns default execution option for customized binary.
func Binary(path string, args ...string) *ExecOption {
return &ExecOption{
binary: path,
args: args,
timeout: 10 * time.Second,
shouldFail: false,
}
}

// WithFailureCheck sets failure exit code checking for the execution.
func (opts *ExecOption) WithFailureCheck() *ExecOption {
opts.shouldFail = true
return opts
}

// WithTimeOut sets timeout for the execution.
func (opts *ExecOption) WithTimeOut(timeout time.Duration) *ExecOption {
opts.timeout = timeout
return opts
}

// WithDescription sets description text for the execution.
func (opts *ExecOption) WithDescription(text string) *ExecOption {
opts.text = text
return opts
}

// WithWorkDir sets working directory for the execution.
func (opts *ExecOption) WithWorkDir(path string) *ExecOption {
opts.workDir = path
return opts
}

// WithInput redirects stdin to r for the execution.
func (opts *ExecOption) WithInput(r io.Reader) *ExecOption {
opts.stdin = r
return opts
}

// MatchKeyWords adds keywords matching to stdout.
func (opts *ExecOption) MatchKeyWords(keywords ...string) *ExecOption {
opts.stdout = append(opts.stdout, match.NewKeywordMatcher(keywords))
return opts
}

// MatchErrKeyWords adds keywords matching to stderr.
func (opts *ExecOption) MatchErrKeyWords(keywords ...string) *ExecOption {
opts.stderr = append(opts.stderr, match.NewKeywordMatcher(keywords))
return opts
}

// MatchContent adds full content matching to the execution.
func (opts *ExecOption) MatchContent(content string) *ExecOption {
if !opts.shouldFail {
opts.stdout = append(opts.stdout, match.NewContentMatcher(content))
} else {
opts.stderr = append(opts.stderr, match.NewContentMatcher(content))
}
return opts
}

// MatchStatus adds full content matching to the execution option.
func (opts *ExecOption) MatchStatus(keys []match.StateKey, verbose bool, successCount int) *ExecOption {
opts.stdout = append(opts.stdout, match.NewStatusMatcher(keys, opts.args[0], verbose, successCount))
return opts
}

// Exec run the execution based on opts.
func (opts *ExecOption) Exec() *gexec.Session {
if opts == nil {
// this should be a code error but can only be caught during runtime
panic("Nil option for command execution")
}

if opts.text == "" {
if opts.shouldFail {
opts.text = "fail"
} else {
opts.text = "pass"
}
}
description := fmt.Sprintf(">> should %s: %s %s >>", opts.text, opts.binary, strings.Join(opts.args, " "))
ginkgo.By(description)

var cmd *exec.Cmd
if opts.binary == orasBinary {
opts.binary = ORASPath
}
cmd = exec.Command(opts.binary, opts.args...)
cmd.Stdin = opts.stdin
if opts.workDir != "" {
// switch working directory
wd, err := os.Getwd()
Expect(err).ShouldNot(HaveOccurred())
Expect(os.Chdir(opts.workDir)).ShouldNot(HaveOccurred())
defer os.Chdir(wd)
}
fmt.Println(description)
session, err := gexec.Start(cmd, os.Stdout, os.Stderr)
Expect(err).ShouldNot(HaveOccurred())
Expect(session.Wait(opts.timeout).ExitCode() != 0).Should(Equal(opts.shouldFail))

// matching result
for _, s := range opts.stdout {
s.Match(session.Out)
}
for _, s := range opts.stderr {
s.Match(session.Err)
}

return session
}
66 changes: 66 additions & 0 deletions test/e2e/internal/utils/file.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package utils

import (
"io"
"io/fs"
"os"
"path/filepath"
)

var testFileRoot string

// CopyTestData copies test data into the temp test folder.
func CopyTestData(dstRoot string) error {
return filepath.WalkDir(testFileRoot, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() {
qweeah marked this conversation as resolved.
Show resolved Hide resolved
// ignore folder
return nil
}

relPath, err := filepath.Rel(testFileRoot, path)
if err != nil {
return err
}
dstPath := filepath.Join(dstRoot, relPath)
// make sure all parents are created
if err := os.MkdirAll(filepath.Dir(dstPath), 0700); err != nil {
return err
}

// copy with original folder structure
return copyFile(path, dstPath)
})
}

func copyFile(srcFile, dstFile string) error {
to, err := os.Create(dstFile)
if err != nil {
return err
}
defer to.Close()

from, err := os.Open(srcFile)
if err != nil {
return err
}
defer from.Close()

_, err = io.Copy(to, from)
return err
}
27 changes: 16 additions & 11 deletions test/e2e/internal/utils/init.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ package utils
import (
"fmt"
"os"
"os/exec"
"path/filepath"

. "github.com/onsi/ginkgo/v2"
Expand All @@ -42,27 +43,31 @@ func init() {
if err := ref.ValidateRegistry(); err != nil {
panic(err)
}
// setup test data
pwd, err := os.Getwd()
if err != nil {
panic(err)
}
testFileRoot = filepath.Join(pwd, "..", "..", "testdata", "files")
BeforeSuite(func() {
ORASPath = os.Getenv("ORAS_PATH")
if filepath.IsAbs(ORASPath) {
fmt.Printf("Testing based on pre-built binary locates in %q\n", ORASPath)
return
}

var err error
if workspacePath := os.Getenv("GITHUB_WORKSPACE"); ORASPath != "" && workspacePath != "" {
} else if workspacePath := os.Getenv("GITHUB_WORKSPACE"); ORASPath != "" && workspacePath != "" {
// add workspacePath as prefix, both path env should not be empty
ORASPath = filepath.Join(workspacePath, ORASPath)
ORASPath, err = filepath.Abs(ORASPath)
gomega.Expect(err).NotTo(gomega.HaveOccurred())
fmt.Printf("Testing based on pre-built binary locates in %q\n", ORASPath)
return
} else {
// fallback to native build to facilitate local debugging
ORASPath, err = gexec.Build("oras.land/oras/cmd/oras")
gomega.Expect(err).NotTo(gomega.HaveOccurred())
DeferCleanup(gexec.CleanupBuildArtifacts)
fmt.Printf("Testing based on temp binary locates in %q\n", ORASPath)
}

// fallback to native build to facilitate local debugging
ORASPath, err = gexec.Build("oras.land/oras/cmd/oras")
gomega.Expect(err).NotTo(gomega.HaveOccurred())
DeferCleanup(gexec.CleanupBuildArtifacts)
fmt.Printf("Testing based on temp binary locates in %q\n", ORASPath)
cmd := exec.Command(ORASPath, "login", Host, "-u", Username, "-p", Password)
gomega.Expect(cmd.Run()).ShouldNot(gomega.HaveOccurred())
})
}
33 changes: 33 additions & 0 deletions test/e2e/internal/utils/match/content.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package match

import (
. "github.com/onsi/gomega"
"github.com/onsi/gomega/gbytes"
)

// contentMatcher provides whole matching of the output.
type contentMatcher string

// NewContentMatcher returns a content matcher.
func NewContentMatcher(s string) contentMatcher {
return contentMatcher(s)
}

// Match matches got with s.
func (c contentMatcher) Match(got *gbytes.Buffer) {
content := got.Contents()
Expect(contentMatcher(content)).Should(Equal(c))
}
Loading