From 6378f183e82b7c92d00978fee750b7096e307053 Mon Sep 17 00:00:00 2001 From: Gaius Date: Wed, 3 Nov 2021 14:50:04 +0800 Subject: [PATCH] feat: api Signed-off-by: Gaius --- .github/workflows/lint.yml | 3 +- CODE_OF_CONDUCT.md | 49 ++++- CONTRIBUTING.md | 197 +++++++++++++----- README.md | 80 +++++-- api/README.md | 27 ++- build/images/base/README.md | 24 ++- docs/zh-CN/FAQ.md | 1 - docs/zh-CN/README.md | 5 +- docs/zh-CN/ROADMAP.md | 1 - .../runtime-integration/containerd/mirror.md | 23 +- .../runtime-integration/containerd/proxy.md | 55 ++--- docs/zh-CN/runtime-integration/cri-o.md | 16 +- docs/zh-CN/runtime-integration/docker.md | 54 ++--- docs/zh-CN/test-guide/test-guide.md | 10 +- docs/zh-CN/troubleshooting/README.md | 32 +-- test/README.md | 10 +- test/tools/stress/README.md | 63 +++--- 17 files changed, 415 insertions(+), 235 deletions(-) delete mode 100644 docs/zh-CN/FAQ.md delete mode 100644 docs/zh-CN/ROADMAP.md diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index 0a2c8d83cef..ac99f535efe 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -26,4 +26,5 @@ jobs: - name: Markdown lint uses: docker://avtodev/markdown-lint:v1 with: - args: "**/*.md" + args: '**/*.md' + ignore: 'deploy/helm-charts manager/console CHANGELOG.md docs/en/api-reference docs/zh-CN/api-reference' diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md index b2f50cf3e8d..6adeea4b86a 100644 --- a/CODE_OF_CONDUCT.md +++ b/CODE_OF_CONDUCT.md @@ -2,7 +2,15 @@ ## Our Pledge -In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation. +In the interest of fostering an open and welcoming environment, +we as contributors and maintainers pledge to +making participation in our project and +our community a harassment-free experience for everyone, +regardless of age, body size, +disability, ethnicity, gender identity and expression, +level of experience, nationality, +personal appearance, race, religion, +or sexual identity and orientation. ## Our Standards @@ -19,29 +27,50 @@ Examples of unacceptable behavior by participants include: * The use of sexualized language or imagery and unwelcome sexual attention or advances * Trolling, insulting/derogatory comments, and personal or political attacks * Public or private harassment -* Publishing others' private information, such as a physical or electronic address, without explicit permission -* Other conduct which could reasonably be considered inappropriate in a professional setting +* Publishing others' private information, + such as a physical or electronic address, without explicit permission +* Other conduct which could reasonably be considered inappropriate in + a professional setting ## Our Responsibilities -Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior. +Project maintainers are responsible for clarifying the standards of +acceptable behavior and are expected to take appropriate and +fair corrective action in response to any instances of unacceptable behavior. -Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful. +Project maintainers have the right and responsibility to remove, +edit, or reject comments, commits, code, wiki edits, issues, +and other contributions that are not aligned to this Code of Conduct, +or to ban temporarily or permanently any contributor for +other behaviors that they deem inappropriate, threatening, offensive, or harmful. ## Scope -This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers. +This Code of Conduct applies both within project spaces and in public +spaces when an individual is representing the project or its community. +Examples of representing a project or community include using an official +project e-mail address, posting via an official social media account, +or acting as an appointed representative at an online or offline event. +Representation of a project may be further defined and clarified by project maintainers. ## Enforcement -Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at . The project team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately. +Instances of abusive, harassing, or otherwise unacceptable +behavior may be reported by contacting the project team at +. The project team will review +and investigate all complaints, and will respond in a way +that it deems appropriate to the circumstances. The project team is +obligated to maintain confidentiality with regard to the reporter of an incident. +Further details of specific enforcement policies may be posted separately. -Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership. +Project maintainers who do not follow or enforce the Code of +Conduct in good faith may face temporary or permanent repercussions as +determined by other members of the project's leadership. ## Attribution -This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [https://www.contributor-covenant.org/version/2/0/][version] +This Code of Conduct is adapted from the [Contributor Covenant][homepage], +version 1.4, available at [https://www.contributor-covenant.org/version/2/0/][version] [homepage]: https://contributor-covenant.org [version]: https://www.contributor-covenant.org/version/2/0/ - diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d7bf455752c..256b65acbf7 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,6 +1,8 @@ # Contributing to Dragonfly -It is warmly welcomed if you have interest to hack on Dragonfly. First, we encourage this kind of willing very much. And here is a list of contributing guide for you. +It is warmly welcomed if you have interest to hack on Dragonfly. +First, we encourage this kind of willing very much. +And here is a list of contributing guide for you. ## Topics @@ -12,15 +14,32 @@ It is warmly welcomed if you have interest to hack on Dragonfly. First, we encou ## Reporting security issues -Security issues are always treated seriously. As our usual principle, we discourage anyone to spread security issues. If you find a security issue of Dragonfly, please do not discuss it in public and even do not open a public issue. Instead we encourage you to send us a private email to [Dragonfly@noreply.github.com](mailto:Dragonfly@noreply.github.com) to report this. +Security issues are always treated seriously. +As our usual principle, we discourage anyone to spread security issues. +If you find a security issue of Dragonfly, please do not discuss it in +public and even do not open a public issue. +Instead we encourage you to send us a private email to +[Dragonfly@noreply.github.com](mailto:Dragonfly@noreply.github.com) to report this. ## Reporting general issues -To be honest, we regard every user of Dragonfly as a very kind contributor. After experiencing Dragonfly, you may have some feedback for the project. Then feel free to open an issue via [NEW ISSUE](https://github.com/dragonflyoss/Dragonfly2/issues/new). +To be honest, we regard every user of Dragonfly as a very kind contributor. +After experiencing Dragonfly, you may have some feedback for the project. +Then feel free to open an issue via +[NEW ISSUE](https://github.com/dragonflyoss/Dragonfly2/issues/new). -Since we collaborate project Dragonfly in a distributed way, we appreciate **WELL-WRITTEN**, **DETAILED**, **EXPLICIT** issue reports. To make the communication more efficient, we wish everyone could search if your issue is an existing one in the searching list. If you find it existing, please add your details in comments under the existing issue instead of opening a brand new one. +Since we collaborate project Dragonfly in a distributed way, +we appreciate **WELL-WRITTEN**, **DETAILED**, **EXPLICIT** issue reports. +To make the communication more efficient, we wish everyone could +search if your issue is an existing one in the searching list. +If you find it existing, please add your details in comments +under the existing issue instead of opening a brand new one. -To make the issue details as standard as possible, we setup an [ISSUE TEMPLATE](./.github/ISSUE_TEMPLATE) for issue reporters. You can find three kinds of issue templates there: question, bug report and feature request. Please **BE SURE** to follow the instructions to fill fields in template. +To make the issue details as standard as possible, +we setup an [ISSUE TEMPLATE](./.github/ISSUE_TEMPLATE) for issue reporters. +You can find three kinds of issue templates there: question, +bug report and feature request. Please **BE SURE** to follow +the instructions to fill fields in template. There are lot of cases when you could open an issue: @@ -35,11 +54,16 @@ There are lot of cases when you could open an issue: * any questions on project * and so on -Also we must remind that when filing a new issue, please remember to remove the sensitive data from your post. Sensitive data could be password, secret key, network locations, private business data and so on. +Also we must remind that when filing a new issue, +please remember to remove the sensitive data from your post. +Sensitive data could be password, secret key, +network locations, private business data and so on. ## Code and doc contribution -Every action to make project Dragonfly better is encouraged. On GitHub, every improvement for Dragonfly could be via a PR (short for pull request). +Every action to make project Dragonfly better is encouraged. +On GitHub, every improvement for Dragonfly could be +via a PR (short for pull request). * If you find a typo, try to fix it! * If you find a bug, try to fix it! @@ -52,11 +76,13 @@ Every action to make project Dragonfly better is encouraged. On GitHub, every im * If you find document incorrect, just do it and fix that! * ... -Actually it is impossible to list them completely. Just remember one princinple: +Actually it is impossible to list them completely. +Just remember one princinple: > WE ARE LOOKING FORWARD TO ANY PR FROM YOU. -Since you are ready to improve Dragonfly with a PR, we suggest you could take a look at the PR rules here. +Since you are ready to improve Dragonfly with a PR, +we suggest you could take a look at the PR rules here. * [Workspace Preparation](#workspace-preparation) * [Branch Definition](#branch-definition) @@ -67,61 +93,87 @@ Since you are ready to improve Dragonfly with a PR, we suggest you could take a ### Workspace Preparation -To put forward a PR, we assume you have registered a GitHub ID. Then you could finish the preparation in the following steps: +To put forward a PR, we assume you have registered +a GitHub ID. Then you could finish +the preparation in the following steps: -1. **FORK** Dragonfly to your repository. To make this work, you just need to click the button Fork in right-left of [dragonflyoss/Dragonfly2](https://github.com/dragonflyoss/Dragonfly2) main page. Then you will end up with your repository in `https://github.com//Dragonfly`, in which `your-username` is your GitHub username. +1. **FORK** Dragonfly to your repository. +To make this work, you just need to click the button Fork +in right-left of [dragonflyoss/Dragonfly2](https://github.com/dragonflyoss/Dragonfly2) +main page. Then you will end up with your repository in +`https://github.com//Dragonfly`, +in which `your-username` is your GitHub username. -2. **CLONE** your own repository to develop locally. Use `git clone https://github.com//Dragonfly2.git` to clone repository to your local machine. Then you can create new branches to finish the change you wish to make. +2. **CLONE** your own repository to develop locally. +Use `git clone https://github.com//Dragonfly2.git` +to clone repository to your local machine. +Then you can create new branches to finish the change you wish to make. -3. **Set Remote** upstream to be `https://github.com/dragonflyoss/Dragonfly2.git` using the following two commands: +3. **Set Remote** upstream to be +`https://github.com/dragonflyoss/Dragonfly2.git` - ``` - git remote add upstream https://github.com/dragonflyoss/Dragonfly2.git - git remote set-url --push upstream no-pushing - ``` + using the following two commands: - With this remote setting, you can check your git remote configuration like this: + ```bash + git remote add upstream https://github.com/dragonflyoss/Dragonfly2.git + git remote set-url --push upstream no-pushing + ``` - ``` - $ git remote -v - origin https://github.com//Dragonfly2.git (fetch) - origin https://github.com//Dragonfly2.git (push) - upstream https://github.com/dragonflyoss/Dragonfly2.git (fetch) - upstream no-pushing (push) - ``` + With this remote setting, you can check your git remote configuration like this: - Adding this, we can easily synchronize local branches with upstream branches. + ```bash + $ git remote -v + origin https://github.com//Dragonfly2.git (fetch) + origin https://github.com//Dragonfly2.git (push) + upstream https://github.com/dragonflyoss/Dragonfly2.git (fetch) + upstream no-pushing (push) + ``` + + Adding this, we can easily synchronize local branches with upstream branches. 4. **Create a branch** to add a new feature or fix issues - Update local working directory: + Update local working directory: - ``` - cd Dragonfly2 - git fetch upstream - git checkout main - git rebase upstream/main - ``` + ```bash + cd Dragonfly2 + git fetch upstream + git checkout main + git rebase upstream/main + ``` - Create a new branch: + Create a new branch: - ``` - git checkout -b - ``` + ```bash + git checkout -b + ``` - Make any change on the `new-branch` then build and test your codes. + Make any change on the `new-branch` then build and test your codes. ### Branch Definition -Right now we assume every contribution via pull request is for [branch main](https://github.com/dragonflyoss/Dragonfly2/tree/main) in Dragonfly. Before contributing, be aware of branch definition would help a lot. +Right now we assume every contribution via pull +request is for [branch main](https://github.com/dragonflyoss/Dragonfly2/tree/main) +in Dragonfly. Before contributing, +be aware of branch definition would help a lot. -As a contributor, keep in mind again that every contribution via pull request is for branch main. While in project Dragonfly, there are several other branches, we generally call them rc branches, release branches and backport branches. +As a contributor, keep in mind again that every +contribution via pull request is for branch main. +While in project Dragonfly, there are several other branches, +we generally call them rc branches, release branches and backport branches. -Before officially releasing a version, we will checkout a rc(release candidate) branch. In this branch, we will test more than branch main. +Before officially releasing a version, +we will checkout a rc(release candidate) branch. +In this branch, we will test more than branch main. -When officially releasing a version, there will be a release branch before tagging. After tagging, we will delete the release branch. +When officially releasing a version, +there will be a release branch before tagging. +After tagging, we will delete the release branch. -When backporting some fixes to existing released version, we will checkout backport branches. After backporting, the backporting effects will be in PATCH number in MAJOR.MINOR.PATCH of [SemVer](http://semver.org/). +When backporting some fixes to existing released version, +we will checkout backport branches. After backporting, +the backporting effects will be in PATCH number in +MAJOR.MINOR.PATCH of [SemVer](http://semver.org/). ### Commit Rules @@ -132,7 +184,12 @@ Actually in Dragonfly, we take two rules serious when committing: #### Commit Message -Commit message could help reviewers better understand what the purpose of submitted PR is. It could help accelerate the code review procedure as well. We encourage contributors to use **EXPLICIT** commit message rather than ambiguous message. In general, we advocate the following commit message type: +Commit message could help reviewers better understand +what the purpose of submitted PR is. +It could help accelerate the code review procedure as well. +We encourage contributors to use **EXPLICIT** commit +message rather than ambiguous message. In general, +we advocate the following commit message type: * feat: xxxx.For example, "feat: make result show in sorted order". * fix: xxxx. For example, "fix: fix panic when input nil parameter". @@ -140,9 +197,11 @@ Commit message could help reviewers better understand what the purpose of submit * style: xxxx. For example, "style: format the code style of Constants.java". * refactor: xxxx. For example, "refactor: simplify to make codes more readable". * test: xxx. For example, "test: add unit test case for func InsertIntoArray". -* chore: xxx. For example, "chore: integrate travis-ci". It's the type of mantainance change. +* chore: xxx. For example, "chore: integrate travis-ci". +It's the type of mantainance change. -On the other side, we discourage contributors from committing message like the following ways: +On the other side, we discourage contributors +from committing message like the following ways: * ~~fix bug~~ * ~~update~~ @@ -150,20 +209,32 @@ On the other side, we discourage contributors from committing message like the f #### Commit Content -Commit content represents all content changes included in one commit. We had better include things in one single commit which could support reviewer's complete review without any other commits' help. In another word, contents in one single commit can pass the CI to avoid code mess. In brief, there are two minor rules for us to keep in mind: +Commit content represents all content changes +included in one commit. We had better include things +in one single commit which could support reviewer's complete +review without any other commits' help. In another word, +contents in one single commit can pass the CI to avoid code mess. +In brief, there are two minor rules for us to keep in mind: * avoid very large change in a commit. * complete and reviewable for each commit. -No matter what the commit message, or commit content is, we do take more emphasis on code review. +No matter what the commit message, or commit content is, +we do take more emphasis on code review. ### PR Description -PR is the only way to make change to Dragonfly project files. To help reviewers better get your purpose, PR description could not be too detailed. We encourage contributors to follow the [PR template](./.github/PULL_REQUEST_TEMPLATE.md) to finish the pull request. +PR is the only way to make change to Dragonfly project files. +To help reviewers better get your purpose, +PR description could not be too detailed. +We encourage contributors to follow the +[PR template](./.github/PULL_REQUEST_TEMPLATE.md) to finish the pull request. ### Developing Environment -As a contributor, if you want to make any contribution to Dragonfly project, we should reach an agreement on the version of tools used in the development environment. +As a contributor, if you want to make any contribution to +Dragonfly project, we should reach an agreement on +the version of tools used in the development environment. Here are some dependents with specific version: * golang : v1.15 @@ -171,11 +242,19 @@ Here are some dependents with specific version: * shellCheck : latest * docker: latest -When you develop the Dragonfly project at the local environment, you should use subcommands of Makefile to help yourself to check and build the latest version of Dragonfly. For the convenience of developers, we use the docker to build Dragonfly. It can reduce problems of the developing environment. +When you develop the Dragonfly project at the local environment, +you should use subcommands of Makefile to help yourself +to check and build the latest version of Dragonfly. +For the convenience of developers, +we use the docker to build Dragonfly. +It can reduce problems of the developing environment. ### Golang Dependency Management -The Dragonfly project uses [Go modules](https://golang.org/cmd/go/#hdr-Modules__module_versions__and_more) to manage dependencies on external packages. This requires a working Go environment with version 1.15 or greater installed. +The Dragonfly project uses +[Go modules](https://golang.org/cmd/go/#hdr-Modules__module_versions__and_more) +to manage dependencies on external packages. +This requires a working Go environment with version 1.15 or greater installed. To add or update a new dependency, use the `go get` command: @@ -193,7 +272,8 @@ Tidy up the `go.mod` and `go.sum` files: go mod tidy ``` -You have to commit the changes to `go.mod`, `go.sum` before submitting the pull request. +You have to commit the changes to `go.mod`, +`go.sum` before submitting the pull request. ```bash git add go.mod go.sum @@ -202,7 +282,10 @@ git commit ## Engage to help anything -We choose GitHub as the primary place for Dragonfly to collaborate. So the latest updates of Dragonfly are always here. Although contributions via PR is an explicit way to help, we still call for any other ways. +We choose GitHub as the primary place for Dragonfly to +collaborate. So the latest updates of Dragonfly are always here. +Although contributions via PR is an explicit way to help, +we still call for any other ways. * Reply to other's issues if you could. * Help solve other user's problems. @@ -216,7 +299,8 @@ In a word, **ANY HELP IS CONTRIBUTION.** ## Join Dragonfly as a member -It is also welcomed to join Dragonfly team if you are willing to participate in Dragonfly community continuously and keep active. +It is also welcomed to join Dragonfly team if you are +willing to participate in Dragonfly community continuously and keep active. ### Requirements @@ -233,5 +317,6 @@ It is also welcomed to join Dragonfly team if you are willing to participate in You can do it in either of two ways: -* Submit a PR in the [dragonflyoss/Dragonfly2](https://github.com/dragonflyoss/Dragonfly2) repo. +* Submit a PR in the +[dragonflyoss/Dragonfly2](https://github.com/dragonflyoss/Dragonfly2) repo. * Contact with the community's [maintainers](MAINTAINERS.md) offline. diff --git a/README.md b/README.md index 72cbd75fb1d..63dd22765e5 100644 --- a/README.md +++ b/README.md @@ -14,46 +14,76 @@ [![GoDoc](https://godoc.org/github.com/dragonflyoss/Dragonfly2?status.svg)](https://godoc.org/github.com/dragonflyoss/Dragonfly2) [![LICENSE](https://img.shields.io/github/license/dragonflyoss/Dragonfly2.svg?style=flat-square)](https://github.com/dragonflyoss/Dragonfly2/blob/main/LICENSE) -Provide efficient, stable, secure, low-cost file and image distribution services to be the best practice and standard solution in the related Cloud-Native area. +Provide efficient, stable, secure, low-cost file and +image distribution services to be the best practice and +standard solution in the related Cloud-Native area. ## Introduction -Dragonfly is an open source intelligent P2P based image and file distribution system. Its goal is to tackle all distribution problems in cloud native scenarios. Currently Dragonfly focuses on being: +Dragonfly is an open source intelligent P2P based image and +file distribution system. Its goal is to +tackle all distribution problems in cloud native scenarios. +Currently Dragonfly focuses on being: - Simple: well-defined user-facing API (HTTP), non-invasive to all container engines; - Efficient: CDN support, P2P based file distribution to save enterprise bandwidth; - Intelligent: host level speed limit, intelligent flow control due to host detection; - Secure: block transmission encryption, HTTPS connection support. -Dragonfly is now hosted by the Cloud Native Computing Foundation (CNCF) as an Incubating Level Project. Originally it was born to solve all kinds of distribution at very large scales, such as application distribution, cache distribution, log distribution, image distribution, and so on. +Dragonfly is now hosted by the Cloud Native Computing Foundation (CNCF) as +an Incubating Level Project. Originally it was born to +solve all kinds of distribution at very large scales, +such as application distribution, cache distribution, +log distribution, image distribution, and so on. ## Features -- Implement P2P files distribution with various storage types (HDFS, storage services from various cloud vendors, Maven, Yum, etc.) through a unified back-to-source adapter layer. -- Support more distribution modes: active pull, active push, real-time synchronization, remote replication, automatic warm-up, cross-cloud transmission, etc. -- Provide separation and decoupling between systems, scheduling and plug-in CDN. Support on-demand deployment with flexible types: light or heavy, inside or outside, to meet the actual needs of different scenarios. -- Newly designed P2P protocol framework based on GRPC with improved efficiency and stability. -- Perform encrypted transmission, account-based transmission authentication and rate limit, and multi-tenant isolation mechanism. +- Implement P2P files distribution with various storage types +(HDFS, storage services from various cloud vendors, Maven, Yum, etc.) +through a unified back-to-source adapter layer. +- Support more distribution modes: +active pull, active push, real-time synchronization, +remote replication, automatic warm-up, +cross-cloud transmission, etc. +- Provide separation and decoupling between systems, +scheduling and plug-in CDN. Support on-demand deployment with +flexible types: light or heavy, inside or outside, +to meet the actual needs of different scenarios. +- Newly designed P2P protocol framework based on +GRPC with improved efficiency and stability. +- Perform encrypted transmission, account-based transmission +authentication and rate limit, and multi-tenant isolation mechanism. - Bear more efficient IO methods: multithreaded IO, memory mapping, DMA, etc. -- Advocate dynamic compression, in-memory file systems, and more efficient scheduling algorithms to improve distribution efficiency. -- Client allows third-party software to natively integrate Dragonfly’s P2P capabilities through C/S mode. -- Productivity: Support file uploading, task management of various distribution modes, data visualization, global control, etc. -- Consistent internal and external versions, shared core features, and individual extensions of non-generic features. -- Enhanced integration with ecology: Harbor, Nydus (on-demand image download), warehouse services for various cloud vendors, etc. +- Advocate dynamic compression, in-memory file systems, +and more efficient scheduling algorithms to improve distribution efficiency. +- Client allows third-party software to +natively integrate Dragonfly’s P2P capabilities through C/S mode. +- Productivity: Support file uploading, task management of +various distribution modes, data visualization, global control, etc. +- Consistent internal and external versions, shared core features, +and individual extensions of non-generic features. +- Enhanced integration with ecology: +Harbor, Nydus (on-demand image download), +warehouse services for various cloud vendors, etc. ## Architecture ![alt][arch] -**Manager:** Used to manage the dynamic configuration that each module depends on, and provide keepalive and metrics functions. +**Manager:** Used to manage the dynamic configuration that +each module depends on, and provide keepalive and metrics functions. -**Scheduler:** The tracker and scheduler in the P2P network that choose appropriate downloading net-path for each peer. +**Scheduler:** The tracker and scheduler in the P2P +network that choose appropriate downloading net-path for each peer. -**CDN:** A CDN server that caches downloaded data from source to avoid downloading same files repeatedly. +**CDN:** A CDN server that caches downloaded data +from source to avoid downloading same files repeatedly. -**Daemon:** It's a daemon of dfget client. It establishes a proxy between containerd/CRI-O and registry. +**Daemon:** It's a daemon of dfget client. +It establishes a proxy between containerd/CRI-O and registry. -**Dfget:** The client of Dragonfly used for downloading files. It's similar to wget. +**Dfget:** The client of Dragonfly used for +downloading files. It's similar to wget. ## Documentation @@ -61,20 +91,27 @@ You can find the full documentation [on the repo][document]. ## Community -Welcome developers to actively participate in community discussions and contribute code to Dragonfly. We will remain concerned about the issues discussed in the community and respond quickly. +Welcome developers to actively participate in community discussions +and contribute code to Dragonfly. We will remain +concerned about the issues discussed in the community and respond quickly. - Discussions: [Github Discussion Forum][discussion] - Twitter: [@dragonfly_oss](https://twitter.com/dragonfly_oss) - DingTalk: 23304666 +
- +
+ ## Contributing -You should check out our [CONTRIBUTING][contributing] and develop the project together. + +You should check out our +[CONTRIBUTING][contributing] and develop the project together. ## Code of Conduct + Please refer to our [Code of Conduct][codeconduct]. [arch]: docs/en/images/arch.png @@ -84,3 +121,4 @@ Please refer to our [Code of Conduct][codeconduct]. [contributing]: CONTRIBUTING.md [codeconduct]: CODE_OF_CONDUCT.md [document]: docs/README.md +[dingtalk]: docs/en/images/community/dingtalk-group.jpeg diff --git a/api/README.md b/api/README.md index cbd5c2c634d..b0131151f9d 100644 --- a/api/README.md +++ b/api/README.md @@ -1,18 +1,25 @@ # Dragonfly V2 API -We encourage users to experience Dragonfly in different ways. When doing this, there are three ways users could choose to interact with dragonfly mostly: +We encourage users to experience Dragonfly in different ways. +When doing this, there are three ways users +could choose to interact with dragonfly mostly: * For end-users, command line tool `dfget`, `dfdaemon` is mostly used. -* For developers, Dragonfly manager's raw API is the original thing they would make use of. For more details about API docs, please refer to - [api.md](./manager/swagger.json). We should also keep it in mind that doc [apis.md](./manager/swagger.json) is - automatically - generated by - [swagger2markup](https://github.com/Swagger2Markup/swagger2markup). Please **DO NOT** edit [api.md](./manager/swagger.json) directly. +* For developers, Dragonfly manager's raw API is + the original thing they would make use of. + For more details about API docs, please refer to [api.md](./manager/swagger.json). + We should also keep it in mind that doc + [apis.md](./manager/swagger.json) is automatically generated by + [swagger2markup](https://github.com/Swagger2Markup/swagger2markup). + Please **DO NOT** edit [api.md](./manager/swagger.json) directly. Directory `/api` mainly describes the second part **Dragonfly Manager's Raw API**. -Currently, the description of swagger format is provided, which can be viewed in `manager`. It contains two format files of swagger, JSON and yaml. You can -browse through relevant viewing tools to obtain the specific call method of API, or view online through the `/swagger/doc.json` endpoint of manager service. - -Through manager's API, you can integrate with your own services, so you can automatically manage dragonfly and improve the level of Devops. +Currently, the description of swagger format is provided, +which can be viewed in `manager`. +It contains two format files of swagger, JSON and yaml. You can +browse through relevant viewing tools to obtain the specific call method of API, +or view online through the `/swagger/doc.json` endpoint of manager service. +Through manager's API, you can integrate with your own services, +so you can automatically manage dragonfly and improve the level of Devops. diff --git a/build/images/base/README.md b/build/images/base/README.md index 0494de04542..fefa2790dfb 100644 --- a/build/images/base/README.md +++ b/build/images/base/README.md @@ -1,6 +1,8 @@ -# Base Image for Debugging +# Base image -## Build Base Image +## Base Image for Debugging + +### Build Base Image ```shell BASE_IMAGE=dragonflyoss/base:bpftrace-v0.13.0-go-v1.16.6 @@ -12,7 +14,7 @@ docker run -ti -v /usr/src:/usr/src:ro \ ${BASE_IMAGE} ``` -## Build Dragonfly Images +### Build Dragonfly Images ```shell # tags for condition compiling @@ -25,9 +27,9 @@ export BASE_IMAGE=dragonflyoss/base:bpftrace-v0.13.0-go-v1.16.6 make docker-build ``` -# Debug With Delve +## Debug With Delve -## Prepare Code for Debug +### Prepare Code for Debug ```shell COMMIT_ID=c1c3d652 @@ -36,15 +38,15 @@ git clone https://github.com/dragonflyoss/Dragonfly2.git /go/src/d7y.io/dragonfl git reset --hard ${COMMIT_ID} ``` -## Debug Operations +### Debug Operations 1. Attach Process -```shell -pid=$(pidof scheduler) # or dfget, cdn, manager -dlv attach $pid -``` + ```shell + pid=$(pidof scheduler) # or dfget, cdn, manager + dlv attach $pid + ``` 2. Debug -Follow https://github.com/go-delve/delve/tree/v1.7.0/Documentation/cli \ No newline at end of file + Follow diff --git a/docs/zh-CN/FAQ.md b/docs/zh-CN/FAQ.md deleted file mode 100644 index 32cce9075d6..00000000000 --- a/docs/zh-CN/FAQ.md +++ /dev/null @@ -1 +0,0 @@ -# FAQ \ No newline at end of file diff --git a/docs/zh-CN/README.md b/docs/zh-CN/README.md index 7bf12bde170..d6bb9670c59 100644 --- a/docs/zh-CN/README.md +++ b/docs/zh-CN/README.md @@ -1,6 +1,9 @@ # 蜻蜓文档 -Dragonfly Document 是关于 Dragonfly 的介绍。对于一般对 Dragonfly 感兴趣的人，这个 repo 的 README.md 就足够了。对于最终用户来说，`/docs` 中包含的所有详细信息是最好的指南。而对于开发者来说，[开发者指南](developer-guide/developer-guide.md)部分的内容才是最需要的。 +Dragonfly Document 是关于 Dragonfly 的介绍。对于一般对 Dragonfly 感兴趣的人， +这个 repo 的 README.md 就足够了。对于最终用户来说 +`/docs` 中包含的所有详细信息是最好的指南。 +而对于开发者来说，[开发者指南](developer-guide/developer-guide.md)部分的内容才是最需要的。 文档文件组织如下： diff --git a/docs/zh-CN/ROADMAP.md b/docs/zh-CN/ROADMAP.md deleted file mode 100644 index 971d14480b7..00000000000 --- a/docs/zh-CN/ROADMAP.md +++ /dev/null @@ -1 +0,0 @@ -# Roadmap \ No newline at end of file diff --git a/docs/zh-CN/runtime-integration/containerd/mirror.md b/docs/zh-CN/runtime-integration/containerd/mirror.md index ae06d483984..9a42ab0f62c 100644 --- a/docs/zh-CN/runtime-integration/containerd/mirror.md +++ b/docs/zh-CN/runtime-integration/containerd/mirror.md @@ -45,9 +45,8 @@ version = 2 配置支持两个镜像仓库地址 `http://127.0.0.1:65001` 以及 `https://registry-1.docker.io`. -> 详细 Containerd 配置文档参照: https://github.com/containerd/containerd/blob/v1.5.2/docs/cri/registry.md#configure-registry-endpoint - -> 镜像仓库配置参照文档: https://github.com/containerd/containerd/blob/v1.5.2/docs/cri/config.md#registry-configuration +> 详细 Containerd 配置文档参照: +> 镜像仓库配置参照文档: ### 选项 2: 多镜像仓库 @@ -84,17 +83,17 @@ server = "https://example.com" ##### 选项 2: 自动生成 hosts.toml -自动生成 hosts.toml 脚本为 https://github.com/dragonflyoss/Dragonfly2/blob/main/hack/gen-containerd-hosts.sh +自动生成 hosts.toml 脚本为 ```shell bash gen-containerd-hosts.sh example.com ``` -> 镜像仓库配置详细文档参照: https://github.com/containerd/containerd/blob/main/docs/hosts.md#registry-configuration---examples +> 镜像仓库配置详细文档参照: ## Step 3: 重启 Containerd -``` +```shell systemctl restart containerd ``` @@ -102,7 +101,7 @@ systemctl restart containerd 使用以下命令拉取镜像: -``` +```shell crictl pull docker.io/library/busybox ``` @@ -110,12 +109,16 @@ crictl pull docker.io/library/busybox 可以查看日志，判断 busybox 镜像正常拉取。 -```bash +```shell grep 'register peer task result' /var/log/dragonfly/daemon/*.log ``` 如果正常日志输出如下: -``` -{"level":"info","ts":"2021-02-23 20:03:20.306","caller":"client/client.go:83","msg":"register peer task result:true[200] for taskId:adf62a86f001e17037eedeaaba3393f3519b80ce,peerIp:10.15.233.91,securityDomain:,idc:,scheduler:127.0.0.1:8002","peerId":"10.15.233.91-65000-43096-1614081800301788000","errMsg":null} +```shell +{"level":"info","ts":"2021-02-23 20:03:20.306","caller":"client/client.go:83", +"msg":"register peer task result:true[200] for +taskId:adf62a86f001e17037eedeaaba3393f3519b80ce, +peerIp:10.15.233.91,securityDomain:,idc:,scheduler:127.0.0.1:8002", +"peerId":"10.15.233.91-65000-43096-1614081800301788000","errMsg":null} ``` diff --git a/docs/zh-CN/runtime-integration/containerd/proxy.md b/docs/zh-CN/runtime-integration/containerd/proxy.md index 26d8a595d68..efedc257d95 100644 --- a/docs/zh-CN/runtime-integration/containerd/proxy.md +++ b/docs/zh-CN/runtime-integration/containerd/proxy.md @@ -1,6 +1,7 @@ # 使用 dfget daemon 作为 containerd 的 http 代理 -目前 containerd 的 `ctr` 命令不支持带有 registry-mirrors 的私有注册表，为此，我们需要为 containerd 使用 HTTP 代理。 +目前 containerd 的 `ctr` 命令不支持带有 +registry-mirrors 的私有注册表，为此我们需要为 containerd 使用 HTTP 代理。 ## 快速开始 @@ -16,31 +17,31 @@ openssl genrsa -out ca.key 2048 ```text [ req ] -#default_bits = 2048 -#default_md = sha256 -#default_keyfile = privkey.pem -distinguished_name = req_distinguished_name -attributes = req_attributes +#default_bits = 2048 +#default_md = sha256 +#default_keyfile = privkey.pem +distinguished_name = req_distinguished_name +attributes = req_attributes extensions = v3_ca req_extensions = v3_ca [ req_distinguished_name ] -countryName = Country Name (2 letter code) -countryName_min = 2 -countryName_max = 2 -stateOrProvinceName = State or Province Name (full name) -localityName = Locality Name (eg, city) -0.organizationName = Organization Name (eg, company) -organizationalUnitName = Organizational Unit Name (eg, section) -commonName = Common Name (eg, fully qualified host name) -commonName_max = 64 -emailAddress = Email Address -emailAddress_max = 64 +countryName = Country Name (2 letter code) +countryName_min = 2 +countryName_max = 2 +stateOrProvinceName = State or Province Name (full name) +localityName = Locality Name (eg, city) +0.organizationName = Organization Name (eg, company) +organizationalUnitName = Organizational Unit Name (eg, section) +commonName = Common Name (eg, fully qualified host name) +commonName_max = 64 +emailAddress = Email Address +emailAddress_max = 64 [ req_attributes ] -challengePassword = A challenge password -challengePassword_min = 4 -challengePassword_max = 20 +challengePassword = A challenge password +challengePassword_min = 4 +challengePassword_max = 20 [ v3_ca ] basicConstraints = CA:TRUE @@ -50,7 +51,8 @@ basicConstraints = CA:TRUE ```bash openssl req -new -key ca.key -nodes -out ca.csr -config openssl.conf -openssl x509 -req -days 36500 -extfile openssl.conf -extensions v3_ca -in ca.csr -signkey ca.key -out ca.crt +openssl x509 -req -days 36500 -extfile openssl.conf \ + -extensions v3_ca -in ca.csr -signkey ca.key -out ca.crt ``` ### 第二步：配置 dfget daemon @@ -77,10 +79,10 @@ proxy: ### 第三步：配置 containerd -在 `/etc/systemd/system/containerd.service.d/http-proxy.conf` 设置 dfdaemon 为 docker daemon 的 -`HTTP_PROXY` 和 `HTTPS_PROXY` 代理： +在 `/etc/systemd/system/containerd.service.d/http-proxy.conf` +设置 dfdaemon 为 docker daemon 的 `HTTP_PROXY` 和 `HTTPS_PROXY` 代理： -``` +```toml [Service] Environment="HTTP_PROXY=http://127.0.0.1:65001" Environment="HTTPS_PROXY=http://127.0.0.1:65001" @@ -142,6 +144,7 @@ proxy: 您能使用以下命令获取您服务器的证书： -``` -openssl x509 -in <(openssl s_client -showcerts -servername xxx -connect xxx:443 -prexit 2>/dev/null) +```bash +openssl x509 -in <(openssl s_client -showcerts \ + -servername your.domain.com -connect your.domain.com:443 -prexit 2>/dev/null) ``` diff --git a/docs/zh-CN/runtime-integration/cri-o.md b/docs/zh-CN/runtime-integration/cri-o.md index cf10e514f95..8ecd178ba5e 100644 --- a/docs/zh-CN/runtime-integration/cri-o.md +++ b/docs/zh-CN/runtime-integration/cri-o.md @@ -33,17 +33,18 @@ location = "docker.io" ## Step 3: 重启 CRI-O -``` +```shell systemctl restart crio ``` -如果遇到如下错误 `mixing sysregistry v1/v2 is not supported` 或 `registry must be in v2 format but is in v1`, 请将您的镜像仓库配置为 v2。 +如果遇到如下错误 `mixing sysregistry v1/v2 is not supported` 或 +`registry must be in v2 format but is in v1`, 请将您的镜像仓库配置为 v2。 ## Step 4: 拉取镜像 使用以下命令拉取镜像: -``` +```shell crictl pull docker.io/library/busybox ``` @@ -51,12 +52,15 @@ crictl pull docker.io/library/busybox 可以查看日志，判断 busybox 镜像正常拉取。 -```bash +```shell grep 'register peer task result' /var/log/dragonfly/daemon/*.log ``` 如果正常日志输出如下: -``` -{"level":"info","ts":"2021-02-23 20:03:20.306","caller":"client/client.go:83","msg":"register peer task result:true[200] for taskId:adf62a86f001e17037eedeaaba3393f3519b80ce,peerIp:10.15.233.91,securityDomain:,idc:,scheduler:127.0.0.1:8002","peerId":"10.15.233.91-65000-43096-1614081800301788000","errMsg":null} +```shell +{"level":"info","ts":"2021-02-23 20:03:20.306","caller":"client/client.go:83", +"msg":"register peer task result:true[200] for taskId:adf62a86f001e17037eedeaaba3393f3519b80ce, +peerIp:10.15.233.91,securityDomain:,idc:,scheduler:127.0.0.1:8002", +"peerId":"10.15.233.91-65000-43096-1614081800301788000","errMsg":null} ``` diff --git a/docs/zh-CN/runtime-integration/docker.md b/docs/zh-CN/runtime-integration/docker.md index 7297669f138..bb4f9b5e502 100644 --- a/docs/zh-CN/runtime-integration/docker.md +++ b/docs/zh-CN/runtime-integration/docker.md @@ -16,29 +16,29 @@ openssl genrsa -out ca.key 2048 ```text [ req ] -#default_bits = 2048 -#default_md = sha256 -#default_keyfile = privkey.pem -distinguished_name = req_distinguished_name -attributes = req_attributes +#default_bits = 2048 +#default_md = sha256 +#default_keyfile = privkey.pem +distinguished_name = req_distinguished_name +attributes = req_attributes extensions = v3_ca req_extensions = v3_ca [ req_distinguished_name ] -countryName = Country Name (2 letter code) -countryName_min = 2 -countryName_max = 2 -stateOrProvinceName = State or Province Name (full name) -localityName = Locality Name (eg, city) -0.organizationName = Organization Name (eg, company) -organizationalUnitName = Organizational Unit Name (eg, section) -commonName = Common Name (eg, fully qualified host name) -commonName_max = 64 -emailAddress = Email Address -emailAddress_max = 64 +countryName = Country Name (2 letter code) +countryName_min = 2 +countryName_max = 2 +stateOrProvinceName = State or Province Name (full name) +localityName = Locality Name (eg, city) +0.organizationName = Organization Name (eg, company) +organizationalUnitName = Organizational Unit Name (eg, section) +commonName = Common Name (eg, fully qualified host name) +commonName_max = 64 +emailAddress = Email Address +emailAddress_max = 64 [ req_attributes ] -challengePassword = A challenge password -challengePassword_min = 4 -challengePassword_max = 20 +challengePassword = A challenge password +challengePassword_min = 4 +challengePassword_max = 20 [ v3_ca ] basicConstraints = CA:TRUE ``` @@ -47,7 +47,8 @@ basicConstraints = CA:TRUE ```bash openssl req -new -key ca.key -nodes -out ca.csr -config openssl.conf -openssl x509 -req -days 36500 -extfile openssl.conf -extensions v3_ca -in ca.csr -signkey ca.key -out ca.crt +openssl x509 -req -days 36500 -extfile openssl.conf \ + -extensions v3_ca -in ca.csr -signkey ca.key -out ca.crt ``` ### 第二步：配置 dfget daemon @@ -74,7 +75,7 @@ proxy: ### 第三步：配置 Docker daemon -为了忽略您的证书错误，您需要在 +为了忽略您的证书错误，您需要在 `/etc/docker/daemon.json` 中把 `insecure-registries` 设置为您的私有代理： ```json @@ -85,10 +86,10 @@ proxy: ### 第四步：继续配置 Docker daemon -在 `/etc/systemd/system/docker.service.d/http-proxy.conf` 设置 dfdaemon 为 docker daemon 的 - `HTTP_PROXY` 和 `HTTPS_PROXY` 代理： +在 `/etc/systemd/system/docker.service.d/http-proxy.conf` 设置 dfdaemon 为 +docker daemon 的 `HTTP_PROXY` 和 `HTTPS_PROXY` 代理： -``` +```toml [Service] Environment="HTTP_PROXY=http://127.0.0.1:65001" Environment="HTTPS_PROXY=http://127.0.0.1:65001" @@ -150,6 +151,7 @@ proxy: 您能使用以下命令获取您服务器的证书： -``` -openssl x509 -in <(openssl s_client -showcerts -servername xxx -connect xxx:443 -prexit 2>/dev/null) +```shell +openssl x509 -in <(openssl s_client -showcerts \ + -servername your.domain.com -connect your.domain.com:443 -prexit 2>/dev/null) ``` diff --git a/docs/zh-CN/test-guide/test-guide.md b/docs/zh-CN/test-guide/test-guide.md index ac794b74b24..908677cba20 100644 --- a/docs/zh-CN/test-guide/test-guide.md +++ b/docs/zh-CN/test-guide/test-guide.md @@ -9,13 +9,13 @@ Dragonfly 包括单元测试和 E2E 测试。 ### 运行单元测试 ```bash -$ make test +make test ``` ### 运行单元测试并产出测试报告 ```bash -$ make test-coverage +make test-coverage ``` ## E2E 测试 @@ -25,17 +25,17 @@ E2E 测试代码在 `test/e2e` 目录中。 ### 运行 E2E 测试 ```bash -$ make e2e-test +make e2e-test ``` ### 运行 E2E 测试并产出测试报告 ```bash -$ make e2e-test-coverage +make e2e-test-coverage ``` ### 清理 E2E 测试 ```bash -$ make clean-e2e-test +make clean-e2e-test ``` diff --git a/docs/zh-CN/troubleshooting/README.md b/docs/zh-CN/troubleshooting/README.md index 4eccce71a2e..18f37673b51 100644 --- a/docs/zh-CN/troubleshooting/README.md +++ b/docs/zh-CN/troubleshooting/README.md @@ -4,27 +4,29 @@ 1. 确认限速值是否合适 [dfget.yaml](https://github.com/dragonflyoss/Dragonfly2/blob/main/docs/zh-CN/config/dfget.yaml#L61) -```yaml -download: - # 总下载限速 - totalRateLimit: 200Mi - # 单个任务下载限速 - perPeerRateLimit: 100Mi # 为了兼容极限环境下，默认值为 20Mi，可以按需调整 -upload: - # 上传限速 - rateLimit: 100Mi -``` + ```yaml + download: + # 总下载限速 + totalRateLimit: 200Mi + # 单个任务下载限速 + perPeerRateLimit: 100Mi # 为了兼容极限环境下，默认值为 20Mi，可以按需调整 + upload: + # 上传限速 + rateLimit: 100Mi + ``` 2. 确认回源速度是否正常 ## 500 Internal Server Error 1. 检查日志 /var/log/dragonfly/daemon/core.log + 2. 检查源站可连接行(DNS 错误 or 证书) -示例: -```shell -curl https://example.harbor.local/ -``` + 示例: + + ```shell + curl https://example.harbor.local/ + ``` -如果`curl`有报错，请查看具体错误 + 如果`curl`有报错，请查看具体错误 diff --git a/test/README.md b/test/README.md index e25bea1feb6..02925426768 100644 --- a/test/README.md +++ b/test/README.md @@ -9,13 +9,13 @@ Unit tests is in the project directory. ### Running unit tests ```bash -$ make test +make test ``` ### Runing uint tests with coverage reports ```bash -$ make test-coverage +make test-coverage ``` ## E2E tests @@ -25,17 +25,17 @@ E2E tests is in `test/e2e` path. ### Running E2E tests ```bash -$ make e2e-test +make e2e-test ``` ### Runing E2E tests with coverage reports ```bash -$ make e2e-test-coverage +make e2e-test-coverage ``` ### Clean E2E tests environment ```bash -$ make clean-e2e-test +make clean-e2e-test ``` diff --git a/test/tools/stress/README.md b/test/tools/stress/README.md index d95273574d3..b7b9579f94a 100644 --- a/test/tools/stress/README.md +++ b/test/tools/stress/README.md @@ -5,46 +5,49 @@ ### Build and Run 1. Build tool: -```shell -go build -o bin/stress test/stress/main.go -``` + + ```shell + go build -o bin/stress test/stress/main.go + ``` 2. Run stress: -```shell -bin/stress -connections 100 -duration 1s -proxy http://127.0.0.1:65001 \ - --url http://localhost/misc/d7y-test/blobs/sha256/128K -``` -Example output: -``` -Latency - avg 17.286522ms - min 617.801µs - max 84.201941ms -Latency Distribution - 50% 11.39049ms - 75% 18.308966ms - 90% 49.052485ms - 95% 55.886513ms - 99% 65.013042ms -HTTP codes - 200 5849 -Throughput 731.1MB -Request 5849/s -``` + ```shell + bin/stress -connections 100 -duration 1s -proxy http://127.0.0.1:65001 \ + --url http://localhost/misc/d7y-test/blobs/sha256/128K + ``` + + Example output: + + ```text + Latency + avg 17.286522ms + min 617.801µs + max 84.201941ms + Latency Distribution + 50% 11.39049ms + 75% 18.308966ms + 90% 49.052485ms + 95% 55.886513ms + 99% 65.013042ms + HTTP codes + 200 5849 + Throughput 731.1MB + Request 5849/s + ``` ### CLI Reference -``` +```text Usage of ./stress: -connections int - concurrency count of connections (default 100) + concurrency count of connections (default 100) -duration duration - testing duration (default 1m40s) + testing duration (default 1m40s) -output string - all request statistics (default "/tmp/statistics.txt") + all request statistics (default "/tmp/statistics.txt") -proxy string - target proxy for downloading, example: http://127.0.0.1:65001 + target proxy for downloading, example: http://127.0.0.1:65001 -url string - target url for stress testing, example: http://localhost + target url for stress testing, example: http://localhost ```