diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index 61f2878f5c2..d627844235e 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -22,3 +22,9 @@ jobs: uses: golangci/golangci-lint-action@v2 with: version: v1.40.1 + + - name: Markdown lint + uses: docker://avtodev/markdown-lint:v1 + with: + args: '**/*.md' + ignore: 'deploy/helm-charts manager/console CHANGELOG.md docs/en/api-reference docs/zh-CN/api-reference docs/en/cli-reference/dfget.1.md' diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md index b2f50cf3e8d..6adeea4b86a 100644 --- a/CODE_OF_CONDUCT.md +++ b/CODE_OF_CONDUCT.md @@ -2,7 +2,15 @@ ## Our Pledge -In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation. +In the interest of fostering an open and welcoming environment, +we as contributors and maintainers pledge to +making participation in our project and +our community a harassment-free experience for everyone, +regardless of age, body size, +disability, ethnicity, gender identity and expression, +level of experience, nationality, +personal appearance, race, religion, +or sexual identity and orientation. ## Our Standards @@ -19,29 +27,50 @@ Examples of unacceptable behavior by participants include: * The use of sexualized language or imagery and unwelcome sexual attention or advances * Trolling, insulting/derogatory comments, and personal or political attacks * Public or private harassment -* Publishing others' private information, such as a physical or electronic address, without explicit permission -* Other conduct which could reasonably be considered inappropriate in a professional setting +* Publishing others' private information, + such as a physical or electronic address, without explicit permission +* Other conduct which could reasonably be considered inappropriate in + a professional setting ## Our Responsibilities -Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior. +Project maintainers are responsible for clarifying the standards of +acceptable behavior and are expected to take appropriate and +fair corrective action in response to any instances of unacceptable behavior. -Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful. +Project maintainers have the right and responsibility to remove, +edit, or reject comments, commits, code, wiki edits, issues, +and other contributions that are not aligned to this Code of Conduct, +or to ban temporarily or permanently any contributor for +other behaviors that they deem inappropriate, threatening, offensive, or harmful. ## Scope -This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers. +This Code of Conduct applies both within project spaces and in public +spaces when an individual is representing the project or its community. +Examples of representing a project or community include using an official +project e-mail address, posting via an official social media account, +or acting as an appointed representative at an online or offline event. +Representation of a project may be further defined and clarified by project maintainers. ## Enforcement -Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at . The project team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately. +Instances of abusive, harassing, or otherwise unacceptable +behavior may be reported by contacting the project team at +. The project team will review +and investigate all complaints, and will respond in a way +that it deems appropriate to the circumstances. The project team is +obligated to maintain confidentiality with regard to the reporter of an incident. +Further details of specific enforcement policies may be posted separately. -Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership. +Project maintainers who do not follow or enforce the Code of +Conduct in good faith may face temporary or permanent repercussions as +determined by other members of the project's leadership. ## Attribution -This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [https://www.contributor-covenant.org/version/2/0/][version] +This Code of Conduct is adapted from the [Contributor Covenant][homepage], +version 1.4, available at [https://www.contributor-covenant.org/version/2/0/][version] [homepage]: https://contributor-covenant.org [version]: https://www.contributor-covenant.org/version/2/0/ - diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d7bf455752c..256b65acbf7 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,6 +1,8 @@ # Contributing to Dragonfly -It is warmly welcomed if you have interest to hack on Dragonfly. First, we encourage this kind of willing very much. And here is a list of contributing guide for you. +It is warmly welcomed if you have interest to hack on Dragonfly. +First, we encourage this kind of willing very much. +And here is a list of contributing guide for you. ## Topics @@ -12,15 +14,32 @@ It is warmly welcomed if you have interest to hack on Dragonfly. First, we encou ## Reporting security issues -Security issues are always treated seriously. As our usual principle, we discourage anyone to spread security issues. If you find a security issue of Dragonfly, please do not discuss it in public and even do not open a public issue. Instead we encourage you to send us a private email to [Dragonfly@noreply.github.com](mailto:Dragonfly@noreply.github.com) to report this. +Security issues are always treated seriously. +As our usual principle, we discourage anyone to spread security issues. +If you find a security issue of Dragonfly, please do not discuss it in +public and even do not open a public issue. +Instead we encourage you to send us a private email to +[Dragonfly@noreply.github.com](mailto:Dragonfly@noreply.github.com) to report this. ## Reporting general issues -To be honest, we regard every user of Dragonfly as a very kind contributor. After experiencing Dragonfly, you may have some feedback for the project. Then feel free to open an issue via [NEW ISSUE](https://github.com/dragonflyoss/Dragonfly2/issues/new). +To be honest, we regard every user of Dragonfly as a very kind contributor. +After experiencing Dragonfly, you may have some feedback for the project. +Then feel free to open an issue via +[NEW ISSUE](https://github.com/dragonflyoss/Dragonfly2/issues/new). -Since we collaborate project Dragonfly in a distributed way, we appreciate **WELL-WRITTEN**, **DETAILED**, **EXPLICIT** issue reports. To make the communication more efficient, we wish everyone could search if your issue is an existing one in the searching list. If you find it existing, please add your details in comments under the existing issue instead of opening a brand new one. +Since we collaborate project Dragonfly in a distributed way, +we appreciate **WELL-WRITTEN**, **DETAILED**, **EXPLICIT** issue reports. +To make the communication more efficient, we wish everyone could +search if your issue is an existing one in the searching list. +If you find it existing, please add your details in comments +under the existing issue instead of opening a brand new one. -To make the issue details as standard as possible, we setup an [ISSUE TEMPLATE](./.github/ISSUE_TEMPLATE) for issue reporters. You can find three kinds of issue templates there: question, bug report and feature request. Please **BE SURE** to follow the instructions to fill fields in template. +To make the issue details as standard as possible, +we setup an [ISSUE TEMPLATE](./.github/ISSUE_TEMPLATE) for issue reporters. +You can find three kinds of issue templates there: question, +bug report and feature request. Please **BE SURE** to follow +the instructions to fill fields in template. There are lot of cases when you could open an issue: @@ -35,11 +54,16 @@ There are lot of cases when you could open an issue: * any questions on project * and so on -Also we must remind that when filing a new issue, please remember to remove the sensitive data from your post. Sensitive data could be password, secret key, network locations, private business data and so on. +Also we must remind that when filing a new issue, +please remember to remove the sensitive data from your post. +Sensitive data could be password, secret key, +network locations, private business data and so on. ## Code and doc contribution -Every action to make project Dragonfly better is encouraged. On GitHub, every improvement for Dragonfly could be via a PR (short for pull request). +Every action to make project Dragonfly better is encouraged. +On GitHub, every improvement for Dragonfly could be +via a PR (short for pull request). * If you find a typo, try to fix it! * If you find a bug, try to fix it! @@ -52,11 +76,13 @@ Every action to make project Dragonfly better is encouraged. On GitHub, every im * If you find document incorrect, just do it and fix that! * ... -Actually it is impossible to list them completely. Just remember one princinple: +Actually it is impossible to list them completely. +Just remember one princinple: > WE ARE LOOKING FORWARD TO ANY PR FROM YOU. -Since you are ready to improve Dragonfly with a PR, we suggest you could take a look at the PR rules here. +Since you are ready to improve Dragonfly with a PR, +we suggest you could take a look at the PR rules here. * [Workspace Preparation](#workspace-preparation) * [Branch Definition](#branch-definition) @@ -67,61 +93,87 @@ Since you are ready to improve Dragonfly with a PR, we suggest you could take a ### Workspace Preparation -To put forward a PR, we assume you have registered a GitHub ID. Then you could finish the preparation in the following steps: +To put forward a PR, we assume you have registered +a GitHub ID. Then you could finish +the preparation in the following steps: -1. **FORK** Dragonfly to your repository. To make this work, you just need to click the button Fork in right-left of [dragonflyoss/Dragonfly2](https://github.com/dragonflyoss/Dragonfly2) main page. Then you will end up with your repository in `https://github.com//Dragonfly`, in which `your-username` is your GitHub username. +1. **FORK** Dragonfly to your repository. +To make this work, you just need to click the button Fork +in right-left of [dragonflyoss/Dragonfly2](https://github.com/dragonflyoss/Dragonfly2) +main page. Then you will end up with your repository in +`https://github.com//Dragonfly`, +in which `your-username` is your GitHub username. -2. **CLONE** your own repository to develop locally. Use `git clone https://github.com//Dragonfly2.git` to clone repository to your local machine. Then you can create new branches to finish the change you wish to make. +2. **CLONE** your own repository to develop locally. +Use `git clone https://github.com//Dragonfly2.git` +to clone repository to your local machine. +Then you can create new branches to finish the change you wish to make. -3. **Set Remote** upstream to be `https://github.com/dragonflyoss/Dragonfly2.git` using the following two commands: +3. **Set Remote** upstream to be +`https://github.com/dragonflyoss/Dragonfly2.git` - ``` - git remote add upstream https://github.com/dragonflyoss/Dragonfly2.git - git remote set-url --push upstream no-pushing - ``` + using the following two commands: - With this remote setting, you can check your git remote configuration like this: + ```bash + git remote add upstream https://github.com/dragonflyoss/Dragonfly2.git + git remote set-url --push upstream no-pushing + ``` - ``` - $ git remote -v - origin https://github.com//Dragonfly2.git (fetch) - origin https://github.com//Dragonfly2.git (push) - upstream https://github.com/dragonflyoss/Dragonfly2.git (fetch) - upstream no-pushing (push) - ``` + With this remote setting, you can check your git remote configuration like this: - Adding this, we can easily synchronize local branches with upstream branches. + ```bash + $ git remote -v + origin https://github.com//Dragonfly2.git (fetch) + origin https://github.com//Dragonfly2.git (push) + upstream https://github.com/dragonflyoss/Dragonfly2.git (fetch) + upstream no-pushing (push) + ``` + + Adding this, we can easily synchronize local branches with upstream branches. 4. **Create a branch** to add a new feature or fix issues - Update local working directory: + Update local working directory: - ``` - cd Dragonfly2 - git fetch upstream - git checkout main - git rebase upstream/main - ``` + ```bash + cd Dragonfly2 + git fetch upstream + git checkout main + git rebase upstream/main + ``` - Create a new branch: + Create a new branch: - ``` - git checkout -b - ``` + ```bash + git checkout -b + ``` - Make any change on the `new-branch` then build and test your codes. + Make any change on the `new-branch` then build and test your codes. ### Branch Definition -Right now we assume every contribution via pull request is for [branch main](https://github.com/dragonflyoss/Dragonfly2/tree/main) in Dragonfly. Before contributing, be aware of branch definition would help a lot. +Right now we assume every contribution via pull +request is for [branch main](https://github.com/dragonflyoss/Dragonfly2/tree/main) +in Dragonfly. Before contributing, +be aware of branch definition would help a lot. -As a contributor, keep in mind again that every contribution via pull request is for branch main. While in project Dragonfly, there are several other branches, we generally call them rc branches, release branches and backport branches. +As a contributor, keep in mind again that every +contribution via pull request is for branch main. +While in project Dragonfly, there are several other branches, +we generally call them rc branches, release branches and backport branches. -Before officially releasing a version, we will checkout a rc(release candidate) branch. In this branch, we will test more than branch main. +Before officially releasing a version, +we will checkout a rc(release candidate) branch. +In this branch, we will test more than branch main. -When officially releasing a version, there will be a release branch before tagging. After tagging, we will delete the release branch. +When officially releasing a version, +there will be a release branch before tagging. +After tagging, we will delete the release branch. -When backporting some fixes to existing released version, we will checkout backport branches. After backporting, the backporting effects will be in PATCH number in MAJOR.MINOR.PATCH of [SemVer](http://semver.org/). +When backporting some fixes to existing released version, +we will checkout backport branches. After backporting, +the backporting effects will be in PATCH number in +MAJOR.MINOR.PATCH of [SemVer](http://semver.org/). ### Commit Rules @@ -132,7 +184,12 @@ Actually in Dragonfly, we take two rules serious when committing: #### Commit Message -Commit message could help reviewers better understand what the purpose of submitted PR is. It could help accelerate the code review procedure as well. We encourage contributors to use **EXPLICIT** commit message rather than ambiguous message. In general, we advocate the following commit message type: +Commit message could help reviewers better understand +what the purpose of submitted PR is. +It could help accelerate the code review procedure as well. +We encourage contributors to use **EXPLICIT** commit +message rather than ambiguous message. In general, +we advocate the following commit message type: * feat: xxxx.For example, "feat: make result show in sorted order". * fix: xxxx. For example, "fix: fix panic when input nil parameter". @@ -140,9 +197,11 @@ Commit message could help reviewers better understand what the purpose of submit * style: xxxx. For example, "style: format the code style of Constants.java". * refactor: xxxx. For example, "refactor: simplify to make codes more readable". * test: xxx. For example, "test: add unit test case for func InsertIntoArray". -* chore: xxx. For example, "chore: integrate travis-ci". It's the type of mantainance change. +* chore: xxx. For example, "chore: integrate travis-ci". +It's the type of mantainance change. -On the other side, we discourage contributors from committing message like the following ways: +On the other side, we discourage contributors +from committing message like the following ways: * ~~fix bug~~ * ~~update~~ @@ -150,20 +209,32 @@ On the other side, we discourage contributors from committing message like the f #### Commit Content -Commit content represents all content changes included in one commit. We had better include things in one single commit which could support reviewer's complete review without any other commits' help. In another word, contents in one single commit can pass the CI to avoid code mess. In brief, there are two minor rules for us to keep in mind: +Commit content represents all content changes +included in one commit. We had better include things +in one single commit which could support reviewer's complete +review without any other commits' help. In another word, +contents in one single commit can pass the CI to avoid code mess. +In brief, there are two minor rules for us to keep in mind: * avoid very large change in a commit. * complete and reviewable for each commit. -No matter what the commit message, or commit content is, we do take more emphasis on code review. +No matter what the commit message, or commit content is, +we do take more emphasis on code review. ### PR Description -PR is the only way to make change to Dragonfly project files. To help reviewers better get your purpose, PR description could not be too detailed. We encourage contributors to follow the [PR template](./.github/PULL_REQUEST_TEMPLATE.md) to finish the pull request. +PR is the only way to make change to Dragonfly project files. +To help reviewers better get your purpose, +PR description could not be too detailed. +We encourage contributors to follow the +[PR template](./.github/PULL_REQUEST_TEMPLATE.md) to finish the pull request. ### Developing Environment -As a contributor, if you want to make any contribution to Dragonfly project, we should reach an agreement on the version of tools used in the development environment. +As a contributor, if you want to make any contribution to +Dragonfly project, we should reach an agreement on +the version of tools used in the development environment. Here are some dependents with specific version: * golang : v1.15 @@ -171,11 +242,19 @@ Here are some dependents with specific version: * shellCheck : latest * docker: latest -When you develop the Dragonfly project at the local environment, you should use subcommands of Makefile to help yourself to check and build the latest version of Dragonfly. For the convenience of developers, we use the docker to build Dragonfly. It can reduce problems of the developing environment. +When you develop the Dragonfly project at the local environment, +you should use subcommands of Makefile to help yourself +to check and build the latest version of Dragonfly. +For the convenience of developers, +we use the docker to build Dragonfly. +It can reduce problems of the developing environment. ### Golang Dependency Management -The Dragonfly project uses [Go modules](https://golang.org/cmd/go/#hdr-Modules__module_versions__and_more) to manage dependencies on external packages. This requires a working Go environment with version 1.15 or greater installed. +The Dragonfly project uses +[Go modules](https://golang.org/cmd/go/#hdr-Modules__module_versions__and_more) +to manage dependencies on external packages. +This requires a working Go environment with version 1.15 or greater installed. To add or update a new dependency, use the `go get` command: @@ -193,7 +272,8 @@ Tidy up the `go.mod` and `go.sum` files: go mod tidy ``` -You have to commit the changes to `go.mod`, `go.sum` before submitting the pull request. +You have to commit the changes to `go.mod`, +`go.sum` before submitting the pull request. ```bash git add go.mod go.sum @@ -202,7 +282,10 @@ git commit ## Engage to help anything -We choose GitHub as the primary place for Dragonfly to collaborate. So the latest updates of Dragonfly are always here. Although contributions via PR is an explicit way to help, we still call for any other ways. +We choose GitHub as the primary place for Dragonfly to +collaborate. So the latest updates of Dragonfly are always here. +Although contributions via PR is an explicit way to help, +we still call for any other ways. * Reply to other's issues if you could. * Help solve other user's problems. @@ -216,7 +299,8 @@ In a word, **ANY HELP IS CONTRIBUTION.** ## Join Dragonfly as a member -It is also welcomed to join Dragonfly team if you are willing to participate in Dragonfly community continuously and keep active. +It is also welcomed to join Dragonfly team if you are +willing to participate in Dragonfly community continuously and keep active. ### Requirements @@ -233,5 +317,6 @@ It is also welcomed to join Dragonfly team if you are willing to participate in You can do it in either of two ways: -* Submit a PR in the [dragonflyoss/Dragonfly2](https://github.com/dragonflyoss/Dragonfly2) repo. +* Submit a PR in the +[dragonflyoss/Dragonfly2](https://github.com/dragonflyoss/Dragonfly2) repo. * Contact with the community's [maintainers](MAINTAINERS.md) offline. diff --git a/README.md b/README.md index 72cbd75fb1d..63dd22765e5 100644 --- a/README.md +++ b/README.md @@ -14,46 +14,76 @@ [![GoDoc](https://godoc.org/github.com/dragonflyoss/Dragonfly2?status.svg)](https://godoc.org/github.com/dragonflyoss/Dragonfly2) [![LICENSE](https://img.shields.io/github/license/dragonflyoss/Dragonfly2.svg?style=flat-square)](https://github.com/dragonflyoss/Dragonfly2/blob/main/LICENSE) -Provide efficient, stable, secure, low-cost file and image distribution services to be the best practice and standard solution in the related Cloud-Native area. +Provide efficient, stable, secure, low-cost file and +image distribution services to be the best practice and +standard solution in the related Cloud-Native area. ## Introduction -Dragonfly is an open source intelligent P2P based image and file distribution system. Its goal is to tackle all distribution problems in cloud native scenarios. Currently Dragonfly focuses on being: +Dragonfly is an open source intelligent P2P based image and +file distribution system. Its goal is to +tackle all distribution problems in cloud native scenarios. +Currently Dragonfly focuses on being: - Simple: well-defined user-facing API (HTTP), non-invasive to all container engines; - Efficient: CDN support, P2P based file distribution to save enterprise bandwidth; - Intelligent: host level speed limit, intelligent flow control due to host detection; - Secure: block transmission encryption, HTTPS connection support. -Dragonfly is now hosted by the Cloud Native Computing Foundation (CNCF) as an Incubating Level Project. Originally it was born to solve all kinds of distribution at very large scales, such as application distribution, cache distribution, log distribution, image distribution, and so on. +Dragonfly is now hosted by the Cloud Native Computing Foundation (CNCF) as +an Incubating Level Project. Originally it was born to +solve all kinds of distribution at very large scales, +such as application distribution, cache distribution, +log distribution, image distribution, and so on. ## Features -- Implement P2P files distribution with various storage types (HDFS, storage services from various cloud vendors, Maven, Yum, etc.) through a unified back-to-source adapter layer. -- Support more distribution modes: active pull, active push, real-time synchronization, remote replication, automatic warm-up, cross-cloud transmission, etc. -- Provide separation and decoupling between systems, scheduling and plug-in CDN. Support on-demand deployment with flexible types: light or heavy, inside or outside, to meet the actual needs of different scenarios. -- Newly designed P2P protocol framework based on GRPC with improved efficiency and stability. -- Perform encrypted transmission, account-based transmission authentication and rate limit, and multi-tenant isolation mechanism. +- Implement P2P files distribution with various storage types +(HDFS, storage services from various cloud vendors, Maven, Yum, etc.) +through a unified back-to-source adapter layer. +- Support more distribution modes: +active pull, active push, real-time synchronization, +remote replication, automatic warm-up, +cross-cloud transmission, etc. +- Provide separation and decoupling between systems, +scheduling and plug-in CDN. Support on-demand deployment with +flexible types: light or heavy, inside or outside, +to meet the actual needs of different scenarios. +- Newly designed P2P protocol framework based on +GRPC with improved efficiency and stability. +- Perform encrypted transmission, account-based transmission +authentication and rate limit, and multi-tenant isolation mechanism. - Bear more efficient IO methods: multithreaded IO, memory mapping, DMA, etc. -- Advocate dynamic compression, in-memory file systems, and more efficient scheduling algorithms to improve distribution efficiency. -- Client allows third-party software to natively integrate Dragonfly’s P2P capabilities through C/S mode. -- Productivity: Support file uploading, task management of various distribution modes, data visualization, global control, etc. -- Consistent internal and external versions, shared core features, and individual extensions of non-generic features. -- Enhanced integration with ecology: Harbor, Nydus (on-demand image download), warehouse services for various cloud vendors, etc. +- Advocate dynamic compression, in-memory file systems, +and more efficient scheduling algorithms to improve distribution efficiency. +- Client allows third-party software to +natively integrate Dragonfly’s P2P capabilities through C/S mode. +- Productivity: Support file uploading, task management of +various distribution modes, data visualization, global control, etc. +- Consistent internal and external versions, shared core features, +and individual extensions of non-generic features. +- Enhanced integration with ecology: +Harbor, Nydus (on-demand image download), +warehouse services for various cloud vendors, etc. ## Architecture ![alt][arch] -**Manager:** Used to manage the dynamic configuration that each module depends on, and provide keepalive and metrics functions. +**Manager:** Used to manage the dynamic configuration that +each module depends on, and provide keepalive and metrics functions. -**Scheduler:** The tracker and scheduler in the P2P network that choose appropriate downloading net-path for each peer. +**Scheduler:** The tracker and scheduler in the P2P +network that choose appropriate downloading net-path for each peer. -**CDN:** A CDN server that caches downloaded data from source to avoid downloading same files repeatedly. +**CDN:** A CDN server that caches downloaded data +from source to avoid downloading same files repeatedly. -**Daemon:** It's a daemon of dfget client. It establishes a proxy between containerd/CRI-O and registry. +**Daemon:** It's a daemon of dfget client. +It establishes a proxy between containerd/CRI-O and registry. -**Dfget:** The client of Dragonfly used for downloading files. It's similar to wget. +**Dfget:** The client of Dragonfly used for +downloading files. It's similar to wget. ## Documentation @@ -61,20 +91,27 @@ You can find the full documentation [on the repo][document]. ## Community -Welcome developers to actively participate in community discussions and contribute code to Dragonfly. We will remain concerned about the issues discussed in the community and respond quickly. +Welcome developers to actively participate in community discussions +and contribute code to Dragonfly. We will remain +concerned about the issues discussed in the community and respond quickly. - Discussions: [Github Discussion Forum][discussion] - Twitter: [@dragonfly_oss](https://twitter.com/dragonfly_oss) - DingTalk: 23304666 +
- +
+ ## Contributing -You should check out our [CONTRIBUTING][contributing] and develop the project together. + +You should check out our +[CONTRIBUTING][contributing] and develop the project together. ## Code of Conduct + Please refer to our [Code of Conduct][codeconduct]. [arch]: docs/en/images/arch.png @@ -84,3 +121,4 @@ Please refer to our [Code of Conduct][codeconduct]. [contributing]: CONTRIBUTING.md [codeconduct]: CODE_OF_CONDUCT.md [document]: docs/README.md +[dingtalk]: docs/en/images/community/dingtalk-group.jpeg diff --git a/api/README.md b/api/README.md index cbd5c2c634d..b0131151f9d 100644 --- a/api/README.md +++ b/api/README.md @@ -1,18 +1,25 @@ # Dragonfly V2 API -We encourage users to experience Dragonfly in different ways. When doing this, there are three ways users could choose to interact with dragonfly mostly: +We encourage users to experience Dragonfly in different ways. +When doing this, there are three ways users +could choose to interact with dragonfly mostly: * For end-users, command line tool `dfget`, `dfdaemon` is mostly used. -* For developers, Dragonfly manager's raw API is the original thing they would make use of. For more details about API docs, please refer to - [api.md](./manager/swagger.json). We should also keep it in mind that doc [apis.md](./manager/swagger.json) is - automatically - generated by - [swagger2markup](https://github.com/Swagger2Markup/swagger2markup). Please **DO NOT** edit [api.md](./manager/swagger.json) directly. +* For developers, Dragonfly manager's raw API is + the original thing they would make use of. + For more details about API docs, please refer to [api.md](./manager/swagger.json). + We should also keep it in mind that doc + [apis.md](./manager/swagger.json) is automatically generated by + [swagger2markup](https://github.com/Swagger2Markup/swagger2markup). + Please **DO NOT** edit [api.md](./manager/swagger.json) directly. Directory `/api` mainly describes the second part **Dragonfly Manager's Raw API**. -Currently, the description of swagger format is provided, which can be viewed in `manager`. It contains two format files of swagger, JSON and yaml. You can -browse through relevant viewing tools to obtain the specific call method of API, or view online through the `/swagger/doc.json` endpoint of manager service. - -Through manager's API, you can integrate with your own services, so you can automatically manage dragonfly and improve the level of Devops. +Currently, the description of swagger format is provided, +which can be viewed in `manager`. +It contains two format files of swagger, JSON and yaml. You can +browse through relevant viewing tools to obtain the specific call method of API, +or view online through the `/swagger/doc.json` endpoint of manager service. +Through manager's API, you can integrate with your own services, +so you can automatically manage dragonfly and improve the level of Devops. diff --git a/build/images/base/README.md b/build/images/base/README.md index 0494de04542..fefa2790dfb 100644 --- a/build/images/base/README.md +++ b/build/images/base/README.md @@ -1,6 +1,8 @@ -# Base Image for Debugging +# Base image -## Build Base Image +## Base Image for Debugging + +### Build Base Image ```shell BASE_IMAGE=dragonflyoss/base:bpftrace-v0.13.0-go-v1.16.6 @@ -12,7 +14,7 @@ docker run -ti -v /usr/src:/usr/src:ro \ ${BASE_IMAGE} ``` -## Build Dragonfly Images +### Build Dragonfly Images ```shell # tags for condition compiling @@ -25,9 +27,9 @@ export BASE_IMAGE=dragonflyoss/base:bpftrace-v0.13.0-go-v1.16.6 make docker-build ``` -# Debug With Delve +## Debug With Delve -## Prepare Code for Debug +### Prepare Code for Debug ```shell COMMIT_ID=c1c3d652 @@ -36,15 +38,15 @@ git clone https://github.com/dragonflyoss/Dragonfly2.git /go/src/d7y.io/dragonfl git reset --hard ${COMMIT_ID} ``` -## Debug Operations +### Debug Operations 1. Attach Process -```shell -pid=$(pidof scheduler) # or dfget, cdn, manager -dlv attach $pid -``` + ```shell + pid=$(pidof scheduler) # or dfget, cdn, manager + dlv attach $pid + ``` 2. Debug -Follow https://github.com/go-delve/delve/tree/v1.7.0/Documentation/cli \ No newline at end of file + Follow diff --git a/docs/en/FAQ.md b/docs/en/FAQ.md deleted file mode 100644 index 4514b4c1065..00000000000 --- a/docs/en/FAQ.md +++ /dev/null @@ -1 +0,0 @@ -# FAQ diff --git a/docs/en/README.md b/docs/en/README.md index 074098e337a..ef6a3cc3cb6 100644 --- a/docs/en/README.md +++ b/docs/en/README.md @@ -1,6 +1,11 @@ # Dragonfly Document -Dragonfly Document is written, drawn, memorialized representation of all things about Dragonfly. For those who are generally interested in Dragonfly, README.md of this repo is sufficient. While for end users, all details contained in `/docs` is the best guide all should have. And for developers, contents in [Developer Guide](developer-guide/developer-guide.md) part is that all need. +Dragonfly Document is written, drawn, memorialized representation of +all things about Dragonfly. For those who are generally interested in Dragonfly, +README.md of this repo is sufficient. While for end users, +all details contained in `/docs` is the best guide all should have. +And for developers, contents in [Developer Guide](developer-guide/developer-guide.md) +part is that all need. Organization of document is as following: diff --git a/docs/en/ROADMAP.md b/docs/en/ROADMAP.md deleted file mode 100644 index 437766dd8f3..00000000000 --- a/docs/en/ROADMAP.md +++ /dev/null @@ -1 +0,0 @@ -# Roadmap diff --git a/docs/en/cli-reference/README.md b/docs/en/cli-reference/README.md index 06b65f90d5e..be2d7e7e9b0 100644 --- a/docs/en/cli-reference/README.md +++ b/docs/en/cli-reference/README.md @@ -1,10 +1,16 @@ # CLI Reference -For almost all users, commandline is the first reference you may need. Document in directory `CLI Reference` is about command detailed usage of Dragonfly CLI including `dfget`, `cdn`, `scheduler` and `manager`. You can get introductions, synopsis, examples, options about command. Last but not least, Dragonfly can guarantee commandline docs is strongly consistent with Dragonfly CLI's source code. What's more, all commandline docs are auto generated via source code. +For almost all users, commandline is the first reference you may need. +Document in directory `CLI Reference` is about command detailed usage of +Dragonfly CLI including `dfget`, `cdn`, `scheduler` and `manager`. +You can get introductions, synopsis, examples, options about command. +Last but not least, Dragonfly can guarantee commandline docs is strongly +consistent with Dragonfly CLI's source code. What's more, +all commandline docs are auto generated via source code. Table of contents: * [dfget](dfget.md) * [cdn](cdn.md) * [scheduler](scheduler.md) -* [manager](manager.md) \ No newline at end of file +* [manager](manager.md) diff --git a/docs/en/cli-reference/cdn.md b/docs/en/cli-reference/cdn.md index cab3b441e01..de087741532 100644 --- a/docs/en/cli-reference/cdn.md +++ b/docs/en/cli-reference/cdn.md @@ -1,17 +1,18 @@ # CDN -CDN is a long-running process which caches downloaded data from source to avoid downloading the same files from source repeatedly +CDN is a long-running process which caches downloaded data +from source to avoid downloading the same files from source repeatedly. ## Usage -``` +```shell cdn [flags] cdn [command] ``` -## Available Commands: +## Available Commands -``` +```text doc generate documents help Help about any command version show version @@ -19,7 +20,8 @@ version show version ## Flags -``` + +```text --config string the path of configuration file with yaml extension name, default is /Users/${USER_HOME}/.dragonfly/config/cdn.yaml, it can also be set by environment variable cdn_config --console whether logger output records to the stdout @@ -29,3 +31,4 @@ version show version --service-name string name of the service for tracer (default "dragonfly-cdn") --verbose whether logger use debug level ``` + diff --git a/docs/en/cli-reference/dfget.md b/docs/en/cli-reference/dfget.md index c66b04f2b69..a861b195c78 100644 --- a/docs/en/cli-reference/dfget.md +++ b/docs/en/cli-reference/dfget.md @@ -1,30 +1,37 @@ -# dfget +# dfdaemon + +## dfget `dfget` is the client of Dragonfly used to download and upload files ### Synopsis -dfget is the client of Dragonfly which takes a role of peer in a P2P network. When user triggers a file downloading -task, dfget will download the pieces of file from other peers. Meanwhile, it will act as an uploader to support other -peers to download pieces from it if it owns them. In addition, dfget has the abilities to provide more advanced -functionality, such as network bandwidth limit, transmission encryption and so on. +dfget is the client of Dragonfly which takes +a role of peer in a P2P network. When user triggers a file downloading +task, dfget will download the pieces of +file from other peers. Meanwhile, it will act as an uploader to support other +peers to download pieces from it if it owns them. +In addition, dfget has the abilities to provide more advanced +functionality, such as network bandwidth limit, +transmission encryption and so on. -``` +```shell dfget [flags] ``` ### Example -``` +```shell dfget --schedulers 127.0.0.1:8002 -o /path/to/output -u "http://example.com/object" ``` -## Log configuration +### Log configuration set environment variable console=true if you want to print logs to Terminal ### Options + ``` --alivetime duration alive duration for which uploader keeps no accessing by any uploading requests, after this period uploader will automatically exit (default 5m0s) --cacerts strings the cacert file which is used to verify remote server when supernode interact with the source. @@ -57,23 +64,24 @@ set environment variable console=true if you want to print logs to Terminal --totallimit ratelimit network bandwidth rate limit for the whole host, in format of G(B)/g/M(B)/m/K(B)/k/B, pure number will also be parsed as Byte (default 104857600.000000) -u, --url string URL of user requested downloading file(only HTTP/HTTPs supported) --verbose enable verbose mode, all debug log will be display - ``` + -# dfget daemon +## dfget daemon -## Log configuration +### Daemon Log configuration set environment variable console=true if you want to print logs to Terminal -### Options +### Daemon Options + ``` --advertise-ip string the ip report to scheduler, normal same with listen ip (default "10.15.232.63") - --alivetime duration alive duration for which uploader keeps no accessing by any uploading requests, after this period uploader will automatically exit (default 5m0s) + --alivetime duration alive duration for which uploader keeps no accessing by any uploading requests, after this period uploader will automatically exit (default 5m0s) --data string local directory which stores temporary files for p2p uploading --download-rate ratelimit download rate limit for other peers and back source (default 104857600.000000) - --expiretime duration caching duration for which cached file keeps no accessed by any process, after this period cache file will be deleted (default 3m0s) + --expiretime duration caching duration for which cached file keeps no accessed by any process, after this period cache file will be deleted (default 3m0s) --gc-interval duration gc interval (default 1m0s) --grpc-port int the listen address for grpc with other peers (default 65000) --grpc-port-end int the listen address for grpc with other peers (default 65000) @@ -96,3 +104,4 @@ set environment variable console=true if you want to print logs to Terminal --upload-rate ratelimit upload rate limit for other peers (default 104857600.000000) --verbose print verbose log and enable golang debug info ``` + diff --git a/docs/en/cli-reference/manager.md b/docs/en/cli-reference/manager.md index 7af33b449a0..811e196f1f2 100644 --- a/docs/en/cli-reference/manager.md +++ b/docs/en/cli-reference/manager.md @@ -1,49 +1,52 @@ -## Manager +# Manager -Manager is a process that runs in the background and plays the role of the brain of each subsystem cluster in Dragonfly. It is used to manage the dynamic -configuration of each system module and provide functions such as heartbeat keeping alive, monitoring the market, and product functions. +Manager is a process that runs in the background +and plays the role of the brain of each subsystem cluster in Dragonfly. +It is used to manage the dynamic +configuration of each system module and provide functions +such as heartbeat keeping alive, monitoring the market, and product functions. -### Try it +## Try it -``` +```text manager [Option] ``` -### Log configuration +## Log configuration -``` +```text 1. set option --console if you want to print logs to Terminal 2. log path: /var/log/dragonfly/manager/ ``` -### Runtime metrics monitoring +## Runtime metrics monitoring -``` +```text manager --pprof-port port ``` -### Swagger support +## Swagger support -endpoint: /swagger/doc.json +endpoint: /swagger/doc.json -### Prometheus metrics monitoring +## Prometheus metrics monitoring endpoint: /metrics -### HealthCheck +## HealthCheck endpoint: /healthy/ +## Enable jaeger -### Enable jaeger - -``` +```text manager --jaeger http://localhost:14250/api/traces ``` -### Options +## Options -``` + +```text --config string the path of configuration file with yaml extension name, default is /etc/dragonfly/manager.yaml, it can also be set by env var: MANAGER_CONFIG --console whether logger output records to the stdout -h, --help help for manager @@ -52,6 +55,8 @@ manager --jaeger http://localhost:14250/api/traces --service-name string name of the service for tracer (default "dragonfly-manager") --verbose whether logger use debug level ``` + + +## Manager Config -### Manager Config -[Config Example](../config/manager.yaml) \ No newline at end of file +[Config Example](../config/manager.yaml) diff --git a/docs/en/cli-reference/scheduler.md b/docs/en/cli-reference/scheduler.md index 3ec3c781ffc..53cc02d0b10 100644 --- a/docs/en/cli-reference/scheduler.md +++ b/docs/en/cli-reference/scheduler.md @@ -1,18 +1,21 @@ # Scheduler -Scheduler is a long-running process which receives and manages download tasks from the client, notify the CDN to return to the source, generate and maintain a -P2P network during the download process, and push suitable download nodes to the client +Scheduler is a long-running process which receives +and manages download tasks from the client, +notify the CDN to return to the source, generate and maintain a +P2P network during the download process, +and push suitable download nodes to the client ## Usage -``` +```text scheduler [flags] scheduler [command] ``` -## Available Commands: +## Available Commands -``` +```text doc generate documents help Help about any command version show version @@ -20,9 +23,10 @@ version show version ## Flags -``` + +```text --config string the path of configuration file with yaml extension name, default is /Users/${USER_HOME}/.dragonfly/config/scheduler.yaml, it can - also be set by environment variable scheduler_config + also be set by environment variable scheduler_config --console whether logger output records to the stdout -h, --help help for scheduler --jaeger string jaeger endpoint url, like: http://localhost:14250/api/traces @@ -30,3 +34,4 @@ version show version --service-name string name of the service for tracer (default "dragonfly-scheduler") --verbose whether logger use debug level ``` + diff --git a/docs/en/deployment/README.md b/docs/en/deployment/README.md index 156ef101f4c..6432fa27458 100644 --- a/docs/en/deployment/README.md +++ b/docs/en/deployment/README.md @@ -7,8 +7,12 @@ Table of contents: ## Installation -Installation documents provide you with several options available to deploy the Dragonfly, such as [Kubernetes](installation/kubernetes/README.md) and [Docker](installation/docker/README.md). +Installation documents provide you with several options available to +deploy the Dragonfly, such as [Kubernetes](installation/kubernetes/README.md) +and [Docker](installation/docker/README.md). ## Configuration -[Configuration](configuration) is content all about customizing the Dragonfly. It includes information about the configuration of all components and their corresponding explanations. +[Configuration](configuration) is content all about customizing the Dragonfly. +It includes information about the configuration of all components and +their corresponding explanations. diff --git a/docs/en/deployment/installation/docker/README.md b/docs/en/deployment/installation/docker/README.md index d9efdfeb2d2..4ae0d666ce9 100644 --- a/docs/en/deployment/installation/docker/README.md +++ b/docs/en/deployment/installation/docker/README.md @@ -1,6 +1,8 @@ # Installation -There are two installation methods, it is recommended to use docker-compose to install as a whole, or install separately according to modules. +There are two installation methods, it is recommended to +use docker-compose to install as a whole, +or install separately according to modules. Table of contents: diff --git a/docs/en/deployment/installation/docker/standalone/README.md b/docs/en/deployment/installation/docker/standalone/README.md index 55e63410e1c..19d5e47366c 100644 --- a/docs/en/deployment/installation/docker/standalone/README.md +++ b/docs/en/deployment/installation/docker/standalone/README.md @@ -1,12 +1,16 @@ # Installation -There are two installation methods, it is recommended to use helm charts to install as a whole, or install separately according to modules +There are two installation methods, it is recommended to +use helm charts to install as a whole, or install separately according to modules ## Install with Helm Charts -> ArtifactHub Link: https://artifacthub.io/packages/helm/dragonfly/dragonfly +> ArtifactHub Link: -It is recommended to use helm charts for installation, which is a simple and quick installation. [Kubernetes-with-Dragonfly](../../kubernetes/README.md) is the installation instructions of dragonfly. +It is recommended to use helm charts for +installation, which is a simple and quick installation. +[Kubernetes-with-Dragonfly](../../kubernetes/README.md) +is the installation instructions of dragonfly. ## Install separately @@ -17,4 +21,3 @@ It is recommended to use helm charts for installation, which is a simple and qui * [install-scheduler](./scheduler.md) is the installation instructions of scheduler * [install-cdn](./cdn.md) is the installation instructions of CDN - diff --git a/docs/en/deployment/installation/docker/standalone/cdn.md b/docs/en/deployment/installation/docker/standalone/cdn.md index 0c829cc1590..a75b8b8c11a 100644 --- a/docs/en/deployment/installation/docker/standalone/cdn.md +++ b/docs/en/deployment/installation/docker/standalone/cdn.md @@ -19,49 +19,53 @@ You can get it from [DockerHub](https://hub.docker.com/) directly. 1. Obtain the latest Docker image of the cdn. -```sh -docker pull dragonflyoss/cdn -``` + ```sh + docker pull dragonflyoss/cdn + ``` Or you can build your own cdn image. 1. Obtain the source code of Dragonfly. -```sh -git clone https://github.com/dragonflyoss/Dragonfly2.git -``` + ```sh + git clone https://github.com/dragonflyoss/Dragonfly2.git + ``` 2. Enter the project directory. -```sh -cd Dragonfly2 -``` + ```sh + cd Dragonfly2 + ``` 3. Build the Docker image. -```sh -TAG="2.0.0" -make docker-build-cdn D7Y_VERSION=$TAG -``` + ```sh + TAG="2.0.0" + make docker-build-cdn D7Y_VERSION=$TAG + ``` 4. Obtain the latest Docker image ID of the cdn. -```sh -docker image ls | grep 'cdn' | awk '{print $3}' | head -n1 -``` + ```sh + docker image ls | grep 'cdn' | awk '{print $3}' | head -n1 + ``` ### Start cdn **NOTE:** Replace ${cdnDockerImageId} with the ID obtained at the previous step. ```sh -docker run -d --name cdn --restart=always -p 8001:8001 -p 8003:8003 -v /home/admin/ftp:/home/admin/ftp ${cdnDockerImageId} ---download-port=8001 +docker run -d --name cdn --restart=always \ + -p 8001:8001 -p 8003:8003 \ + -v /home/admin/ftp:/home/admin/ftp ${cdnDockerImageId} \ + --download-port=8001 ``` -After cdn is installed, run the following commands to verify if Nginx and **cdn** are started, and if Port `8001` and `8003` are available. +After cdn is installed, run the following commands to +verify if Nginx and **cdn** are started, +and if Port `8001` and `8003` are available. ```sh telnet 127.0.0.1 8001 telnet 127.0.0.1 8003 -``` \ No newline at end of file +``` diff --git a/docs/en/deployment/installation/docker/standalone/dfdaemon.md b/docs/en/deployment/installation/docker/standalone/dfdaemon.md index 539b86a9c34..e76f37a5b53 100644 --- a/docs/en/deployment/installation/docker/standalone/dfdaemon.md +++ b/docs/en/deployment/installation/docker/standalone/dfdaemon.md @@ -19,49 +19,52 @@ You can get it from [DockerHub](https://hub.docker.com/) directly. 1. Obtain the latest Docker image of the dfdaemon. -```sh -docker pull dragonflyoss/dfdaemon -``` + ```sh + docker pull dragonflyoss/dfdaemon + ``` Or you can build your own dfdaemon image. 1. Obtain the source code of Dragonfly. -```sh -git clone https://github.com/dragonflyoss/Dragonfly2.git -``` + ```sh + git clone https://github.com/dragonflyoss/Dragonfly2.git + ``` 2. Enter the project directory. -```sh -cd Dragonfly2 -``` + ```sh + cd Dragonfly2 + ``` 3. Build the Docker image. -```sh -TAG="2.0.0" -make docker-build-dfdaemon D7Y_VERSION=$TAG -``` + ```sh + TAG="2.0.0" + make docker-build-dfdaemon D7Y_VERSION=$TAG + ``` 4. Obtain the latest Docker image ID of the dfdaemon. -```sh -docker image ls | grep 'dfdaemon' | awk '{print $3}' | head -n1 -``` + ```sh + docker image ls | grep 'dfdaemon' | awk '{print $3}' | head -n1 + ``` ### Start dfdaemon **NOTE:** Replace ${dfdaemonDockerImageId} with the ID obtained at the previous step. ```sh -docker run -d --name dfdaemon --restart=always -p 65000:65000 -p 65001:65001 -p 65002:65002 ${dfdaemonDockerImageId} daemon +docker run -d --name dfdaemon --restart=always \ + -p 65000:65000 -p 65001:65001 -p 65002:65002 ${dfdaemonDockerImageId} daemon ``` -After dfget is installed, run the following commands to verify if **dfdaemon** is started, and if Port `65000`, `65001` and `65002` is available. +After dfget is installed, run the following commands to +verify if **dfdaemon** is started, +and if Port `65000`, `65001` and `65002` is available. ```sh telnet 127.0.0.1 65000 telnet 127.0.0.1 65001 telnet 127.0.0.1 65002 -``` \ No newline at end of file +``` diff --git a/docs/en/deployment/installation/docker/standalone/manager.md b/docs/en/deployment/installation/docker/standalone/manager.md index b3fd6649754..e10a9b84709 100644 --- a/docs/en/deployment/installation/docker/standalone/manager.md +++ b/docs/en/deployment/installation/docker/standalone/manager.md @@ -19,36 +19,36 @@ You can get it from [DockerHub](https://hub.docker.com/) directly. 1. Obtain the latest Docker image of the manager. -```sh -docker pull d7yio/manager -``` + ```sh + docker pull d7yio/manager + ``` Or you can build your own manager image. 1. Obtain the source code of Dragonfly. -```sh -git clone https://github.com/dragonflyoss/Dragonfly2.git -``` + ```sh + git clone https://github.com/dragonflyoss/Dragonfly2.git + ``` 2. Enter the project directory. -```sh -cd Dragonfly2 -``` + ```sh + cd Dragonfly2 + ``` 3. Build the Docker image. -```sh -TAG="2.0.0" -make docker-build-manager D7Y_VERSION=$TAG -``` + ```sh + TAG="2.0.0" + make docker-build-manager D7Y_VERSION=$TAG + ``` 4. Obtain the latest Docker image ID of the manager. -```sh -docker image ls | grep 'manager' | awk '{print $3}' | head -n1 -``` + ```sh + docker image ls | grep 'manager' | awk '{print $3}' | head -n1 + ``` ### Start manager @@ -58,7 +58,8 @@ docker image ls | grep 'manager' | awk '{print $3}' | head -n1 docker run -d --name manager --restart=always -p 8080:8080 -p 65003:65003 ${managerDockerImageId} ``` -After manager is installed, run the following commands to verify if **manager** is started, and if Port `8080` and `65003` is available. +After manager is installed, run the following commands to +verify if **manager** is started, and if Port `8080` and `65003` is available. ```sh telnet 127.0.0.1 8080 @@ -69,4 +70,5 @@ telnet 127.0.0.1 65003 Now you can open brower and visit [console](http://localhost:8080) -Console features preview reference document [console preview](../../../../design/manager.md). +Console features preview reference document +[console preview](../../../../design/manager.md). diff --git a/docs/en/deployment/installation/docker/standalone/scheduler.md b/docs/en/deployment/installation/docker/standalone/scheduler.md index a48b9eb3fe4..b0b9eb6a22f 100644 --- a/docs/en/deployment/installation/docker/standalone/scheduler.md +++ b/docs/en/deployment/installation/docker/standalone/scheduler.md @@ -19,46 +19,48 @@ You can get it from [DockerHub](https://hub.docker.com/) directly. 1. Obtain the latest Docker image of the scheduler. -```sh -docker pull dragonflyoss/scheduler -``` + ```sh + docker pull dragonflyoss/scheduler + ``` Or you can build your own scheduler image. 1. Obtain the source code of Dragonfly. -```sh -git clone https://github.com/dragonflyoss/Dragonfly2.git -``` + ```sh + git clone https://github.com/dragonflyoss/Dragonfly2.git + ``` 2. Enter the project directory. -```sh -cd Dragonfly2 -``` + ```sh + cd Dragonfly2 + ``` 3. Build the Docker image. -```sh -TAG="2.0.0" -make docker-build-scheduler D7Y_VERSION=$TAG -``` + ```sh + TAG="2.0.0" + make docker-build-scheduler D7Y_VERSION=$TAG + ``` 4. Obtain the latest Docker image ID of the scheduler. -```sh -docker image ls | grep 'scheduler' | awk '{print $3}' | head -n1 -``` + ```sh + docker image ls | grep 'scheduler' | awk '{print $3}' | head -n1 + ``` ### Start scheduler -**NOTE:** Replace ${schedulerDockerImageId} with the ID obtained at the previous step. +**NOTE:** Replace ${schedulerDockerImageId} with +the ID obtained at the previous step. ```sh docker run -d --name scheduler --restart=always -p 8002:8002 ${schedulerDockerImageId} ``` -After scheduler is installed, run the following commands to verify if **scheduler** is started, and if Port `8002` is available. +After scheduler is installed, run the following commands to +verify if **scheduler** is started, and if Port `8002` is available. ```sh telnet 127.0.0.1 8002 diff --git a/docs/en/deployment/installation/executable-files/README.md b/docs/en/deployment/installation/executable-files/README.md index c4b5c11e808..eb9a1a10a5f 100644 --- a/docs/en/deployment/installation/executable-files/README.md +++ b/docs/en/deployment/installation/executable-files/README.md @@ -1,6 +1,7 @@ # Installation -There are installation methods to install the executable files separately according to the modules. +There are installation methods to install +the executable files separately according to the modules. ## Install separately diff --git a/docs/en/deployment/installation/executable-files/cdn.md b/docs/en/deployment/installation/executable-files/cdn.md index 1c23c42b93c..03acac5ef6c 100644 --- a/docs/en/deployment/installation/executable-files/cdn.md +++ b/docs/en/deployment/installation/executable-files/cdn.md @@ -16,42 +16,44 @@ Nginx|0.8+ ### Get cdn executable file -1. Download a binary package of the cdn. You can download one of the latest builds for Dragonfly on the [github releases page](https://github. - com/dragonflyoss/Dragonfly2/releases). +1. Download a binary package of the cdn. You can download one of +the latest builds for Dragonfly on the +[github releases page](https://github.com/dragonflyoss/Dragonfly2/releases). -```sh -version=2.0.0 -wget https://github.com/dragonflyoss/Dragonfly2/releases/download/v$version/Dragonfly2_$version_linux_amd64.tar.gz -``` + ```sh + version=2.0.0 + wget https://github.com/dragonflyoss/Dragonfly2/releases/download/v$version/Dragonfly2_$version_linux_amd64.tar.gz + ``` 2. Unzip the package. -```bash -# Replace `xxx` with the installation directory. -tar -zxf Dragonfly2_2.0.0_linux_amd64.tar.gz -C xxx -``` + ```bash + # Replace `xxx` with the installation directory. + tar -zxf Dragonfly2_2.0.0_linux_amd64.tar.gz -C xxx + ``` -3. Move the `cdn` to your `PATH` environment variable to make sure you can directly use `cdn` command. +3. Move the `cdn` to your `PATH` environment variable to +make sure you can directly use `cdn` command. Or you can build your own cdn executable file. 1. Obtain the source code of Dragonfly. -```sh -git clone https://github.com/dragonflyoss/Dragonfly2.git -``` + ```sh + git clone https://github.com/dragonflyoss/Dragonfly2.git + ``` 2. Enter the project directory. -```sh -cd Dragonfly2 -``` + ```sh + cd Dragonfly2 + ``` 3. Compile the source code. -```sh -make build-cdn && make install-cdn -``` + ```sh + make build-cdn && make install-cdn + ``` ### Start cdn @@ -63,33 +65,39 @@ cdn --port=8003 --download-port=$cdnDownloadPort ### Start file server -You can start a file server in any way. However, the following conditions must be met: +You can start a file server in any way. +However, the following conditions must be met: -- It must be rooted at `cdnHomeDir/ftp` which is defined in the previous step. -- It must listen on the port `cdnDownloadPort` which is defined in the previous step. +- It must be rooted at `cdnHomeDir/ftp` which is +defined in the previous step. +- It must listen on the port `cdnDownloadPort` which is +defined in the previous step. Let's take nginx as an example. -1. Add the following configuration items to the Nginx configuration file. - -```conf -server { -# Must be ${cdnDownloadPort} -listen 8001; -location / { - # Must be ${cdnHomeDir}/ftp - root /home/admin/ftp; -} -} -``` +1. Add the following configuration items to +the Nginx configuration file. + + ```conf + server { + # Must be ${cdnDownloadPort} + listen 8001; + location / { + # Must be ${cdnHomeDir}/ftp + root /home/admin/ftp; + } + } + ``` 2. Start Nginx. -```sh -sudo nginx -``` + ```sh + sudo nginx + ``` -After cdn is installed, run the following commands to verify if Nginx and **cdn** are started, and if Port `8001` and `8003` are available. +After cdn is installed, run the following commands to +verify if Nginx and **cdn** are started, +and if Port `8001` and `8003` are available. ```sh telnet 127.0.0.1 8001 diff --git a/docs/en/deployment/installation/executable-files/dfdaemon.md b/docs/en/deployment/installation/executable-files/dfdaemon.md index 141de22f9bd..5fcb2cb968d 100644 --- a/docs/en/deployment/installation/executable-files/dfdaemon.md +++ b/docs/en/deployment/installation/executable-files/dfdaemon.md @@ -15,42 +15,44 @@ Golang|1.12.x ### Get dfget executable file -1. Download a binary package of the dfget. You can download one of the latest builds for Dragonfly on the [github releases page](https://github. - com/dragonflyoss/Dragonfly2/releases). +1. Download a binary package of the dfget. You can download one of +the latest builds for Dragonfly on the +[github releases page](https://github.com/dragonflyoss/Dragonfly2/releases). -```sh -version=2.0.0 -wget https://github.com/dragonflyoss/Dragonfly2/releases/download/v$version/Dragonfly2_$version_linux_amd64.tar.gz -``` + ```sh + version=2.0.0 + wget https://github.com/dragonflyoss/Dragonfly2/releases/download/v$version/Dragonfly2_$version_linux_amd64.tar.gz + ``` 2. Unzip the package. -```bash -# Replace `xxx` with the installation directory. -tar -zxf Dragonfly2_2.0.0_linux_amd64.tar.gz -C xxx -``` + ```bash + # Replace `xxx` with the installation directory. + tar -zxf Dragonfly2_2.0.0_linux_amd64.tar.gz -C xxx + ``` -3. Move the `dfget` to your `PATH` environment variable to make sure you can directly use `dfget` command. +3. Move the `dfget` to your `PATH` environment variable to +make sure you can directly use `dfget` command. Or you can build your own dfget executable file. 1. Obtain the source code of Dragonfly. -```sh -git clone https://github.com/dragonflyoss/Dragonfly2.git -``` + ```sh + git clone https://github.com/dragonflyoss/Dragonfly2.git + ``` 2. Enter the project directory. -```sh -cd Dragonfly2 -``` + ```sh + cd Dragonfly2 + ``` 3. Compile the source code. -```sh -make build-dfget && make install-dfget -``` + ```sh + make build-dfget && make install-dfget + ``` ### Start dfdaemon @@ -58,7 +60,9 @@ make build-dfget && make install-dfget dfget daemon --options ``` -After dfget is installed, run the following commands to verify if **dfdaemon** is started, and if Port `65000`, `65001` and `65002` is available. +After dfget is installed, run the following commands to +verify if **dfdaemon** is started, +and if Port `65000`, `65001` and `65002` is available. ```sh telnet 127.0.0.1 65000 diff --git a/docs/en/deployment/installation/executable-files/manager.md b/docs/en/deployment/installation/executable-files/manager.md index a0a6e682647..b2d8dc78acb 100644 --- a/docs/en/deployment/installation/executable-files/manager.md +++ b/docs/en/deployment/installation/executable-files/manager.md @@ -16,42 +16,44 @@ Nginx|0.8+ ### Get manager executable file -1. Download a binary package of the manager. You can download one of the latest builds for Dragonfly on the [github releases page](https://github. - com/dragonflyoss/Dragonfly2/releases). +1. Download a binary package of the manager. You can download +one of the latest builds for Dragonfly on +the [github releases page](https://github.com/dragonflyoss/Dragonfly2/releases). -```sh -version=2.0.0 -wget https://github.com/dragonflyoss/Dragonfly2/releases/download/v$version/Dragonfly2_$version_linux_amd64.tar.gz -``` + ```sh + version=2.0.0 + wget https://github.com/dragonflyoss/Dragonfly2/releases/download/v$version/Dragonfly2_$version_linux_amd64.tar.gz + ``` 2. Unzip the package. -```bash -# Replace `xxx` with the installation directory. -tar -zxf Dragonfly2_2.0.0_linux_amd64.tar.gz -C xxx -``` + ```sh + # Replace `xxx` with the installation directory. + tar -zxf Dragonfly2_2.0.0_linux_amd64.tar.gz -C xxx + ``` -3. Move the `manager` to your `PATH` environment variable to make sure you can directly use `manager` command. +3. Move the `manager` to your `PATH` environment variable +to make sure you can directly use `manager` command. Or you can build your own manager executable file. 1. Obtain the source code of Dragonfly. -```sh -git clone https://github.com/dragonflyoss/Dragonfly2.git -``` + ```sh + git clone https://github.com/dragonflyoss/Dragonfly2.git + ``` 2. Enter the project directory. -```sh -cd Dragonfly2 -``` + ```sh + cd Dragonfly2 + ``` 3. Compile the source code. -```sh -make build-manager && make install-manager -``` + ```sh + make build-manager && make install-manager + ``` ### Start manager @@ -59,7 +61,8 @@ make build-manager && make install-manager manager --options ``` -After manager is installed, run the following commands to verify if **manager** is started, and if Port `8080` and `65003` is available. +After manager is installed, run the following commands to +verify if **manager** is started, and if Port `8080` and `65003` is available. ```sh telnet 127.0.0.1 8080 @@ -70,4 +73,5 @@ telnet 127.0.0.1 65003 Now you can open brower and visit [console](http://localhost:8080) -Console features preview reference document [console preview](../../../design/manager.md). +Console features preview reference document +[console preview](../../../design/manager.md). diff --git a/docs/en/deployment/installation/executable-files/scheduler.md b/docs/en/deployment/installation/executable-files/scheduler.md index 3fab79a5c03..0c6d6be96d4 100644 --- a/docs/en/deployment/installation/executable-files/scheduler.md +++ b/docs/en/deployment/installation/executable-files/scheduler.md @@ -16,42 +16,44 @@ Nginx|0.8+ ### Get scheduler executable file -1. Download a binary package of the scheduler. You can download one of the latest builds for Dragonfly on the [github releases page](https://github. - com/dragonflyoss/Dragonfly2/releases). +1. Download a binary package of the scheduler. +You can download one of the latest builds for +Dragonfly on the [github releases page](https://github.com/dragonflyoss/Dragonfly2/releases). -```sh -version=2.0.0 -wget https://github.com/dragonflyoss/Dragonfly2/releases/download/v$version/Dragonfly2_$version_linux_amd64.tar.gz -``` + ```sh + version=2.0.0 + wget https://github.com/dragonflyoss/Dragonfly2/releases/download/v$version/Dragonfly2_$version_linux_amd64.tar.gz + ``` 2. Unzip the package. -```bash -# Replace `xxx` with the installation directory. -tar -zxf Dragonfly2_2.0.0_linux_amd64.tar.gz -C xxx -``` + ```bash + # Replace `xxx` with the installation directory. + tar -zxf Dragonfly2_2.0.0_linux_amd64.tar.gz -C xxx + ``` -3. Move the `scheduler` to your `PATH` environment variable to make sure you can directly use `scheduler` command. +3. Move the `scheduler` to your `PATH` environment +variable to make sure you can directly use `scheduler` command. Or you can build your own scheduler executable file. 1. Obtain the source code of Dragonfly. -```sh -git clone https://github.com/dragonflyoss/Dragonfly2.git -``` + ```sh + git clone https://github.com/dragonflyoss/Dragonfly2.git + ``` 2. Enter the project directory. -```sh -cd Dragonfly2 -``` + ```sh + cd Dragonfly2 + ``` 3. Compile the source code. -```sh -make build-scheduler && make install-scheduler -``` + ```sh + make build-scheduler && make install-scheduler + ``` ### Start scheduler @@ -59,7 +61,8 @@ make build-scheduler && make install-scheduler scheduler --options ``` -After scheduler is installed, run the following commands to verify if **scheduler** is started, and if Port `8002` is available. +After scheduler is installed, run the following commands to +verify if **scheduler** is started, and if Port `8002` is available. ```sh telnet 127.0.0.1 8002 diff --git a/docs/en/deployment/installation/kubernetes/README.md b/docs/en/deployment/installation/kubernetes/README.md index bcd10d96e19..a91c0658bab 100644 --- a/docs/en/deployment/installation/kubernetes/README.md +++ b/docs/en/deployment/installation/kubernetes/README.md @@ -1,6 +1,7 @@ # Kubernetes with Dragonfly -Now we can deploy all components of Dragonfly in Kubernetes cluster. We deploy scheduler and cdn as `StatefulSets`, +Now we can deploy all components of Dragonfly in +Kubernetes cluster. We deploy scheduler and cdn as `StatefulSets`, daemon as `DaemonSets`, manager as `Deployments`. Table of contents: diff --git a/docs/en/deployment/installation/kubernetes/helm.md b/docs/en/deployment/installation/kubernetes/helm.md index bd1478bce5e..4fb3bda86ff 100644 --- a/docs/en/deployment/installation/kubernetes/helm.md +++ b/docs/en/deployment/installation/kubernetes/helm.md @@ -2,25 +2,37 @@ ## Runtime Configuration Guide for Dragonfly Helm Chart -When enable runtime configuration in dragonfly, you can skip [Configure Runtime](#configure-runtime-manually) manually. +When enable runtime configuration in dragonfly, +you can skip [Configure Runtime](#configure-runtime-manually) manually. ### 1. Docker -> **We did not recommend to using dragonfly with docker in Kubernetes** due to many reasons: 1. no fallback image pulling policy. 2. deprecated in Kubernetes. -> Because the original `daemonset` in Kubernetes did not support `Surging Rolling Update` policy. -> When kill current dfdaemon pod, the new pod image can not be pulled anymore. - -> If you can not change runtime from docker to others, remind to choose a plan when upgrade dfdaemon: -> Option 1: pull newly dfdaemon image manually before upgrade dragonfly, or use [ImagePullJob](https://openkruise.io/docs/user-manuals/imagepulljob) to pull image automate. -> Option 2: keep the image registry of dragonfly is different from common registries and add host in `containerRuntime.docker.skipHosts`. +> **We did not recommend to using dragonfly +with docker in Kubernetes** +due to many reasons: +no fallback image pulling policy. +deprecated in Kubernetes. +> Because the original `daemonset` in +Kubernetes did not support `Surging Rolling Update` policy. +> When kill current dfdaemon pod, +the new pod image can not be pulled anymore. +> If you can not change runtime from docker to others, +remind to choose a plan when upgrade dfdaemon: +> pull newly dfdaemon image manually before upgrade dragonfly, +or use [ImagePullJob](https://openkruise.io/docs/user-manuals/imagepulljob) to +pull image automate. +> keep the image registry of dragonfly is +different from common registries and add host in `containerRuntime.docker.skipHosts`. Dragonfly helm supports config docker automatically. Config cases: -**Case 1: [Preferred] Implicit registries support without restart docker** +#### Case 1: [Preferred] Implicit registries support without restart docker Chart customize values.yaml: + + ```yaml containerRuntime: docker: @@ -33,22 +45,30 @@ containerRuntime: - "harbor.example.com" - "harbor.example.net" ``` + -This config enables docker pulling images from registries `harbor.example.com` and `harbor.example.net` via Dragonfly. -When deploying Dragonfly with above config, it's unnecessary to restart docker daemon. +This config enables docker pulling images from +registries `harbor.example.com` and `harbor.example.net` via Dragonfly. +When deploying Dragonfly with above config, +it's unnecessary to restart docker daemon. Advantages: -* Support upgrade dfdaemon smoothness -> In this mode, when dfdaemon pod deleted, the `preStop` hook will remove all injected hosts info in /etc/hosts, +- Support upgrade dfdaemon smoothness + +> In this mode, when dfdaemon pod deleted, + the `preStop` hook will remove all injected hosts info in /etc/hosts, > all images traffic fallbacks to original registries. Limitations: -* Only support implicit registries -**Case 2: Arbitrary registries support with restart docker** +- Only support implicit registries + +#### Case 2: Arbitrary registries support with restart docker Chart customize values.yaml: + + ```yaml containerRuntime: docker: @@ -61,25 +81,30 @@ containerRuntime: - "127.0.0.1" - "docker.io" # Dragonfly use this image registry to upgrade itself, so we need skip it. Change it in real environment. ``` + This config enables docker pulling images from arbitrary registries via Dragonfly. When deploying Dragonfly with above config, dfdaemon will restart docker daemon. Advantages: -* Support arbitrary registries + +- Support arbitrary registries Limitations: -* Must enable live-restore feature in docker -* Need restart docker daemon -* When upgrade dfdaemon, new image must be pulled beforehand. + +- Must enable live-restore feature in docker +- Need restart docker daemon +- When upgrade dfdaemon, new image must be pulled beforehand. ### 2. Containerd -The config of containerd has two version with complicated fields. These are many cases to consider: +The config of containerd has two version with complicated fields. +These are many cases to consider: -**Case 1: Version 2 config with config_path** +#### Case 1: Version 2 config with config_path There is `config_path` in `/etc/containerd/config.toml`: + ```toml [plugins."io.containerd.grpc.v1.cri".registry] config_path = "/etc/containerd/certs.d" @@ -88,41 +113,46 @@ There is `config_path` in `/etc/containerd/config.toml`: This case is very simple to enable multiple registry mirrors support. Chart customize values.yaml: + ```yaml containerRuntime: containerd: enable: true ``` -**Case 2: Version 2 config without config_path** +#### Case 2: Version 2 config without config_path -* Option 1 - Allow charts to inject config_path and restart containerd. +- Option 1 - Allow charts to inject config_path and restart containerd. -This option also enable multiple registry mirrors support. + This option also enable multiple registry mirrors support. -> Caution: if there are already many other mirror config in config.toml, should not use this option, or migrate your config with `config_path`. + > Caution: if there are already many other mirror config in config.toml, + should not use this option, or migrate your config with `config_path`. -Chart customize values.yaml: -```yaml -containerRuntime: - containerd: - enable: true - injectConfigPath: true -``` + Chart customize values.yaml: -* Option 2 - Just mirror only one registry which `dfdaemon.config.proxy.registryMirror.url` is + ```yaml + containerRuntime: + containerd: + enable: true + injectConfigPath: true + ``` -Chart customize values.yaml: +- Option 2 - Just mirror only one registry +which `dfdaemon.config.proxy.registryMirror.url` is -```yaml -containerRuntime: - containerd: - enable: true -``` + Chart customize values.yaml: + + ```yaml + containerRuntime: + containerd: + enable: true + ``` -**Case 3: Version 1** +#### Case 3: Version 1 -With version 1 config.toml, only support the registry which `dfdaemon.config.proxy.registryMirror.url` is. +With version 1 config.toml, only support +the registry which `dfdaemon.config.proxy.registryMirror.url` is. Chart customize values.yaml: @@ -139,6 +169,7 @@ containerRuntime: Dragonfly helm supports config CRI-O automatically with drop-in registries. Chart customize values.yaml: + ```yaml containerRuntime: crio: @@ -154,7 +185,8 @@ containerRuntime: ## Prepare Kubernetes Cluster -If there is no available Kubernetes cluster for testing, [minikube](https://minikube.sigs.k8s.io/docs/start/) is +If there is no available Kubernetes cluster for testing, +[minikube](https://minikube.sigs.k8s.io/docs/start/) is recommended. Just run `minikube start`. ## Install Dragonfly @@ -168,9 +200,12 @@ helm install --create-namespace --namespace dragonfly-system dragonfly dragonfly ### Install with custom configuration -Create the `values.yaml` configuration file. It is recommended to use external redis and mysql instead of containers. +Create the `values.yaml` configuration file. +It is recommended to use external redis and mysql instead of containers. -The example uses external mysql and redis. Refer to the document for [configuration](https://artifacthub.io/packages/helm/dragonfly/dragonfly#values). +The example uses external mysql and redis. +Refer to the document for +[configuration](https://artifacthub.io/packages/helm/dragonfly/dragonfly#values). ```yaml mysql: @@ -197,14 +232,17 @@ Install dragonfly with `values.yaml`. ```shell helm repo add dragonfly https://dragonflyoss.github.io/helm-charts/ -helm install --create-namespace --namespace dragonfly-system dragonfly dragonfly/dragonfly -f values.yaml +helm install --create-namespace --namespace dragonfly-system \ + dragonfly dragonfly/dragonfly -f values.yaml ``` ### Install with an existing manager -Create the `values.yaml` configuration file. Need to configure the cluster id associated with scheduler and cdn. +Create the `values.yaml` configuration file. +Need to configure the cluster id associated with scheduler and cdn. -The example is to deploy a cluster using the existing manager and redis. Refer to the document for [configuration](https://artifacthub.io/packages/helm/dragonfly/dragonfly#values). +The example is to deploy a cluster using the existing manager and redis. +Refer to the document for [configuration](https://artifacthub.io/packages/helm/dragonfly/dragonfly#values). ```yaml scheduler: @@ -251,7 +289,9 @@ kubectl -n dragonfly-system wait --for=condition=ready --all --timeout=10m pod The console page will be displayed on `dragonfly-manager.dragonfly-system.svc.cluster.local:8080`. -If you need to bind Ingress, you can refer to [configuration options](https://artifacthub.io/packages/helm/dragonfly/dragonfly#values) of Helm Charts, or create it manually. +If you need to bind Ingress, you can refer to +[configuration options](https://artifacthub.io/packages/helm/dragonfly/dragonfly#values) +of Helm Charts, or create it manually. Console features preview reference document [console preview](../../../design/manager.md). @@ -287,7 +327,8 @@ systemctl restart containerd ## Using Dragonfly -After all above steps, create a new pod with target registry. Or just pull an image with `crictl`: +After all above steps, create a new pod with +target registry. Or just pull an image with `crictl`: ```shell crictl harbor.example.com/library/alpine:latest @@ -298,6 +339,7 @@ crictl pull docker.io/library/alpine:latest ``` After pulled images, find logs in dfdaemon pod: + ```shell # find pods kubectl -n dragonfly-system get pod -l component=dfdaemon @@ -307,6 +349,15 @@ kubectl -n dragonfly-system exec -it ${pod_name} -- grep "peer task done" /var/l ``` Example output: + +```shell +{ + "level": "info", + "ts": "2021-06-28 06:02:30.924", + "caller": "peer/peertask_stream_callback.go:77", + "msg": "stream peer task done, cost: 2838ms", + "peer": "172.17.0.9-1-ed7a32ae-3f18-4095-9f54-6ccfc248b16e", + "task": "3c658c488fd0868847fab30976c2a079d8fd63df148fb3b53fd1a418015723d7", + "component": "streamPeerTask" +} ``` -{"level":"info","ts":"2021-06-28 06:02:30.924","caller":"peer/peertask_stream_callback.go:77","msg":"stream peer task done, cost: 2838ms","peer":"172.17.0.9-1-ed7a32ae-3f18-4095-9f54-6ccfc248b16e","task":"3c658c488fd0868847fab30976c2a079d8fd63df148fb3b53fd1a418015723d7","component":"streamPeerTask"} -``` \ No newline at end of file diff --git a/docs/en/deployment/installation/kubernetes/kustomize.md b/docs/en/deployment/installation/kubernetes/kustomize.md index 475892c4ebf..5e8c37e9c94 100644 --- a/docs/en/deployment/installation/kubernetes/kustomize.md +++ b/docs/en/deployment/installation/kubernetes/kustomize.md @@ -2,15 +2,18 @@ ## Prepare Kubernetes Cluster -If there is no available Kubernetes cluster for testing, [minikube](https://minikube.sigs.k8s.io/docs/start/) is +If there is no available Kubernetes cluster for testing, +[minikube](https://minikube.sigs.k8s.io/docs/start/) is recommended. Just run `minikube start`. ## Build and Apply Kustomize Configuration + ```shell git clone https://github.com/dragonflyoss/Dragonfly2.git kustomize build Dragonfly2/deploy/kustomize/single-cluster-native/overlays/sample | kubectl apply -f - ``` + ## Wait Dragonfly Ready @@ -24,7 +27,9 @@ kubectl -n dragonfly-system wait --for=condition=ready --all --timeout=10m pod The console page will be displayed on `dragonfly-manager.dragonfly-system.svc.cluster.local:8080`. -If you need to bind Ingress, you can refer to [configuration options](https://artifacthub.io/packages/helm/dragonfly/dragonfly#values) of Helm Charts, or create it manually. +If you need to bind Ingress, you can refer to +[configuration options](https://artifacthub.io/packages/helm/dragonfly/dragonfly#values) +of Helm Charts, or create it manually. Console features preview reference document [console preview](../../../design/manager.md). @@ -60,7 +65,8 @@ systemctl restart containerd ## Using Dragonfly -After all above steps, create a new pod with target registry. Or just pull an image with `crictl`: +After all above steps, create a new pod with target registry. +Or just pull an image with `crictl`: ```shell crictl harbor.example.com/library/alpine:latest @@ -71,6 +77,7 @@ crictl pull docker.io/library/alpine:latest ``` After pulled images, find logs in dfdaemon pod: + ```shell # find pods kubectl -n dragonfly-system get pod -l component=dfdaemon @@ -80,6 +87,15 @@ kubectl -n dragonfly-system exec -it ${pod_name} -- grep "peer task done" /var/l ``` Example output: -``` -{"level":"info","ts":"2021-06-28 06:02:30.924","caller":"peer/peertask_stream_callback.go:77","msg":"stream peer task done, cost: 2838ms","peer":"172.17.0.9-1-ed7a32ae-3f18-4095-9f54-6ccfc248b16e","task":"3c658c488fd0868847fab30976c2a079d8fd63df148fb3b53fd1a418015723d7","component":"streamPeerTask"} + +```shell +{ + "level": "info", + "ts": "2021-06-28 06:02:30.924", + "caller": "peer/peertask_stream_callback.go:77", + "msg": "stream peer task done, cost: 2838ms", + "peer": "172.17.0.9-1-ed7a32ae-3f18-4095-9f54-6ccfc248b16e", + "task": "3c658c488fd0868847fab30976c2a079d8fd63df148fb3b53fd1a418015723d7", + "component": "streamPeerTask" +} ``` diff --git a/docs/en/deployment/installation/kubernetes/upgrade-guide.md b/docs/en/deployment/installation/kubernetes/upgrade-guide.md index 76fc8815243..ebcb1df0947 100644 --- a/docs/en/deployment/installation/kubernetes/upgrade-guide.md +++ b/docs/en/deployment/installation/kubernetes/upgrade-guide.md @@ -1 +1,3 @@ -# WIP Upgrade guide \ No newline at end of file +# WIP Upgrade guide + +TODO diff --git a/docs/en/design/README.md b/docs/en/design/README.md index 9c621cea4bb..f3eb7e71c27 100644 --- a/docs/en/design/README.md +++ b/docs/en/design/README.md @@ -1,6 +1,10 @@ # Design -Design is content all about design of Dragonfly. It includes all things taken into consideration at the very beginning, the architecture designed for all components in Dragonfly, the interactive workflow between components, all APIs in Dragonfly and some technical things else. +Design is content all about design of Dragonfly. +It includes all things taken into consideration at the very beginning, +the architecture designed for all components in Dragonfly, +the interactive workflow between components, +all APIs in Dragonfly and some technical things else. Table of contents: diff --git a/docs/en/design/architecture.md b/docs/en/design/architecture.md index ed9ab5e3335..0e8dcdaca90 100644 --- a/docs/en/design/architecture.md +++ b/docs/en/design/architecture.md @@ -2,37 +2,62 @@ ## Positioning -Provide enterprise-level (efficient, stable, secure, low-cost, product-oriented) file distribution and management services, and become the cloud-native best practice and standard solution in this realm. +Provide enterprise-level (efficient, stable, secure, low-cost, product-oriented) +file distribution and management services, +and become the cloud-native best practice and standard solution in this realm. ## What problems are solved -**Architecture design flaws:** The existing architecture is difficult to meet the growing business needs of file distribution, and has gradually exposed deficiencies in stability, efficiency, and security, and is facing more and more challenges +**Architecture design flaws:** The existing architecture is difficult to +meet the growing business needs of file distribution, +and has gradually exposed deficiencies in stability, +efficiency, and security, and is facing more and more challenges -**Insufficient value penetration:** Currently only supports HTTP back-to-source protocol, and lacks adaptation to other types of storage (HDFS, storage services of various cloud vendors, Maven, YUM, etc.) +**Insufficient value penetration:** Currently only supports HTTP back-to-source protocol, +and lacks adaptation to other types of +storage (HDFS, storage services of various cloud vendors, Maven, YUM, etc.) -**Single distribution mode:** Currently only supports active pull mode, lack of active push and active synchronization capabilities +**Single distribution mode:** Currently only supports active pull mode, +lack of active push and active synchronization capabilities -**Lack of product capabilities:** No well-experienced console functions are capabilities, such as distribution task management and control, data market, multi-tenancy, and permission control, etc. +**Lack of product capabilities:** No well-experienced console +functions are capabilities, such as distribution task management and control, +data market, multi-tenancy, and permission control, etc. ## Core Competence -- Through a unified back-to-source adaptation layer and realization of P2P file distribution capabilities that support various types of storage (HDFS, +- Through a unified back-to-source adaptation layer and + realization of P2P file distribution capabilities that + support various types of storage (HDFS, storage services of various cloud vendors, Maven, YUM, etc.) -- Support more distribution modes: active pull, active push, real-time synchronization, remote copy, automatic warm-up, cross-cloud transmission, etc. -- Separation and decoupling between systems, scheduling and CDN plug-in, and support on-demand deployment, light or heavy, internal or external, flexible to +- Support more distribution modes: active pull, active push, + real-time synchronization, remote copy, automatic warm-up, + cross-cloud transmission, etc. +- Separation and decoupling between systems, scheduling and CDN plug-in, + and support on-demand deployment, light or heavy, + internal or external, flexible to meet the actual needs of different scenarios -- Based on the newly designed P2P protocol framework of grpc, with better efficiency and stability -- Support encrypted transmission, account-based transmission authentication and current limiting, and multi-tenant isolation mechanism +- Based on the newly designed P2P protocol framework of grpc, + with better efficiency and stability +- Support encrypted transmission, account-based transmission + authentication and current limiting, and multi-tenant isolation mechanism - Support more efficient IO methods: multi-threaded IO, memory mapping, DMA, etc. -- Support dynamic compression, memory file system and more efficient scheduling algorithms to improve distribution efficiency -- The client supports third-party software's native integration of Dragonfly's P2P capabilities through the C/S mode -- Production capabilities: support file upload, task management of various distribution modes, data views, and global management and control functions -- One set of internal and external, core features are shared with each other, and non-common features are extended separately -- Further integration with the ecosystem: Harbor, Nydus (image downloading solution on demand), warehouse services of various cloud vendors, etc. +- Support dynamic compression, memory file system and + more efficient scheduling algorithms to improve distribution efficiency +- The client supports third-party software's native integration + of Dragonfly's P2P capabilities through the C/S mode +- Production capabilities: support file upload, task management of + various distribution modes, data views, and global management and control functions +- One set of internal and external, core features are + shared with each other, and non-common features are extended separately +- Further integration with the ecosystem: Harbor, Nydus + (image downloading solution on demand), + warehouse services of various cloud vendors, etc. ## Architecture Design ### Overall structure + ![alt][arch] ### Entity relationship @@ -51,7 +76,8 @@ Provide enterprise-level (efficient, stable, secure, low-cost, product-oriented) ### Sub-system function -**Manager** +#### Manager + - Dynamic configuration management - Data Big Disk & Bandwidth Measurement - File upload & distribution management and control @@ -61,16 +87,20 @@ Provide enterprise-level (efficient, stable, secure, low-cost, product-oriented) - Multi-tenant management - Command channel service (channel integration) -**Scheduler** +#### Scheduler + - Multi-dimensional P2P scheduling strategy - Intelligent scheduling algorithm (advanced) - The scheduling results can be quantitatively compared and support A/B Testing - Scheduling strategy plug-in and CDN subsystem plug-in - Meta-information distribution pipeline -- Client download results and back-to-source results statistics and docking with monitoring services +- Client download results and back-to-source results statistics and + docking with monitoring services + +#### CDN -**CDN** -- Multi-source adaptation capabilities, including catalog download and range download functions +- Multi-source adaptation capabilities, including catalog download + and range download functions - DMA read and write - Transfer back to the source - Distribute warm-up @@ -80,11 +110,14 @@ Provide enterprise-level (efficient, stable, secure, low-cost, product-oriented) - Secure transmission (symmetric encryption, integrity check, etc.) - Storage media plug-in -**Dfget daemon** -- Multi-source adaptation capabilities, including catalog download and range download functions +#### Dfget daemon + +- Multi-source adaptation capabilities, including catalog + download and range download functions - Efficient IO mechanism - IO scheduling of multiple download tasks (high IO throughput) -- Reduce the number of file reads and writes: improve the storage of temporary files and improve file integrity verification +- Reduce the number of file reads and writes: improve + the storage of temporary files and improve file integrity verification - DMA read and write - Streaming - Client download Proxy capability (http & https) @@ -94,12 +127,12 @@ Provide enterprise-level (efficient, stable, secure, low-cost, product-oriented) - Client elegant upgrade and deployment plan - Command execution -**Framework** +#### Framework + - High availability, high performance, easy integration - Consistent Hash algorithm selection - Client connection management - [arch]: ../images/arch.png [association]: ../images/association.png -[download-process]: ../images/download-process.png \ No newline at end of file +[download-process]: ../images/download-process.png diff --git a/docs/en/design/cdn.md b/docs/en/design/cdn.md index 00e5a1ff354..50890197c11 100644 --- a/docs/en/design/cdn.md +++ b/docs/en/design/cdn.md @@ -1 +1,3 @@ -# WIP CDN \ No newline at end of file +# WIP CDN + +TODO diff --git a/docs/en/design/dfdaemon.md b/docs/en/design/dfdaemon.md index 62b20ae8bae..4c61ef7185f 100644 --- a/docs/en/design/dfdaemon.md +++ b/docs/en/design/dfdaemon.md @@ -1 +1,3 @@ -# WIP Dfdaemon \ No newline at end of file +# WIP Dfdaemon + +TODO diff --git a/docs/en/design/manager.md b/docs/en/design/manager.md index 48f5a8911dc..67cd2765f91 100644 --- a/docs/en/design/manager.md +++ b/docs/en/design/manager.md @@ -1,6 +1,7 @@ # Manager -Used to manage the dynamic configuration that each module depends on, and provide keepalive and metrics functions. +Used to manage the dynamic configuration that each module depends on, +and provide keepalive and metrics functions. ## Relationship @@ -8,13 +9,19 @@ Used to manage the dynamic configuration that each module depends on, and provid - CDN cluster and CDN instance have a `1:N` relationship - Scheduler cluster and Scheduler instance have a `1:N` relationship +
+ -When the Scheduler instance starts, it reports to the manager the Scheduler Cluster ID. Refer to [document](../deployment/configuration/scheduler.yaml) to configure `schedulerClusterID`. +When the Scheduler instance starts, +it reports to the manager the Scheduler Cluster ID. +Refer to [document](../deployment/configuration/scheduler.yaml) to configure `schedulerClusterID`. -When the CDN instance starts, it reports to the manager the CDN Cluster ID. Refer to [document](../deployment/configuration/cdn.yaml) to configure `cdnClusterID`. +When the CDN instance starts, +it reports to the manager the CDN Cluster ID. +Refer to [document](../deployment/configuration/cdn.yaml) to configure `cdnClusterID`. ## User account @@ -46,15 +53,19 @@ Default root username: `root` password: `dragonfly`. ##### Configure Scheduler Cluster +

+ ##### Configure Scheduler Cluster's Client +

+ - `load_limit`: client host can provide the maximum upload load. @@ -70,13 +81,14 @@ Default root username: `root` password: `dragonfly`. ##### Configure CDN Cluster +

+ - `load_limit`: CDN host can provide the maximum upload load. - [signin]: ../images/manager-console/signin.jpg [signup]: ../images/manager-console/signup.jpg [scheduler-cluster]: ../images/manager-console/scheduler-cluster.jpg diff --git a/docs/en/design/scheduler.md b/docs/en/design/scheduler.md index 4633fac3a36..e027de2af9b 100644 --- a/docs/en/design/scheduler.md +++ b/docs/en/design/scheduler.md @@ -1 +1,3 @@ -# WIP Scheduler \ No newline at end of file +# WIP Scheduler + +TODO diff --git a/docs/en/developer-guide/developer-guide.md b/docs/en/developer-guide/developer-guide.md index 6bb26a7028c..ba08c74bd34 100644 --- a/docs/en/developer-guide/developer-guide.md +++ b/docs/en/developer-guide/developer-guide.md @@ -2,11 +2,11 @@ Easily set up a local development environment. -### Step 1: Install docker and docker compose +## Step 1: Install docker and docker compose See documentation on [docs.docker.com] -### Step 2: Start dragonfly +## Step 2: Start dragonfly Enter dragonfly documentation and start docker-compose. @@ -19,10 +19,11 @@ Creating dfdaemon ... done Attaching to cdn, scheduler, dfdaemon ``` -### Step 3: Log analysis +## Step 3: Log analysis Show dragonfly logs. + ```bash $ tail -f log/**/*.log ==> log/dragonfly/cdn/core.log <== @@ -48,8 +49,9 @@ $ tail -f log/**/*.log {"level":"info","ts":"2021-02-26 05:43:37.342","caller":"worker/sender.go:49","msg":"start sender worker : 50"} {"level":"info","ts":"2021-02-26 05:43:37.343","caller":"worker/worker_group.go:64","msg":"start scheduler worker number:6"} ``` + -### Step 4: Stop dragonfly +## Step 4: Stop dragonfly ```bash $ docker-compose down diff --git a/docs/en/preheat/README.md b/docs/en/preheat/README.md index e70c1e9b7ed..2803e7331c9 100644 --- a/docs/en/preheat/README.md +++ b/docs/en/preheat/README.md @@ -1,6 +1,7 @@ # Preheat -P2P provides two types of preheating: `image` and `file`. Users can be preheat in the `console` or directly call `preheat api` for preheating +P2P provides two types of preheating: `image` and `file`. +Users can be preheat in the `console` or directly call `preheat api` for preheating Table of contents: diff --git a/docs/en/preheat/api.md b/docs/en/preheat/api.md index 5a9424a6b84..5aaa49d31d5 100644 --- a/docs/en/preheat/api.md +++ b/docs/en/preheat/api.md @@ -1,8 +1,10 @@ # API -Use preheat apis for preheating. First create a POST request for preheating, you can refer to [create job api document](../api-reference/api-reference.md#create-job) +Use preheat apis for preheating. First create a POST request for preheating, +you can refer to [create job api document](../api-reference/api-reference.md#create-job) -If the `scheduler_cluster_ids` does not exist, it means to preheat all scheduler clusters. +If the `scheduler_cluster_ids` does not exist, +it means to preheat all scheduler clusters. ```bash curl --location --request POST 'http://dragonfly-manager:8080/api/v1/jobs' \ @@ -19,10 +21,23 @@ curl --location --request POST 'http://dragonfly-manager:8080/api/v1/jobs' \ If the output of command above has content like ```bash -{ "id": 1 "task_id": "group_4d1ea00e-740f-4dbf-a47e-dbdc08eb33e1", "type": "preheat", "status": "PENDING", "args": { "filter": "", "headers": null, "type": "image", "url": "https://registry-1.docker.io/v2/library/redis/manifests/latest" }} +{ + "id": 1, + "task_id": "group_4d1ea00e-740f-4dbf-a47e-dbdc08eb33e1", + "type": "preheat", + "status": "PENDING", + "args": { + "filter": "", + "headers": null, + "type": "image", + "url": "https://registry-1.docker.io/v2/library/redis/manifests/latest" + } +} ``` -Polling the preheating status with id. if status is `SUCCESS`, preheating is successful, you can refer to [get job api document](../api-reference/api-reference.md#get-job) +Polling the preheating status with id. if status is `SUCCESS`, +preheating is successful, you can refer to +[get job api document](../api-reference/api-reference.md#get-job) ```bash curl --request GET 'http://dragonfly-manager:8080/api/v1/jobs/1' @@ -31,5 +46,16 @@ curl --request GET 'http://dragonfly-manager:8080/api/v1/jobs/1' If the status is `SUCCESS`, the preheating is successful. ```bash -{ "id": 1 "task_id": "group_4d1ea00e-740f-4dbf-a47e-dbdc08eb33e1", "type": "preheat", "status": "SUCCESS", "args": { "filter": "", "headers": null, "type": "image", "url": "https://registry-1.docker.io/v2/library/redis/manifests/latest" }} +{ + "id": 1, + "task_id": "group_4d1ea00e-740f-4dbf-a47e-dbdc08eb33e1", + "type": "preheat", + "status": "SUCCESS", + "args": { + "filter": "", + "headers": null, + "type": "image", + "url": "https://registry-1.docker.io/v2/library/redis/manifests/latest" + } +} ``` diff --git a/docs/en/preheat/console.md b/docs/en/preheat/console.md index 4faa41a6e29..0860bd92b25 100644 --- a/docs/en/preheat/console.md +++ b/docs/en/preheat/console.md @@ -1 +1,3 @@ -# WIP console \ No newline at end of file +# WIP console + +TODO diff --git a/docs/en/quick-start.md b/docs/en/quick-start.md index 98993a9dd27..5374459f3d6 100644 --- a/docs/en/quick-start.md +++ b/docs/en/quick-start.md @@ -1,34 +1,42 @@ # Dragonfly Quick Start -Dragonfly Quick Start document aims to help you to quick start Dragonfly journey. This experiment is quite easy and +Dragonfly Quick Start document aims to help you to +quick start Dragonfly journey. This experiment is quite easy and simplified. -You can have a quick start following [Kubernetes-with-Dragonfly](./deployment/installation/kubernetes/README.md). We recommend to use `Containerd with CRI` and `CRI-O` client. +You can have a quick start following +[Kubernetes-with-Dragonfly](./deployment/installation/kubernetes/README.md). +We recommend to use `Containerd with CRI` and `CRI-O` client. This table describes some container runtimes version and documents. + | Runtime | Version | Document | CRI Support | Pull Command | | --- | --- | --- | --- | --- | | Containerd* | v1.1.0+ | [Link](runtime-integration/containerd/mirror.md) | Yes | crictl pull docker.io/library/alpine:latest | | Containerd without CRI | < v1.1.0 | [Link](runtime-integration/containerd/proxy.md) | No | ctr image pull docker.io/library/alpine | | CRI-O | All | [Link](runtime-integration/cri-o.md) | Yes | crictl pull docker.io/library/alpine:latest | + **:`containerd` is recommended* + ## Runtime Configuration Guide for Dragonfly Helm Chart Dragonfly helm supports config docker automatically. Config cases: -**Implicit registries support** +### Implicit registries support Chart customize values.yaml: + ```yaml containerRuntime: docker: enable: true # -- Inject domains into /etc/hosts to force redirect traffic to dfdaemon. - # Caution: This feature need dfdaemon to implement SNI Proxy, confirm image tag is greater than v0.4.0. + # Caution: This feature need dfdaemon to implement SNI Proxy, + # confirm image tag is greater than v0.4.0. # When use certs and inject hosts in docker, no necessary to restart docker daemon. injectHosts: true registryDomains: @@ -36,15 +44,18 @@ containerRuntime: - "harbor.example.net" ``` -This config enables docker pulling images from registries `harbor.example.com` and `harbor.example.net` via Dragonfly. +This config enables docker pulling images from registries +`harbor.example.com` and `harbor.example.net` via Dragonfly. When deploying Dragonfly with above config, it's unnecessary to restart docker daemon. Limitations: + * Only support implicit registries ## Prepare Kubernetes Cluster -If there is no available Kubernetes cluster for testing, [minikube](https://minikube.sigs.k8s.io/docs/start/) is +If there is no available Kubernetes cluster for testing, +[minikube](https://minikube.sigs.k8s.io/docs/start/) is recommended. Just run `minikube start`. ## Install Dragonfly @@ -66,15 +77,19 @@ kubectl -n dragonfly-system wait --for=condition=ready --all --timeout=10m pod ## Manager Console -The console page will be displayed on `dragonfly-manager.dragonfly-system.svc.cluster.local:8080`. +The console page will be displayed on +`dragonfly-manager.dragonfly-system.svc.cluster.local:8080`. -If you need to bind Ingress, you can refer to [configuration options](https://artifacthub.io/packages/helm/dragonfly/dragonfly#values) of Helm Charts, or create it manually. +If you need to bind Ingress, you can refer to +[configuration options](https://artifacthub.io/packages/helm/dragonfly/dragonfly#values) +of Helm Charts, or create it manually. Console features preview reference document [console preview](design/manager.md). ## Using Dragonfly -After all above steps, create a new pod with target registry. Or just pull an image with `crictl`: +After all above steps, create a new pod with target registry. +Or just pull an image with `crictl`: ```shell crictl harbor.example.com/library/alpine:latest @@ -85,6 +100,7 @@ crictl pull docker.io/library/alpine:latest ``` After pulled images, find logs in dfdaemon pod: + ```shell # find pods kubectl -n dragonfly-system get pod -l component=dfdaemon @@ -94,8 +110,17 @@ kubectl -n dragonfly-system exec -it ${pod_name} -- grep "peer task done" /var/l ``` Example output: -``` -{"level":"info","ts":"2021-06-28 06:02:30.924","caller":"peer/peertask_stream_callback.go:77","msg":"stream peer task done, cost: 2838ms","peer":"172.17.0.9-1-ed7a32ae-3f18-4095-9f54-6ccfc248b16e","task":"3c658c488fd0868847fab30976c2a079d8fd63df148fb3b53fd1a418015723d7","component":"streamPeerTask"} + +```shell +{ + "level": "info", + "ts": "2021-06-28 06:02:30.924", + "caller": "peer/peertask_stream_callback.go:77", + "msg": "stream peer task done, cost: 2838ms", + "peer": "172.17.0.9-1-ed7a32ae-3f18-4095-9f54-6ccfc248b16e", + "task": "3c658c488fd0868847fab30976c2a079d8fd63df148fb3b53fd1a418015723d7", + "component": "streamPeerTask" +} ``` ## Preheat diff --git a/docs/en/runtime-integration/containerd/README.md b/docs/en/runtime-integration/containerd/README.md index 4405fcb6b7d..e1ea07e0ac7 100644 --- a/docs/en/runtime-integration/containerd/README.md +++ b/docs/en/runtime-integration/containerd/README.md @@ -1,6 +1,7 @@ # Containerd -This documentation will help you to integrate Dragonfly2 into Containerd. We recommend to use `mirror`. +This documentation will help you to integrate Dragonfly2 into +Containerd. We recommend to use `mirror`. Table of contents: diff --git a/docs/en/runtime-integration/containerd/mirror.md b/docs/en/runtime-integration/containerd/mirror.md index 3588a7759e9..cd9f5c7d7a0 100644 --- a/docs/en/runtime-integration/containerd/mirror.md +++ b/docs/en/runtime-integration/containerd/mirror.md @@ -1,12 +1,14 @@ # Use dfget daemon for containerd -From v1.1.0, Containerd supports registry mirrors, we can configure Containerd via this feature for HA. +From v1.1.0, Containerd supports registry mirrors, +we can configure Containerd via this feature for HA. ## Quick Start ### Step 1: Configure dfget daemon -To use dfget daemon as registry mirror, first you need to ensure configuration in `/etc/dragonfly/dfget.yaml`: +To use dfget daemon as registry mirror, +first you need to ensure configuration in `/etc/dragonfly/dfget.yaml`: ```yaml proxy: @@ -44,12 +46,14 @@ version = 2 endpoint = ["http://127.0.0.1:65001","https://registry-1.docker.io"] ``` -In this config, there is two mirror endpoints for "docker.io", Containerd will pull images with `http://127.0.0.1:65001` first. -If `http://127.0.0.1:65001` is not available, the default `https://registry-1.docker.io` will be used for HA. +In this config, there is two mirror endpoints for "docker.io", +Containerd will pull images with `http://127.0.0.1:65001` first. +If `http://127.0.0.1:65001` is not available, +the default `https://registry-1.docker.io` will be used for HA. -> More details about Containerd configuration: https://github.com/containerd/containerd/blob/v1.5.2/docs/cri/registry.md#configure-registry-endpoint - -> Containerd has deprecated the above config from v1.4.0, new format for reference: https://github.com/containerd/containerd/blob/v1.5.2/docs/cri/config.md#registry-configuration +> More details about Containerd configuration: +> Containerd has deprecated the above config from v1.4.0, + new format for reference: ### Option 2: Multiple Registries @@ -89,17 +93,17 @@ server = "https://example.com" ##### Option 2: Generate hosts.toml automatically -You can also generate hosts.toml with https://github.com/dragonflyoss/Dragonfly2/blob/main/hack/gen-containerd-hosts.sh +You can also generate hosts.toml with ```shell bash gen-containerd-hosts.sh example.com ``` -> More details about registry configuration: https://github.com/containerd/containerd/blob/main/docs/hosts.md#registry-configuration---examples +> More details about registry configuration: ## Step 3: Restart Containerd Daemon -``` +```shell systemctl restart containerd ``` @@ -107,20 +111,28 @@ systemctl restart containerd You can pull image like this: -``` +```shell crictl pull docker.io/library/busybox ``` ## Step 5: Validate Dragonfly -You can execute the following command to check if the busybox image is distributed via Dragonfly. +You can execute the following command to check +if the busybox image is distributed via Dragonfly. -```bash +```shell grep 'register peer task result' /var/log/dragonfly/daemon/*.log ``` If the output of command above has content like +```shell +{ + "level": "info", + "ts": "2021-02-23 20:03:20.306", + "caller": "client/client.go:83", + "msg": "register peer task result:true[200] for taskId:adf62a86f001e17037eedeaaba3393f3519b80ce,peerIp:10.15.233.91,securityDomain:,idc:,scheduler:127.0.0.1:8002", + "peerId": "10.15.233.91-65000-43096-1614081800301788000", + "errMsg": null +} ``` -{"level":"info","ts":"2021-02-23 20:03:20.306","caller":"client/client.go:83","msg":"register peer task result:true[200] for taskId:adf62a86f001e17037eedeaaba3393f3519b80ce,peerIp:10.15.233.91,securityDomain:,idc:,scheduler:127.0.0.1:8002","peerId":"10.15.233.91-65000-43096-1614081800301788000","errMsg":null} -``` \ No newline at end of file diff --git a/docs/en/runtime-integration/containerd/proxy.md b/docs/en/runtime-integration/containerd/proxy.md index 8e09ad28c51..4cfe047db35 100644 --- a/docs/en/runtime-integration/containerd/proxy.md +++ b/docs/en/runtime-integration/containerd/proxy.md @@ -13,35 +13,36 @@ Generate a CA certificate private key. openssl genrsa -out ca.key 2048 ``` -Open openssl config file `openssl.conf`. Note set `basicConstraints` to true, that you can modify the values. +Open openssl config file `openssl.conf`. +Note set `basicConstraints` to true, that you can modify the values. ```text [ req ] -#default_bits = 2048 -#default_md = sha256 -#default_keyfile = privkey.pem -distinguished_name = req_distinguished_name -attributes = req_attributes +#default_bits = 2048 +#default_md = sha256 +#default_keyfile = privkey.pem +distinguished_name = req_distinguished_name +attributes = req_attributes extensions = v3_ca req_extensions = v3_ca [ req_distinguished_name ] -countryName = Country Name (2 letter code) -countryName_min = 2 -countryName_max = 2 -stateOrProvinceName = State or Province Name (full name) -localityName = Locality Name (eg, city) -0.organizationName = Organization Name (eg, company) -organizationalUnitName = Organizational Unit Name (eg, section) -commonName = Common Name (eg, fully qualified host name) -commonName_max = 64 -emailAddress = Email Address -emailAddress_max = 64 +countryName = Country Name (2 letter code) +countryName_min = 2 +countryName_max = 2 +stateOrProvinceName = State or Province Name (full name) +localityName = Locality Name (eg, city) +0.organizationName = Organization Name (eg, company) +organizationalUnitName = Organizational Unit Name (eg, section) +commonName = Common Name (eg, fully qualified host name) +commonName_max = 64 +emailAddress = Email Address +emailAddress_max = 64 [ req_attributes ] -challengePassword = A challenge password -challengePassword_min = 4 -challengePassword_max = 20 +challengePassword = A challenge password +challengePassword_min = 4 +challengePassword_max = 20 [ v3_ca ] basicConstraints = CA:TRUE @@ -51,13 +52,15 @@ Generate the CA certificate. ```bash openssl req -new -key ca.key -nodes -out ca.csr -config openssl.conf -openssl x509 -req -days 36500 -extfile openssl.conf -extensions v3_ca -in ca.csr -signkey ca.key -out ca.crt +openssl x509 -req -days 36500 -extfile openssl.conf \ + -extensions v3_ca -in ca.csr -signkey ca.key -out ca.crt ``` ### Step 2: Configure dfget daemon To use dfget daemon as HTTP proxy, first you need to append a proxy rule in -`/etc/dragonfly/dfget.yaml`, This will proxy `your.private.registry`'s requests for image layers: +`/etc/dragonfly/dfget.yaml`, This will proxy +`your.private.registry`'s requests for image layers: ```yaml proxy: @@ -81,7 +84,7 @@ proxy: Set dfget damone as `HTTP_PROXY` and `HTTPS_PROXY` for containerd in `/etc/systemd/system/containerd.service.d/http-proxy.conf`: -``` +```toml [Service] Environment="HTTP_PROXY=http://127.0.0.1:65001" Environment="HTTPS_PROXY=http://127.0.0.1:65001" @@ -144,6 +147,7 @@ proxy: You can get the certificate of your server with: +```bash +openssl x509 -in <(openssl s_client -showcerts \ + -servername your.domain.com -connect your.domain.com:443 -prexit 2>/dev/null) ``` -openssl x509 -in <(openssl s_client -showcerts -servername xxx -connect xxx:443 -prexit 2>/dev/null) -``` \ No newline at end of file diff --git a/docs/en/runtime-integration/cri-o.md b/docs/en/runtime-integration/cri-o.md index 0fd823b9126..7d0405246eb 100644 --- a/docs/en/runtime-integration/cri-o.md +++ b/docs/en/runtime-integration/cri-o.md @@ -2,7 +2,8 @@ ## Step 1: Validate Dragonfly Configuration -To use dfget daemon as registry mirror, first you need to ensure configuration in `/etc/dragonfly/dfget.yaml`: +To use dfget daemon as registry mirror, +first you need to ensure configuration in `/etc/dragonfly/dfget.yaml`: ```yaml proxy: @@ -34,32 +35,41 @@ location = "docker.io" ## Step 3: Restart CRI-O Daemon -``` +```shell systemctl restart crio ``` If encounter error like these: -`mixing sysregistry v1/v2 is not supported` or `registry must be in v2 format but is in v1`, +`mixing sysregistry v1/v2 is not supported` or +`registry must be in v2 format but is in v1`, please convert your registries configuration to v2. ## Step 4: Pull Image You can pull image like this: -``` +```shell crictl pull docker.io/library/busybox ``` ## Step 5: Validate Dragonfly -You can execute the following command to check if the busybox image is distributed via Dragonfly. +You can execute the following command to +check if the busybox image is distributed via Dragonfly. -```bash +```shell grep 'register peer task result' /var/log/dragonfly/daemon/*.log ``` If the output of command above has content like -``` -{"level":"info","ts":"2021-02-23 20:03:20.306","caller":"client/client.go:83","msg":"register peer task result:true[200] for taskId:adf62a86f001e17037eedeaaba3393f3519b80ce,peerIp:10.15.233.91,securityDomain:,idc:,scheduler:127.0.0.1:8002","peerId":"10.15.233.91-65000-43096-1614081800301788000","errMsg":null} +```shell +{ + "level": "info", + "ts": "2021-02-23 20:03:20.306", + "caller": "client/client.go:83", + "msg": "register peer task result:true[200] for taskId:adf62a86f001e17037eedeaaba3393f3519b80ce,peerIp:10.15.233.91,securityDomain:,idc:,scheduler:127.0.0.1:8002", + "peerId": "10.15.233.91-65000-43096-1614081800301788000", + "errMsg": null +} ``` diff --git a/docs/en/runtime-integration/docker.md b/docs/en/runtime-integration/docker.md index d1be4a088c1..5cf6916da62 100644 --- a/docs/en/runtime-integration/docker.md +++ b/docs/en/runtime-integration/docker.md @@ -13,33 +13,34 @@ Generate a CA certificate private key. openssl genrsa -out ca.key 2048 ``` -Open openssl config file `openssl.conf`. Note set `basicConstraints` to true, that you can modify the values. +Open openssl config file `openssl.conf`. +Note set `basicConstraints` to true, that you can modify the values. ```text [ req ] -#default_bits = 2048 -#default_md = sha256 -#default_keyfile = privkey.pem -distinguished_name = req_distinguished_name -attributes = req_attributes +#default_bits = 2048 +#default_md = sha256 +#default_keyfile = privkey.pem +distinguished_name = req_distinguished_name +attributes = req_attributes extensions = v3_ca req_extensions = v3_ca [ req_distinguished_name ] -countryName = Country Name (2 letter code) -countryName_min = 2 -countryName_max = 2 -stateOrProvinceName = State or Province Name (full name) -localityName = Locality Name (eg, city) -0.organizationName = Organization Name (eg, company) -organizationalUnitName = Organizational Unit Name (eg, section) -commonName = Common Name (eg, fully qualified host name) -commonName_max = 64 -emailAddress = Email Address -emailAddress_max = 64 +countryName = Country Name (2 letter code) +countryName_min = 2 +countryName_max = 2 +stateOrProvinceName = State or Province Name (full name) +localityName = Locality Name (eg, city) +0.organizationName = Organization Name (eg, company) +organizationalUnitName = Organizational Unit Name (eg, section) +commonName = Common Name (eg, fully qualified host name) +commonName_max = 64 +emailAddress = Email Address +emailAddress_max = 64 [ req_attributes ] -challengePassword = A challenge password -challengePassword_min = 4 -challengePassword_max = 20 +challengePassword = A challenge password +challengePassword_min = 4 +challengePassword_max = 20 [ v3_ca ] basicConstraints = CA:TRUE ``` @@ -48,13 +49,15 @@ Generate the CA certificate. ```bash openssl req -new -key ca.key -nodes -out ca.csr -config openssl.conf -openssl x509 -req -days 36500 -extfile openssl.conf -extensions v3_ca -in ca.csr -signkey ca.key -out ca.crt +openssl x509 -req -days 36500 -extfile openssl.conf \ + -extensions v3_ca -in ca.csr -signkey ca.key -out ca.crt ``` ### Step 2: Configure dfget daemon To use dfget daemon as HTTP proxy, first you need to append a proxy rule in -`/var/log/dragonfly/dfget.yaml`, This will proxy `your.private.registry`'s requests for image layers: +`/var/log/dragonfly/dfget.yaml`, +This will proxy `your.private.registry`'s requests for image layers: ```yaml proxy: @@ -89,7 +92,7 @@ Add your private registry to `insecure-registries` in Set dfdaemon as `HTTP_PROXY` and `HTTPS_PROXY` for docker daemon in `/etc/systemd/system/docker.service.d/http-proxy.conf`: -``` +```toml [Service] Environment="HTTP_PROXY=http://127.0.0.1:65001" Environment="HTTPS_PROXY=http://127.0.0.1:65001" @@ -152,6 +155,7 @@ proxy: You can get the certificate of your server with: -``` -openssl x509 -in <(openssl s_client -showcerts -servername xxx -connect xxx:443 -prexit 2>/dev/null) +```shell +openssl x509 -in <(openssl s_client -showcerts \ + -servername your.domain.com -connect your.domain.com:443 -prexit 2>/dev/null) ``` diff --git a/docs/en/test-guide/test-guide.md b/docs/en/test-guide/test-guide.md index e25bea1feb6..02925426768 100644 --- a/docs/en/test-guide/test-guide.md +++ b/docs/en/test-guide/test-guide.md @@ -9,13 +9,13 @@ Unit tests is in the project directory. ### Running unit tests ```bash -$ make test +make test ``` ### Runing uint tests with coverage reports ```bash -$ make test-coverage +make test-coverage ``` ## E2E tests @@ -25,17 +25,17 @@ E2E tests is in `test/e2e` path. ### Running E2E tests ```bash -$ make e2e-test +make e2e-test ``` ### Runing E2E tests with coverage reports ```bash -$ make e2e-test-coverage +make e2e-test-coverage ``` ### Clean E2E tests environment ```bash -$ make clean-e2e-test +make clean-e2e-test ``` diff --git a/docs/en/troubleshooting/README.md b/docs/en/troubleshooting/README.md index 5ec94b5e10c..d2242f995f1 100644 --- a/docs/en/troubleshooting/README.md +++ b/docs/en/troubleshooting/README.md @@ -4,16 +4,17 @@ 1. Confirm limit rate in [dfget.yaml](https://github.com/dragonflyoss/Dragonfly2/blob/main/docs/en/deployment/configuration/dfget.yaml#L65) -```yaml -download: - # total download limit per second - totalRateLimit: 200Mi - # per peer task download limit per second - perPeerRateLimit: 100Mi # default is 20Mi, this default is in consideration of extreme environments. -upload: - # upload limit per second - rateLimit: 100Mi -``` + ```yaml + download: + # total download limit per second + totalRateLimit: 200Mi + # per peer task download limit per second + # default is 20Mi, this default is in consideration of extreme environments + perPeerRateLimit: 100Mi + upload: + # upload limit per second + rateLimit: 100Mi + ``` 2. Confirm source connection speed in CDN and dfdaemon @@ -23,6 +24,7 @@ upload: 2. Check source connectivity(dns error or certificate error) Example: + ```shell curl https://example.harbor.local/ ``` diff --git a/docs/zh-CN/FAQ.md b/docs/zh-CN/FAQ.md deleted file mode 100644 index 32cce9075d6..00000000000 --- a/docs/zh-CN/FAQ.md +++ /dev/null @@ -1 +0,0 @@ -# FAQ \ No newline at end of file diff --git a/docs/zh-CN/README.md b/docs/zh-CN/README.md index 7bf12bde170..d6bb9670c59 100644 --- a/docs/zh-CN/README.md +++ b/docs/zh-CN/README.md @@ -1,6 +1,9 @@ # 蜻蜓文档 -Dragonfly Document 是关于 Dragonfly 的介绍。对于一般对 Dragonfly 感兴趣的人，这个 repo 的 README.md 就足够了。对于最终用户来说，`/docs` 中包含的所有详细信息是最好的指南。而对于开发者来说，[开发者指南](developer-guide/developer-guide.md)部分的内容才是最需要的。 +Dragonfly Document 是关于 Dragonfly 的介绍。对于一般对 Dragonfly 感兴趣的人， +这个 repo 的 README.md 就足够了。对于最终用户来说 +`/docs` 中包含的所有详细信息是最好的指南。 +而对于开发者来说，[开发者指南](developer-guide/developer-guide.md)部分的内容才是最需要的。 文档文件组织如下： diff --git a/docs/zh-CN/ROADMAP.md b/docs/zh-CN/ROADMAP.md deleted file mode 100644 index 971d14480b7..00000000000 --- a/docs/zh-CN/ROADMAP.md +++ /dev/null @@ -1 +0,0 @@ -# Roadmap \ No newline at end of file diff --git a/docs/zh-CN/cli-reference/README.md b/docs/zh-CN/cli-reference/README.md index a17fe559eef..7e5801cda3b 100644 --- a/docs/zh-CN/cli-reference/README.md +++ b/docs/zh-CN/cli-reference/README.md @@ -1,10 +1,14 @@ # CLI Reference -对于几乎所有用户，命令行是您可能需要的第一个参考。 [CLI Reference](cli-reference)目录下的文档是关于Dragonfly CLI的命令详细使用，包括`dfget`、`cdn`、`scheduler`和`manager`。您可以获得有关命令的介绍、概要、示例和选项。最后但同样重要的是，Dragonfly 可以保证命令行文档与 Dragonfly CLI 的源代码高度一致。更重要的是，所有命令行文档都是通过源代码自动生成的。 +对于几乎所有用户，命令行是您可能需要的第一个参考。 +[CLI Reference](cli-reference)目录下的文档是关于Dragonfly CLI的命令详细使用， +包括`dfget`、`cdn`、`scheduler`和`manager`。您可以获得有关命令的介绍、概要、示例和选项。 +最后但同样重要的是，Dragonfly 可以保证命令行文档与 Dragonfly CLI +的源代码高度一致。更重要的是，所有命令行文档都是通过源代码自动生成的。 内容列表: * [dfget](dfget.md) * [cdn](cdn.md) * [scheduler](scheduler.md) -* [manager](manager.md) \ No newline at end of file +* [manager](manager.md) diff --git a/docs/zh-CN/cli-reference/cdn.md b/docs/zh-CN/cli-reference/cdn.md index 647840ba6a3..7aec15f89f1 100644 --- a/docs/zh-CN/cli-reference/cdn.md +++ b/docs/zh-CN/cli-reference/cdn.md @@ -4,14 +4,14 @@ CDN 是一个长时间运行的服务进程，它缓存从源下载的数据， ## 用法 -``` +```text cdn [flags] cdn [command] ``` ## 可用子命令 -``` +```text doc 生成帮助文档 help 命令的帮助文档 version 查看 cdn 当前版本 @@ -19,7 +19,8 @@ version 查看 cdn 当前版本 ## 可选参数 -``` + +```text --config string 配置文件地址 (默认值为/Users/${USER_HOME}/.dragonfly/config/cdn.yaml)，也可以使用环境变量 cdn_config 进行配置 --console logger 日志是否输出记录到标准输出 -h, --help 显示 cdn 命令帮助信息 @@ -27,5 +28,5 @@ version 查看 cdn 当前版本 --pprof-port int pprof 监听的端口，为 0 时使用随机端口（默认值为 -1） --service-name string tracer分布式追踪日志中的服务名称 (默认 "dragonfly-scheduler") --verbose 是否使用debug级别的日志输出 - ``` + diff --git a/docs/zh-CN/cli-reference/dfget.md b/docs/zh-CN/cli-reference/dfget.md index 54b1ea6c590..da0f1f9ad44 100644 --- a/docs/zh-CN/cli-reference/dfget.md +++ b/docs/zh-CN/cli-reference/dfget.md @@ -1,4 +1,6 @@ -# dfget +# dfdaemon + +## dfget `dfget` 是Dragonfly用以下载和上传文件的客户端。 @@ -8,23 +10,24 @@ dfget 是 Dragonfly 的一个客户端，也是p2p网络中的一个peer。当 dfget 将从其他peer下载文件。同时，它也能作为上传者，以支持其他peer对它已拥有对部分对下载。 此外，dfget 还提供了一些高级功能，如网络带宽限制、加密传输等。 -``` +```text dfget [flags] ``` ### Example -``` +```text dfget --schedulers 127.0.0.1:8002 -o /path/to/output -u "http://example.com/object" ``` -## Log configuration +### Log configuration 如果您想要在Terminal中查看日志，请使用 --console 将环境变量 console 设置为 true。 -### Options +#### Options -``` + +```text --alivetime duration alive duration for which uploader keeps no accessing by any uploading requests, after this period uploader will automatically exit (default 5m0s) --cacerts strings the cacert file which is used to verify remote server when supernode interact with the source. --callsystem string the name of dfget caller which is for debugging. Once set, it will be passed to all components around the request to make debugging easy @@ -56,18 +59,19 @@ dfget --schedulers 127.0.0.1:8002 -o /path/to/output -u "http://example.com/obje --totallimit ratelimit network bandwidth rate limit for the whole host, in format of G(B)/g/M(B)/m/K(B)/k/B, pure number will also be parsed as Byte (default 104857600.000000) -u, --url string URL of user requested downloading file(only HTTP/HTTPs supported) --verbose enable verbose mode, all debug log will be display - ``` + -# dfget daemon +## dfget daemon -## Log configuration +### Daemon Log configuration 如果您想要在Terminal中查看日志，请使用 --console 将环境变量 console 设置为 true。 -### Options +#### Daemon Options -``` + +```text --advertise-ip string the ip report to scheduler, normal same with listen ip (default "10.15.232.63") --alivetime duration alive duration for which uploader keeps no accessing by any uploading requests, after this period uploader will automatically exit (default 5m0s) --data string local directory which stores temporary files for p2p uploading @@ -95,3 +99,4 @@ dfget --schedulers 127.0.0.1:8002 -o /path/to/output -u "http://example.com/obje --upload-rate ratelimit upload rate limit for other peers (default 104857600.000000) --verbose print verbose log and enable golang debug info ``` + diff --git a/docs/zh-CN/cli-reference/manager.md b/docs/zh-CN/cli-reference/manager.md index 5466b58990b..49b1945cbd6 100644 --- a/docs/zh-CN/cli-reference/manager.md +++ b/docs/zh-CN/cli-reference/manager.md @@ -1,55 +1,60 @@ -## Manager +# Manager Manager 是一个常驻后台运行的进程，它在蜻蜓中扮演各个子系统集群大脑的角色， 用于管理各个系统模块依赖的动态配置，以及提供心跳保活、监控大盘和产品化的功能。 -### 用法 +## 用法 -``` +```text manager [Option] ``` -### 日志配置 +## 日志配置 -``` +```text 1. 终端日志输出： --console 2. 日志目录: /var/log/dragonfly/manager/ ``` -### 指标输出 +## 指标输出 -``` +```text 启动时配置参数: --pprof-port port ``` -### Swagger支持 +## Swagger支持 端点: /swagger/doc.json -### Prometheus集成 +## Prometheus集成 端点: /metrics -### 健康检查 +## 健康检查 端点: /healthy/ -### jaeger集成 +## jaeger集成 -``` +```text 启动时配置 --jaeger http://localhost:14250/api/traces ``` -### 可选参数 +## 可选参数 -``` - --config string the path of configuration file with yaml extension name, default is /etc/dragonfly/manager.yaml, it can also be set by env var: MANAGER_CONFIG +```text + --config string the path of configuration file with + yaml extension name, default is + /etc/dragonfly/manager.yaml, + it can also be set by env var: MANAGER_CONFIG --console whether logger output records to the stdout -h, --help help for manager --jaeger string jaeger endpoint url, like: http://localhost:14250/api/traces - --pprof-port int listen port for pprof, 0 represents random port (default -1) + --pprof-port int listen port for pprof, + 0 represents random port (default -1) --service-name string name of the service for tracer (default "dragonfly-manager") --verbose whether logger use debug level ``` -### 配置文件 -[配置样例](../config/manager.yaml) \ No newline at end of file +## 配置文件 + +[配置样例](../config/manager.yaml) diff --git a/docs/zh-CN/cli-reference/scheduler.md b/docs/zh-CN/cli-reference/scheduler.md index 20b673e1b18..f934989e0b4 100644 --- a/docs/zh-CN/cli-reference/scheduler.md +++ b/docs/zh-CN/cli-reference/scheduler.md @@ -1,17 +1,18 @@ # Scheduler -Scheduler 是一个常驻后台运行的进程，用于接收和管理客户端的下载任务，通知CDN进行回源， 在下载过程中生成维护P2P网络，给客户端推送适合的下载节点 +Scheduler 是一个常驻后台运行的进程，用于接收和管理客户端的下载任务， +通知CDN进行回源， 在下载过程中生成维护P2P网络，给客户端推送适合的下载节点 ## 用法 -``` +```text scheduler [flags] scheduler [command] ``` ## 可用子命令 -``` +```text doc 生成帮助文档 help 命令的帮助文档 version 查看 scheduler 当前版本 @@ -19,12 +20,13 @@ version 查看 scheduler 当前版本 ## 可选参数 +```text +--config string 配置文件地址 (默认值为/Users/${USER_HOME}/.dragonfly/config/scheduler.yaml)， + 也可以使用环境变量 scheduler_config 进行配置 +--console logger 日志是否输出记录到标准输出 +-h, --help 显示 scheduler 命令帮助信息 +--jaeger string jaeger 收集器 endpoint url, 例如: http://localhost:14250/api/traces +--pprof-port int pprof 监听的端口，为 0 时使用随机端口（默认值为 -1） +--service-name string tracer分布式追踪日志中的服务名称 (默认 "dragonfly-scheduler") +--verbose 是否使用debug级别的日志输出 ``` - --config string 配置文件地址 (默认值为/Users/${USER_HOME}/.dragonfly/config/scheduler.yaml)，也可以使用环境变量 scheduler_config 进行配置 - --console logger 日志是否输出记录到标准输出 - -h, --help 显示 scheduler 命令帮助信息 - --jaeger string jaeger 收集器 endpoint url, 例如: http://localhost:14250/api/traces - --pprof-port int pprof 监听的端口，为 0 时使用随机端口（默认值为 -1） - --service-name string tracer分布式追踪日志中的服务名称 (默认 "dragonfly-scheduler") - --verbose 是否使用debug级别的日志输出 -``` \ No newline at end of file diff --git a/docs/zh-CN/deployment/README.md b/docs/zh-CN/deployment/README.md index 43d5bfb693b..03b6a43c0f3 100644 --- a/docs/zh-CN/deployment/README.md +++ b/docs/zh-CN/deployment/README.md @@ -7,7 +7,8 @@ ## 安装 -安装文档为您提供了几个可以用部署 Dragonfly 的方式，例如 [Kubernetes](installation/kubernetes/README.md) 和 [Docker](installation/docker/README.md)。 +安装文档为您提供了几个可以用部署 Dragonfly 的方式， +例如 [Kubernetes](installation/kubernetes/README.md) 和 [Docker](installation/docker/README.md)。 ## 配置 diff --git a/docs/zh-CN/deployment/installation/docker/standalone/README.md b/docs/zh-CN/deployment/installation/docker/standalone/README.md index 5833759d699..f7a24dcd945 100644 --- a/docs/zh-CN/deployment/installation/docker/standalone/README.md +++ b/docs/zh-CN/deployment/installation/docker/standalone/README.md @@ -4,7 +4,7 @@ ## 使用 Helm Charts 进行安装 -> ArtifactHub 仓库链接为: https://artifacthub.io/packages/helm/dragonfly/dragonfly +> ArtifactHub 仓库链接为: 推荐使用 Helm Charts 进行安装，简单且快捷。参考文档 [Kubernetes-with-Dragonfly](../../kubernetes/README.md)。 diff --git a/docs/zh-CN/deployment/installation/docker/standalone/cdn.md b/docs/zh-CN/deployment/installation/docker/standalone/cdn.md index 3a46d0c1ea4..1a4e07c8779 100644 --- a/docs/zh-CN/deployment/installation/docker/standalone/cdn.md +++ b/docs/zh-CN/deployment/installation/docker/standalone/cdn.md @@ -19,44 +19,44 @@ Docker|1.12.0+ 1. 获取最新的 CDN 镜像 -```sh -docker pull dragonflyoss/cdn -``` + ```sh + docker pull dragonflyoss/cdn + ``` 或者您可以构建自己的 CDN 镜像 1. 获取 Dragonfly 的源码 -```sh -git clone https://github.com/dragonflyoss/Dragonfly2.git -``` + ```sh + git clone https://github.com/dragonflyoss/Dragonfly2.git + ``` 2. 打开项目文件夹 -```sh -cd Dragonfly2 -``` + ```sh + cd Dragonfly2 + ``` 3. 构建 CDN 的 Docker 镜像 -```sh -TAG="2.0.0" -make docker-build-cdn D7Y_VERSION=$TAG -``` + ```sh + TAG="2.0.0" + make docker-build-cdn D7Y_VERSION=$TAG + ``` 4. 获取最新的 CDN 镜像 ID -```sh -docker image ls | grep 'cdn' | awk '{print $3}' | head -n1 -``` + ```sh + docker image ls | grep 'cdn' | awk '{print $3}' | head -n1 + ``` ### 启动 cdn **注意：** 需要使用上述步骤获得的 ID 替换 ${cdnDockerImageId}。 ```sh -docker run -d --name cdn --restart=always -p 8001:8001 -p 8003:8003 -v /home/admin/ftp:/home/admin/ftp ${cdnDockerImageId} ---download-port=8001 +docker run -d --name cdn --restart=always -p 8001:8001 \ + -p 8003:8003 -v /home/admin/ftp:/home/admin/ftp ${cdnDockerImageId} --download-port=8001 ``` CDN 部署完成之后，运行以下命令以检查 Nginx 和 **cdn** 是否正在运行，以及 `8001` 和 `8003` 端口是否可用。 diff --git a/docs/zh-CN/deployment/installation/docker/standalone/dfdaemon.md b/docs/zh-CN/deployment/installation/docker/standalone/dfdaemon.md index 7d8f38cdf71..908f93bd22b 100644 --- a/docs/zh-CN/deployment/installation/docker/standalone/dfdaemon.md +++ b/docs/zh-CN/deployment/installation/docker/standalone/dfdaemon.md @@ -19,43 +19,44 @@ Docker|1.12.0+ 1. 获取最新的 dfdaemon 镜像 -```sh -docker pull dragonflyoss/dfdaemon -``` + ```sh + docker pull dragonflyoss/dfdaemon + ``` 或者您可以构建自己的 dfdaemon 镜像 1. 获取 Dragonfly 的源码 -```sh -git clone https://github.com/dragonflyoss/Dragonfly2.git -``` + ```sh + git clone https://github.com/dragonflyoss/Dragonfly2.git + ``` 2. 打开项目文件夹 -```sh -cd Dragonfly2 -``` + ```sh + cd Dragonfly2 + ``` 3. 构建 dfdaemon 的 Docker 镜像 -```sh -TAG="2.0.0" -make docker-build-dfdaemon D7Y_VERSION=$TAG -``` + ```sh + TAG="2.0.0" + make docker-build-dfdaemon D7Y_VERSION=$TAG + ``` 4. 获取最新的 dfdaemon 镜像 ID -```sh -docker image ls | grep 'dfdaemon' | awk '{print $3}' | head -n1 -``` + ```sh + docker image ls | grep 'dfdaemon' | awk '{print $3}' | head -n1 + ``` ### 启动 dfdaemon **注意：** 需要使用上述步骤获得的 ID 替换 ${dfdaemonDockerImageId}。 ```sh -docker run -d --name dfdaemon --restart=always -p 65000:65000 -p 65001:65001 -p 65002:65002 ${dfdaemonDockerImageId} daemon +docker run -d --name dfdaemon --restart=always -p 65000:65000 \ + -p 65001:65001 -p 65002:65002 ${dfdaemonDockerImageId} daemon ``` dfget 部署完成之后，运行以下命令以检查 **dfdaemon** 是否正在运行，以及 `65000`, `65001` 和 `65002` 端口是否可用。 diff --git a/docs/zh-CN/deployment/installation/docker/standalone/manager.md b/docs/zh-CN/deployment/installation/docker/standalone/manager.md index 4c4e398fef0..83f58548022 100644 --- a/docs/zh-CN/deployment/installation/docker/standalone/manager.md +++ b/docs/zh-CN/deployment/installation/docker/standalone/manager.md @@ -19,36 +19,36 @@ Docker|1.12.0+ 1. 获取最新的 Manager 镜像 -```sh -docker pull dragonflyoss/manager -``` + ```sh + docker pull dragonflyoss/manager + ``` 或者您可以构建自己的 manager 镜像 1. 获取 Dragonfly 的源码 -```sh -git clone https://github.com/dragonflyoss/Dragonfly2.git -``` + ```sh + git clone https://github.com/dragonflyoss/Dragonfly2.git + ``` 2. 打开项目文件夹 -```sh -cd Dragonfly2 -``` + ```sh + cd Dragonfly2 + ``` 3. 构建 manager 的 Docker 镜像 -```sh -TAG="2.0.0" -make docker-build-manager D7Y_VERSION=$TAG -``` + ```sh + TAG="2.0.0" + make docker-build-manager D7Y_VERSION=$TAG + ``` 4. 获取最新的 manager 镜像 ID -```sh -docker image ls | grep 'manager' | awk '{print $3}' | head -n1 -``` + ```sh + docker image ls | grep 'manager' | awk '{print $3}' | head -n1 + ``` ### 启动 manager diff --git a/docs/zh-CN/deployment/installation/docker/standalone/scheduler.md b/docs/zh-CN/deployment/installation/docker/standalone/scheduler.md index 188d776838a..9675526ac4e 100644 --- a/docs/zh-CN/deployment/installation/docker/standalone/scheduler.md +++ b/docs/zh-CN/deployment/installation/docker/standalone/scheduler.md @@ -19,36 +19,36 @@ Docker|1.12.0+ 1. 获取最新的 scheduler 镜像 -```sh -docker pull dragonflyoss/scheduler -``` + ```sh + docker pull dragonflyoss/scheduler + ``` 或者您可以构建自己的 scheduler 镜像 1. 获取 Dragonfly 的源码 -```sh -git clone https://github.com/dragonflyoss/Dragonfly2.git -``` + ```sh + git clone https://github.com/dragonflyoss/Dragonfly2.git + ``` 2. 打开项目文件夹 -```sh -cd Dragonfly2 -``` + ```sh + cd Dragonfly2 + ``` 3. 构建 scheduler 的 Docker 镜像 -```sh -TAG="2.0.0" -make docker-build-scheduler D7Y_VERSION=$TAG -``` + ```sh + TAG="2.0.0" + make docker-build-scheduler D7Y_VERSION=$TAG + ``` 4. 获取最新的 scheduler 镜像 ID -```sh -docker image ls | grep 'scheduler' | awk '{print $3}' | head -n1 -``` + ```sh + docker image ls | grep 'scheduler' | awk '{print $3}' | head -n1 + ``` ### 启动 scheduler diff --git a/docs/zh-CN/deployment/installation/executable-files/cdn.md b/docs/zh-CN/deployment/installation/executable-files/cdn.md index 9112c321635..5f4714f3aa3 100644 --- a/docs/zh-CN/deployment/installation/executable-files/cdn.md +++ b/docs/zh-CN/deployment/installation/executable-files/cdn.md @@ -16,20 +16,21 @@ Nginx|0.8+ ### 获取 CDN 可执行文件 -1. 下载 Dragonfly 项目的压缩包。您可以从 [github releases page](https://github. - com/dragonflyoss/Dragonfly2/releases) 下载一个已发布的最近版本 - -```sh -version=2.0.0 -wget https://github.com/dragonflyoss/Dragonfly2/releases/download/v$version/Dragonfly2_$version_linux_amd64.tar.gz -``` +1. 下载 Dragonfly 项目的压缩包。您可以从 +[github releases page](https://github.com/dragonflyoss/Dragonfly2/releases) +下载一个已发布的最近版本 + + ```sh + version=2.0.0 + wget https://github.com/dragonflyoss/Dragonfly2/releases/download/v$version/Dragonfly2_$version_linux_amd64.tar.gz + ``` 2. 解压压缩包 -```bash -# Replace `xxx` with the installation directory. -tar -zxf Dragonfly2_2.0.0_linux_amd64.tar.gz -C xxx -``` + ```bash + # Replace `xxx` with the installation directory. + tar -zxf Dragonfly2_2.0.0_linux_amd64.tar.gz -C xxx + ``` 3. 把 `cdn` 移动到环境变量 `PATH` 下以确保您可以直接使用 `cdn` 命令 @@ -37,21 +38,21 @@ tar -zxf Dragonfly2_2.0.0_linux_amd64.tar.gz -C xxx 1. 获取 Dragonfly 的源码 -```sh -git clone https://github.com/dragonflyoss/Dragonfly2.git -``` + ```sh + git clone https://github.com/dragonflyoss/Dragonfly2.git + ``` 2. 打开项目文件夹 -```sh -cd Dragonfly2 -``` + ```sh + cd Dragonfly2 + ``` 3. 编译源码 -```sh -make build-cdn && make install-cdn -``` + ```sh + make build-cdn && make install-cdn + ``` ### 启动 cdn @@ -72,26 +73,26 @@ cdn --port=8003 --download-port=$cdnDownloadPort 1. 将下面的配置添加到 Nginx 配置文件中 -```conf -server { -# Must be ${cdnDownloadPort} -listen 8001; -location / { - # Must be ${cdnHomeDir}/ftp - root /home/admin/ftp; -} -} -``` + ```conf + server { + # Must be ${cdnDownloadPort} + listen 8001; + location / { + # Must be ${cdnHomeDir}/ftp + root /home/admin/ftp; + } + } + ``` 2. 启动 Nginx. -```sh -sudo nginx -``` + ```sh + sudo nginx + ``` -CDN 部署完成之后，运行以下命令以检查 Nginx 和 **cdn** 是否正在运行，以及 `8001` 和 `8003` 端口是否可用。 + CDN 部署完成之后，运行以下命令以检查 Nginx 和 **cdn** 是否正在运行，以及 `8001` 和 `8003` 端口是否可用。 -```sh -telnet 127.0.0.1 8001 -telnet 127.0.0.1 8003 -``` + ```sh + telnet 127.0.0.1 8001 + telnet 127.0.0.1 8003 + ``` diff --git a/docs/zh-CN/deployment/installation/executable-files/dfdaemon.md b/docs/zh-CN/deployment/installation/executable-files/dfdaemon.md index 660754ea90a..7bcd9d1a628 100644 --- a/docs/zh-CN/deployment/installation/executable-files/dfdaemon.md +++ b/docs/zh-CN/deployment/installation/executable-files/dfdaemon.md @@ -15,20 +15,21 @@ Golang|1.12.x ### 获取 dfget 可执行文件 -1. 下载 Dragonfly 项目的压缩包。您可以从 [github releases page](https://github. - com/dragonflyoss/Dragonfly2/releases) 下载一个已发布的最近版本 +1. 下载 Dragonfly 项目的压缩包。您可以从 +[github releases page](https://github.com/dragonflyoss/Dragonfly2/releases) +下载一个已发布的最近版本 -```sh -version=2.0.0 -wget https://github.com/dragonflyoss/Dragonfly2/releases/download/v$version/Dragonfly2_$version_linux_amd64.tar.gz -``` + ```sh + version=2.0.0 + wget https://github.com/dragonflyoss/Dragonfly2/releases/download/v$version/Dragonfly2_$version_linux_amd64.tar.gz + ``` 2. 解压压缩包 -```bash -# Replace `xxx` with the installation directory. -tar -zxf Dragonfly2_2.0.0_linux_amd64.tar.gz -C xxx -``` + ```bash + # Replace `xxx` with the installation directory. + tar -zxf Dragonfly2_2.0.0_linux_amd64.tar.gz -C xxx + ``` 3. 把 `dfget` 移动到环境变量 `PATH` 下以确保您可以直接使用 `dfget` 命令 @@ -36,21 +37,21 @@ tar -zxf Dragonfly2_2.0.0_linux_amd64.tar.gz -C xxx 1. 获取 Dragonfly 的源码 -```sh -git clone https://github.com/dragonflyoss/Dragonfly2.git -``` + ```sh + git clone https://github.com/dragonflyoss/Dragonfly2.git + ``` 2. 打开项目文件夹 -```sh -cd Dragonfly2 -``` + ```sh + cd Dragonfly2 + ``` 3. 编译源码 -```sh -make build-dfget && make install-dfget -``` + ```sh + make build-dfget && make install-dfget + ``` ### 启动 dfdaemon diff --git a/docs/zh-CN/deployment/installation/executable-files/manager.md b/docs/zh-CN/deployment/installation/executable-files/manager.md index 906f56b03b9..f64fc15643b 100644 --- a/docs/zh-CN/deployment/installation/executable-files/manager.md +++ b/docs/zh-CN/deployment/installation/executable-files/manager.md @@ -16,20 +16,21 @@ Nginx|0.8+ ### 获取 manager 可执行文件 -1. 下载 Dragonfly 项目的压缩包。您可以从 [github releases page](https://github. - com/dragonflyoss/Dragonfly2/releases) 下载一个已发布的最近版本 +1. 下载 Dragonfly 项目的压缩包。您可以从 +[github releases page](https://github.com/dragonflyoss/Dragonfly2/releases) +下载一个已发布的最近版本 -```sh -version=2.0.0 -wget https://github.com/dragonflyoss/Dragonfly2/releases/download/v$version/Dragonfly2_$version_linux_amd64.tar.gz -``` + ```sh + version=2.0.0 + wget https://github.com/dragonflyoss/Dragonfly2/releases/download/v$version/Dragonfly2_$version_linux_amd64.tar.gz + ``` 2. 解压压缩包 -```bash -# Replace `xxx` with the installation directory. -tar -zxf Dragonfly2_2.0.0_linux_amd64.tar.gz -C xxx -``` + ```bash + # Replace `xxx` with the installation directory. + tar -zxf Dragonfly2_2.0.0_linux_amd64.tar.gz -C xxx + ``` 3. 把 `manager` 移动到环境变量 `PATH` 下以确保您可以直接使用 `manager` 命令 @@ -37,21 +38,21 @@ tar -zxf Dragonfly2_2.0.0_linux_amd64.tar.gz -C xxx 1. 获取 Dragonfly 的源码 -```sh -git clone https://github.com/dragonflyoss/Dragonfly2.git -``` + ```sh + git clone https://github.com/dragonflyoss/Dragonfly2.git + ``` 2. 打开项目文件夹 -```sh -cd Dragonfly2 -``` + ```sh + cd Dragonfly2 + ``` 3. 编译源码 -```sh -make build-manager && make install-manager -``` + ```sh + make build-manager && make install-manager + ``` ### 启动 manager diff --git a/docs/zh-CN/deployment/installation/executable-files/scheduler.md b/docs/zh-CN/deployment/installation/executable-files/scheduler.md index 9803d8181e1..070be89ed48 100644 --- a/docs/zh-CN/deployment/installation/executable-files/scheduler.md +++ b/docs/zh-CN/deployment/installation/executable-files/scheduler.md @@ -16,20 +16,21 @@ Nginx|0.8+ ### 获取 scheduler 可执行文件 -1. 下载 Dragonfly 项目的压缩包。您可以从 [github releases page](https://github. - com/dragonflyoss/Dragonfly2/releases) 下载一个已发布的最近版本 +1. 下载 Dragonfly 项目的压缩包。您可以从 +[github releases page](https://github.com/dragonflyoss/Dragonfly2/releases) +下载一个已发布的最近版本 -```sh -version=2.0.0 -wget https://github.com/dragonflyoss/Dragonfly2/releases/download/v$version/Dragonfly2_$version_linux_amd64.tar.gz -``` + ```sh + version=2.0.0 + wget https://github.com/dragonflyoss/Dragonfly2/releases/download/v$version/Dragonfly2_$version_linux_amd64.tar.gz + ``` 2. 解压压缩包 -```bash -# Replace `xxx` with the installation directory. -tar -zxf Dragonfly2_2.0.0_linux_amd64.tar.gz -C xxx -``` + ```bash + # Replace `xxx` with the installation directory. + tar -zxf Dragonfly2_2.0.0_linux_amd64.tar.gz -C xxx + ``` 3. 把 `scheduler` 移动到环境变量 `PATH` 下以确保您可以直接使用 `scheduler` 命令 @@ -37,21 +38,21 @@ tar -zxf Dragonfly2_2.0.0_linux_amd64.tar.gz -C xxx 1. 获取 Dragonfly 的源码 -```sh -git clone https://github.com/dragonflyoss/Dragonfly2.git -``` + ```sh + git clone https://github.com/dragonflyoss/Dragonfly2.git + ``` 2. 打开项目文件夹 -```sh -cd Dragonfly2 -``` + ```sh + cd Dragonfly2 + ``` 3. 编译源码 -```sh -make build-scheduler && make install-scheduler -``` + ```sh + make build-scheduler && make install-scheduler + ``` ### 启动 scheduler diff --git a/docs/zh-CN/deployment/installation/kubernetes/README.md b/docs/zh-CN/deployment/installation/kubernetes/README.md index 564d919a81f..5e1d2d68065 100644 --- a/docs/zh-CN/deployment/installation/kubernetes/README.md +++ b/docs/zh-CN/deployment/installation/kubernetes/README.md @@ -1,6 +1,8 @@ # Dragonfly 部署 kubernetes 集群 -下面会解释如何在 Kubernetes 集群内部署 Dragonfly。Dragonfly 部署具体模块包括 4 部分: scheduler 和 cdn 会作为 `StatefulSets` 部署, dfdaemon 会作为 `DaemonSets` 部署, manager 会作为 `Deployments` 部署。 +下面会解释如何在 Kubernetes 集群内部署 Dragonfly。 +Dragonfly 部署具体模块包括 4 部分: scheduler 和 cdn 会作为 `StatefulSets` 部署, +dfdaemon 会作为 `DaemonSets` 部署, manager 会作为 `Deployments` 部署。 部署方式: diff --git a/docs/zh-CN/deployment/installation/kubernetes/helm.md b/docs/zh-CN/deployment/installation/kubernetes/helm.md index 7cb5bf0648c..616a6b345a4 100644 --- a/docs/zh-CN/deployment/installation/kubernetes/helm.md +++ b/docs/zh-CN/deployment/installation/kubernetes/helm.md @@ -2,30 +2,33 @@ ## Helm Chart 运行时配置 -当使用 Helm Chart 运行时配置时，可以忽略 [运行时配置](#运行时配置) 章节。因为 Helm Chart 安装时会自动帮助改变 Docker、Containerd 等配置, 无需再手动配置。 +当使用 Helm Chart 运行时配置时，可以忽略 [运行时配置](#运行时配置) 章节。 +因为 Helm Chart 安装时会自动帮助改变 Docker、Containerd 等配置, 无需再手动配置。 ### 1. Docker > **不推荐在 docker 环境中使用蜻蜓**：1. 拉镜像没有 fallback 机制，2. 在未来的 Kubernetes 中已经废弃。 -> > 因为当前 Kubernetes 里的 `daemonset` 并不支持 `Surging Rolling Update` 策略, > 一旦旧的 dfdaemon pod 被删除后，新的 dfdaemon 就再也拉取不了了。 - > 如果无法更换容器运行时的话，那在升级蜻蜓的时候，请从下面两种方案选择比较适合的： -> 选项1：先手动拉取新的 dfdaemon 镜像，或者使用 [ImagePullJob](https://openkruise.io/docs/user-manuals/imagepulljob) 去自动拉取， +> 选项1：先手动拉取新的 dfdaemon 镜像，或者使用 +[ImagePullJob](https://openkruise.io/docs/user-manuals/imagepulljob) +去自动拉取。 > 选项2：保持蜻蜓的镜像中心和通用的镜像中心不一样，同时将蜻蜓镜像中心相关的 host 加入 `containerRuntime.docker.skipHosts`。 Dragonfly Helm 支持自动更改 docker 配置。 -**情况 1:【推荐的】支持指定仓库** +#### 情况 1:【推荐的】支持指定仓库 定制 values.yaml 文件: + ```yaml containerRuntime: docker: enable: true # -- Inject domains into /etc/hosts to force redirect traffic to dfdaemon. - # Caution: This feature need dfdaemon to implement SNI Proxy, confirm image tag is greater than v2.0.0. + # Caution: This feature need dfdaemon to implement SNI Proxy, + # confirm image tag is greater than v2.0.0. # When use certs and inject hosts in docker, no necessary to restart docker daemon. injectHosts: true registryDomains: @@ -37,23 +40,30 @@ containerRuntime: 使用上述配置部署 Dragonfly 时，无需重新启动 docker。 优点： + * 支持 dfdaemon 自身平滑升级 > 这种模式下，当删除 dfdaemon pod 的时候，`preStop` 钩子将会清理已经注入到 `/etc/hosts` 下的所有主机信息，所有流量将会走原来的镜像中心。 限制: + * 只支持指定域名。 -**情况 2: 支持任意仓库** +#### 情况 2: 支持任意仓库 定制 values.yaml 文件: + ```yaml containerRuntime: docker: enable: true # -- Restart docker daemon to redirect traffic to dfdaemon - # When containerRuntime.docker.restart=true, containerRuntime.docker.injectHosts and containerRuntime.registry.domains is ignored. - # If did not want restart docker daemon, keep containerRuntime.docker.restart=false and containerRuntime.docker.injectHosts=true. + # When containerRuntime.docker.restart=true, + # containerRuntime.docker.injectHosts and + # containerRuntime.registry.domains is ignored. + # If did not want restart docker daemon, + # keep containerRuntime.docker.restart=false and + # containerRuntime.docker.injectHosts=true. restart: true ``` @@ -61,6 +71,7 @@ containerRuntime: 使用上述配置部署 Dragonfly 时，dfdaemon 将重新启动 docker。 限制: + * 必须开启 docker 的 `live-restore` 功能 * 需要重启 docker daemon @@ -68,7 +79,7 @@ containerRuntime: Containerd 的配置有两个版本，字段复杂。有很多情况需要考虑： -**情况 1: V2 版本使用配置文件** +#### 情况 1: V2 版本使用配置文件 配置文件路径是 `/etc/containerd/config.toml`: @@ -80,13 +91,14 @@ Containerd 的配置有两个版本，字段复杂。有很多情况需要考虑 这种情况很简单，并可以启用多个镜像仓库支持。 定制 values.yaml 文件: + ```yaml containerRuntime: containerd: enable: true ``` -**情况 2: V2 版本不使用配置文件** +#### 情况 2: V2 版本不使用配置文件 * 选项 1 - 允许 Charts 注入 `config_path` 并重新启动 containerd。 @@ -113,7 +125,7 @@ containerRuntime: enable: true ``` -**情况 3: V1 版本** +#### 情况 3: V1 版本 对于 V1 版本 config.toml，仅支持 `dfdaemon.config.proxy.registryMirror.url` 镜像仓库。 @@ -146,9 +158,10 @@ containerRuntime: - "https://harbor.example.com:8443" ``` -## 准备 Kubernetes 集群 +## 准备 Kubernetes 集群 -如果没有可用的 Kubernetes 集群进行测试，推荐使用 [minikube](https://minikube.sigs.k8s.io/docs/start/)。只需运行`minikube start`。 +如果没有可用的 Kubernetes 集群进行测试，推荐使用 +[minikube](https://minikube.sigs.k8s.io/docs/start/)。只需运行`minikube start`。 ## 安装 Dragonfly @@ -190,7 +203,8 @@ externalRedis: ```shell helm repo add dragonfly https://dragonflyoss.github.io/helm-charts/ -helm install --create-namespace --namespace dragonfly-system dragonfly dragonfly/dragonfly -f values.yaml +helm install --create-namespace --namespace dragonfly-system \ + dragonfly dragonfly/dragonfly -f values.yaml ``` ### 安装 Drgonfly 使用已经部署的 manager @@ -244,7 +258,9 @@ kubectl -n dragonfly-system wait --for=condition=ready --all --timeout=10m pod 控制台页面会在 `dragonfly-manager.dragonfly-system.svc.cluster.local:8080` 展示。 -需要绑定 Ingress 可以参考 [Helm Charts 配置选项](https://artifacthub.io/packages/helm/dragonfly/dragonfly#values), 或者手动自行创建 Ingress。 +需要绑定 Ingress 可以参考 +[Helm Charts 配置选项](https://artifacthub.io/packages/helm/dragonfly/dragonfly#values), +或者手动自行创建 Ingress。 控制台功能预览参考文档 [console preview](../../../design/manager.md)。 @@ -302,6 +318,14 @@ kubectl -n dragonfly-system exec -it ${pod_name} -- grep "peer task done" /var/l 日志输出例子: -``` -{"level":"info","ts":"2021-06-28 06:02:30.924","caller":"peer/peertask_stream_callback.go:77","msg":"stream peer task done, cost: 2838ms","peer":"172.17.0.9-1-ed7a32ae-3f18-4095-9f54-6ccfc248b16e","task":"3c658c488fd0868847fab30976c2a079d8fd63df148fb3b53fd1a418015723d7","component":"streamPeerTask"} +```shell +{ + "level": "info", + "ts": "2021-06-28 06:02:30.924", + "caller": "peer/peertask_stream_callback.go:77", + "msg": "stream peer task done, cost: 2838ms", + "peer": "172.17.0.9-1-ed7a32ae-3f18-4095-9f54-6ccfc248b16e", + "task": "3c658c488fd0868847fab30976c2a079d8fd63df148fb3b53fd1a418015723d7", + "component": "streamPeerTask" +} ``` diff --git a/docs/zh-CN/deployment/installation/kubernetes/kustomize.md b/docs/zh-CN/deployment/installation/kubernetes/kustomize.md index d68aefeb6b0..fc8785c7108 100644 --- a/docs/zh-CN/deployment/installation/kubernetes/kustomize.md +++ b/docs/zh-CN/deployment/installation/kubernetes/kustomize.md @@ -1,14 +1,17 @@ # Kustomize 支持 -## 准备 Kubernetes 集群 +## 准备 Kubernetes 集群 -如果没有可用的 Kubernetes 集群进行测试，推荐使用 [minikube](https://minikube.sigs.k8s.io/docs/start/)。只需运行`minikube start`。 +如果没有可用的 Kubernetes 集群进行测试， +推荐使用 [minikube](https://minikube.sigs.k8s.io/docs/start/)。只需运行`minikube start`。 ## 构建 Kustomize 模版并部署 ```shell git clone https://github.com/dragonflyoss/Dragonfly2.git -kustomize build Dragonfly2/deploy/kustomize/single-cluster-native/overlays/sample | kubectl apply -f - +kustomize build \ + Dragonfly2/deploy/kustomize/single-cluster-native/overlays/sample | \ + kubectl apply -f - ``` ## 等待部署成功 @@ -23,7 +26,9 @@ kubectl -n dragonfly-system wait --for=condition=ready --all --timeout=10m pod 控制台页面会在 `dragonfly-manager.dragonfly-system.svc.cluster.local:8080` 展示。 -需要绑定 Ingress 可以参考 [Helm Charts 配置选项](https://artifacthub.io/packages/helm/dragonfly/dragonfly#values), 或者手动自行创建 Ingress。 +需要绑定 Ingress 可以参考 +[Helm Charts 配置选项](https://artifacthub.io/packages/helm/dragonfly/dragonfly#values), +或者手动自行创建 Ingress。 控制台功能预览参考文档 [console preview](../../../design/manager.md)。 @@ -81,6 +86,14 @@ kubectl -n dragonfly-system exec -it ${pod_name} -- grep "peer task done" /var/l 日志输出例子: -``` -{"level":"info","ts":"2021-06-28 06:02:30.924","caller":"peer/peertask_stream_callback.go:77","msg":"stream peer task done, cost: 2838ms","peer":"172.17.0.9-1-ed7a32ae-3f18-4095-9f54-6ccfc248b16e","task":"3c658c488fd0868847fab30976c2a079d8fd63df148fb3b53fd1a418015723d7","component":"streamPeerTask"} +```shell +{ + "level": "info", + "ts": "2021-06-28 06:02:30.924", + "caller": "peer/peertask_stream_callback.go:77", + "msg": "stream peer task done, cost: 2838ms", + "peer": "172.17.0.9-1-ed7a32ae-3f18-4095-9f54-6ccfc248b16e", + "task": "3c658c488fd0868847fab30976c2a079d8fd63df148fb3b53fd1a418015723d7", + "component": "streamPeerTask" +} ``` diff --git a/docs/zh-CN/deployment/installation/kubernetes/upgrade-guide.md b/docs/zh-CN/deployment/installation/kubernetes/upgrade-guide.md index 76fc8815243..ebcb1df0947 100644 --- a/docs/zh-CN/deployment/installation/kubernetes/upgrade-guide.md +++ b/docs/zh-CN/deployment/installation/kubernetes/upgrade-guide.md @@ -1 +1,3 @@ -# WIP Upgrade guide \ No newline at end of file +# WIP Upgrade guide + +TODO diff --git a/docs/zh-CN/design/README.md b/docs/zh-CN/design/README.md index b81064a7ecf..094fdb0df6b 100644 --- a/docs/zh-CN/design/README.md +++ b/docs/zh-CN/design/README.md @@ -1,6 +1,7 @@ # Design -这里包含了所有关于 Dragonfly 的设计内容。它包括从最初设计时就考虑到的所有事情， Dragonfly 的所有组件设计的架构，组件之间的交互工作流程，Dragonfly 的所有 API 以及其他一些技术性的东西。 +这里包含了所有关于 Dragonfly 的设计内容。它包括从最初设计时就考虑到的所有事情， +Dragonfly 的所有组件设计的架构，组件之间的交互工作流程，Dragonfly 的所有 API 以及其他一些技术性的东西。 具体设计: diff --git a/docs/zh-CN/design/architecture.md b/docs/zh-CN/design/architecture.md index 0aae672b70c..ee33e97901b 100644 --- a/docs/zh-CN/design/architecture.md +++ b/docs/zh-CN/design/architecture.md @@ -1,6 +1,7 @@ # 产品思考与设计 ## 定位 + 提供企业级（高效、稳定、安全、低成本、产品化）的文件分发与管理服务，并成为在该领域的云原生最佳实践与标准解决方案。 ## 解决的问题 @@ -30,6 +31,7 @@ ## 架构设计 ### 整体架构 + ![alt][arch] ### 关联关系 @@ -48,7 +50,8 @@ ### 子系统功能 -**Manager** +#### Manager + - 动态配置管理 - 数据大盘 & 带宽计量 - 文件上传 & 分发管理与管控 @@ -58,7 +61,8 @@ - 多租户管理 - 命令通道服务（通道合一） -**Scheduler** +#### Scheduler + - 基于多维度的 P2P 调度策略 - 智能化调度算法（高级） - 调度结果可量化对比以及支持 A/B Testing @@ -66,7 +70,8 @@ - 元信息下发管道 - 客户端下载结果和回源结果统计以及与监控服务的对接 -**CDN** +#### CDN + - 多源适配能力，包括目录下载和 Range 下载功能 - DMA 读写 - 回源中转 @@ -77,7 +82,8 @@ - 安全传输（对称加密、完整性校验等） - 存储介质插件化 -**Dfget daemon** +#### Dfget daemon + - 多源适配能力，包括目录下载和 Range 下载功能 - 高效 IO 机制 - 多下载任务的 IO 调度（高 IO 吞吐量） @@ -91,12 +97,12 @@ - 客户端优雅升级与部署方案 - 命令执行 -**Framework** +#### Framework + - 高可用、高性能、易集成 - 一致性 Hash 算法选主 - 客户端连接管理 - [arch]: ../images/arch.png [association]: ../images/association.png [download-process]: ../images/download-process.png diff --git a/docs/zh-CN/design/cdn.md b/docs/zh-CN/design/cdn.md index 00e5a1ff354..50890197c11 100644 --- a/docs/zh-CN/design/cdn.md +++ b/docs/zh-CN/design/cdn.md @@ -1 +1,3 @@ -# WIP CDN \ No newline at end of file +# WIP CDN + +TODO diff --git a/docs/zh-CN/design/dfdaemon.md b/docs/zh-CN/design/dfdaemon.md index 62b20ae8bae..4c61ef7185f 100644 --- a/docs/zh-CN/design/dfdaemon.md +++ b/docs/zh-CN/design/dfdaemon.md @@ -1 +1,3 @@ -# WIP Dfdaemon \ No newline at end of file +# WIP Dfdaemon + +TODO diff --git a/docs/zh-CN/design/manager.md b/docs/zh-CN/design/manager.md index 30f51de6c7f..1f0ffe58ceb 100644 --- a/docs/zh-CN/design/manager.md +++ b/docs/zh-CN/design/manager.md @@ -4,17 +4,21 @@ Manager 控制台方便用户控制集群各模块服务，并且提供动态配 ## 关系模型 +
+ - CDN 集群与 Scheduler 集群为一对多关系 - CDN 集群与 CDN 实例是一对多关系 - Scheduler 集群与 Scheduler 实例是一对多关系 -Scheduler 实例信息通过，配置文件启动实例上报指定 Scheduler 集群 ID。参考[文档配置](../deployment/configuration/scheduler.yaml) `schedulerClusterID`。 +Scheduler 实例信息通过，配置文件启动实例上报指定 Scheduler 集群 ID。 +参考[文档配置](../deployment/configuration/scheduler.yaml) `schedulerClusterID`。 -CDN 实例信息通过，配置文件启动实例上报指定 CDN 集群 ID。参考[文档配置](../deployment/configuration/cdn.yaml) `cdnClusterID`。 +CDN 实例信息通过，配置文件启动实例上报指定 CDN 集群 ID。 +参考[文档配置](../deployment/configuration/cdn.yaml) `cdnClusterID`。 ## 用户账号 @@ -46,15 +50,19 @@ CDN 实例信息通过，配置文件启动实例上报指定 CDN 集群 ID。 ##### 配置 Scheduler 集群 +

+ ##### 配置 Scheduler 集群覆盖的客户端 +

+ - `load_limit`: 客户端可以提供的最大下载任务负载数。 @@ -70,9 +78,11 @@ CDN 实例信息通过，配置文件启动实例上报指定 CDN 集群 ID。 ##### 配置 CDN 集群 +

+ - `load_limit`: CDN 可以提供的最大下载任务负载数。 diff --git a/docs/zh-CN/design/scheduler.md b/docs/zh-CN/design/scheduler.md index 4633fac3a36..e027de2af9b 100644 --- a/docs/zh-CN/design/scheduler.md +++ b/docs/zh-CN/design/scheduler.md @@ -1 +1,3 @@ -# WIP Scheduler \ No newline at end of file +# WIP Scheduler + +TODO diff --git a/docs/zh-CN/developer-guide/developer-guide.md b/docs/zh-CN/developer-guide/developer-guide.md index 03aa49a33f2..d8e2dd485c5 100644 --- a/docs/zh-CN/developer-guide/developer-guide.md +++ b/docs/zh-CN/developer-guide/developer-guide.md @@ -2,11 +2,11 @@ 快速搭建本地开发环境 -### 第一步：安装 docker 和 docker compose +## 第一步：安装 docker 和 docker compose 详见文档 [docs.docker.com] -### 第二步：启动 dragonfly +## 第二步：启动 dragonfly 打开 dragonfly 文档、启动 docker-compose. @@ -19,10 +19,11 @@ Creating dfdaemon ... done Attaching to cdn, scheduler, dfdaemon ``` -### 第三步：日志分析 +## 第三步：日志分析 查看 dragonfly 日志 + ```bash $ tail -f log/**/*.log ==> log/dragonfly/cdn/core.log <== @@ -48,8 +49,9 @@ $ tail -f log/**/*.log {"level":"info","ts":"2021-02-26 05:43:37.342","caller":"worker/sender.go:49","msg":"start sender worker : 50"} {"level":"info","ts":"2021-02-26 05:43:37.343","caller":"worker/worker_group.go:64","msg":"start scheduler worker number:6"} ``` + -### 第四步：关闭 dragonfly +## 第四步：关闭 dragonfly ```bash $ docker-compose down diff --git a/docs/zh-CN/preheat/api.md b/docs/zh-CN/preheat/api.md index bf4a3a84f96..f6022dfa245 100644 --- a/docs/zh-CN/preheat/api.md +++ b/docs/zh-CN/preheat/api.md @@ -19,7 +19,18 @@ curl --location --request POST 'http://dragonfly-manager:8080/api/v1/jobs' \ 命令行日志返回预热任务 ID。 ```bash -{ "id": 1 "task_id": "group_4d1ea00e-740f-4dbf-a47e-dbdc08eb33e1", "type": "preheat", "status": "PENDING", "args": { "filter": "", "headers": null, "type": "image", "url": "https://registry-1.docker.io/v2/library/redis/manifests/latest" }} +{ + "id": 1, + "task_id": "group_4d1ea00e-740f-4dbf-a47e-dbdc08eb33e1", + "type": "preheat", + "status": "PENDING", + "args": { + "filter": "", + "headers": null, + "type": "image", + "url": "https://registry-1.docker.io/v2/library/redis/manifests/latest" + } +} ``` 使用预热任务 ID 轮训查询任务是否成功，具体 api 可以参考文档 [get job api document](../api-reference/api-reference.md#get-job)。 @@ -31,5 +42,16 @@ curl --request GET 'http://dragonfly-manager:8080/api/v1/jobs/1' 如果返回预热任务状态为 `SUCCESS`，表示预热成功。 ```bash -{ "id": 1 "task_id": "group_4d1ea00e-740f-4dbf-a47e-dbdc08eb33e1", "type": "preheat", "status": "SUCCESS", "args": { "filter": "", "headers": null, "type": "image", "url": "https://registry-1.docker.io/v2/library/redis/manifests/latest" }} -``` \ No newline at end of file +{ + "id": 1, + "task_id": "group_4d1ea00e-740f-4dbf-a47e-dbdc08eb33e1", + "type": "preheat", + "status": "SUCCESS", + "args": { + "filter": "", + "headers": null, + "type": "image", + "url": "https://registry-1.docker.io/v2/library/redis/manifests/latest" + } +} +``` diff --git a/docs/zh-CN/preheat/console.md b/docs/zh-CN/preheat/console.md index 4faa41a6e29..0860bd92b25 100644 --- a/docs/zh-CN/preheat/console.md +++ b/docs/zh-CN/preheat/console.md @@ -1 +1,3 @@ -# WIP console \ No newline at end of file +# WIP console + +TODO diff --git a/docs/zh-CN/quick-start.md b/docs/zh-CN/quick-start.md index cf0a2c4b56b..df7752286d7 100644 --- a/docs/zh-CN/quick-start.md +++ b/docs/zh-CN/quick-start.md @@ -2,30 +2,35 @@ 文档的目标是帮助您快速开始使用 Dragonfly。 -您可以根据 [Kubernetes-with-Dragonfly](./deployment/installation/kubernetes/README.md) 文档中的内容快速搭建 Dragonfly 的 Kubernetes 集群。我们推荐使用 `Containerd with CRI` 和 `CRI-O` 客户端。 +您可以根据 [Kubernetes-with-Dragonfly](./deployment/installation/kubernetes/README.md) +文档中的内容快速搭建 Dragonfly 的 Kubernetes 集群。 +我们推荐使用 `Containerd with CRI` 和 `CRI-O` 客户端。 下表列出了一些容器的运行时、版本和文档。 | Runtime | Version | Document | CRI Support | Pull Command | -| --- | --- | --- | --- | --- | +| --- | --- | --- | --- | --- | + | Containerd* | v1.1.0+ | [Link](runtime-integration/containerd/mirror.md) | Yes | crictl pull docker.io/library/alpine:latest | | Containerd without CRI | < v1.1.0 | [Link](runtime-integration/containerd/proxy.md) | No | ctr image pull docker.io/library/alpine | | CRI-O | All | [Link](runtime-integration/cri-o.md) | Yes | crictl pull docker.io/library/alpine:latest | + **:推荐使用`containerd`* + ## Helm Chart 运行时配置 Dragonfly Helm 支持自动更改 docker 配置。 -**支持指定仓库** +**支持指定仓库** 定制 values.yaml 文件: -定制 values.yaml 文件: ```yaml containerRuntime: docker: enable: true # -- Inject domains into /etc/hosts to force redirect traffic to dfdaemon. - # Caution: This feature need dfdaemon to implement SNI Proxy, confirm image tag is greater than v2.0.0. + # Caution: This feature need dfdaemon to implement SNI Proxy, + # confirm image tag is greater than v2.0.0. # When use certs and inject hosts in docker, no necessary to restart docker daemon. injectHosts: true registryDomains: @@ -37,16 +42,19 @@ containerRuntime: 使用上述配置部署 Dragonfly 时，无需重新启动 docker。 优点： + * 支持 dfdaemon 自身平滑升级 > 这种模式下，当删除 dfdaemon pod 的时候，`preStop` 钩子将会清理已经注入到 `/etc/hosts` 下的所有主机信息，所有流量将会走原来的镜像中心。 限制: + * 只支持指定域名。 -## 准备 Kubernetes 集群 +## 准备 Kubernetes 集群 -如果没有可用的 Kubernetes 集群进行测试，推荐使用 [minikube](https://minikube.sigs.k8s.io/docs/start/)。只需运行`minikube start`。 +如果没有可用的 Kubernetes 集群进行测试，推荐使用 [minikube](https://minikube.sigs.k8s.io/docs/start/)。 +只需运行`minikube start`。 ## 安装 Dragonfly @@ -69,7 +77,8 @@ kubectl -n dragonfly-system wait --for=condition=ready --all --timeout=10m pod 控制台页面会在 `dragonfly-manager.dragonfly-system.svc.cluster.local:8080` 展示。 -需要绑定 Ingress 可以参考 [Helm Charts 配置选项](https://artifacthub.io/packages/helm/dragonfly/dragonfly#values), 或者手动自行创建 Ingress。 +需要绑定 Ingress 可以参考 [Helm Charts 配置选项](https://artifacthub.io/packages/helm/dragonfly/dragonfly#values), +或者手动自行创建 Ingress。 控制台功能预览参考文档 [console preview](design/manager.md)。 @@ -97,8 +106,16 @@ kubectl -n dragonfly-system exec -it ${pod_name} -- grep "peer task done" /var/l 日志输出例子: -``` -{"level":"info","ts":"2021-06-28 06:02:30.924","caller":"peer/peertask_stream_callback.go:77","msg":"stream peer task done, cost: 2838ms","peer":"172.17.0.9-1-ed7a32ae-3f18-4095-9f54-6ccfc248b16e","task":"3c658c488fd0868847fab30976c2a079d8fd63df148fb3b53fd1a418015723d7","component":"streamPeerTask"} +```text +{ + "level": "info", + "ts": "2021-06-28 06:02:30.924", + "caller": "peer/peertask_stream_callback.go:77", + "msg": "stream peer task done, cost: 2838ms", + "peer": "172.17.0.9-1-ed7a32ae-3f18-4095-9f54-6ccfc248b16e", + "task": "3c658c488fd0868847fab30976c2a079d8fd63df148fb3b53fd1a418015723d7", + "component": "streamPeerTask" +} ``` ## Preheat diff --git a/docs/zh-CN/runtime-integration/containerd/mirror.md b/docs/zh-CN/runtime-integration/containerd/mirror.md index ae06d483984..a7d9b366389 100644 --- a/docs/zh-CN/runtime-integration/containerd/mirror.md +++ b/docs/zh-CN/runtime-integration/containerd/mirror.md @@ -45,9 +45,8 @@ version = 2 配置支持两个镜像仓库地址 `http://127.0.0.1:65001` 以及 `https://registry-1.docker.io`. -> 详细 Containerd 配置文档参照: https://github.com/containerd/containerd/blob/v1.5.2/docs/cri/registry.md#configure-registry-endpoint - -> 镜像仓库配置参照文档: https://github.com/containerd/containerd/blob/v1.5.2/docs/cri/config.md#registry-configuration +> 详细 Containerd 配置文档参照: +> 镜像仓库配置参照文档: ### 选项 2: 多镜像仓库 @@ -84,17 +83,17 @@ server = "https://example.com" ##### 选项 2: 自动生成 hosts.toml -自动生成 hosts.toml 脚本为 https://github.com/dragonflyoss/Dragonfly2/blob/main/hack/gen-containerd-hosts.sh +自动生成 hosts.toml 脚本为 ```shell bash gen-containerd-hosts.sh example.com ``` -> 镜像仓库配置详细文档参照: https://github.com/containerd/containerd/blob/main/docs/hosts.md#registry-configuration---examples +> 镜像仓库配置详细文档参照: ## Step 3: 重启 Containerd -``` +```shell systemctl restart containerd ``` @@ -102,7 +101,7 @@ systemctl restart containerd 使用以下命令拉取镜像: -``` +```shell crictl pull docker.io/library/busybox ``` @@ -110,12 +109,19 @@ crictl pull docker.io/library/busybox 可以查看日志，判断 busybox 镜像正常拉取。 -```bash +```shell grep 'register peer task result' /var/log/dragonfly/daemon/*.log ``` 如果正常日志输出如下: -``` -{"level":"info","ts":"2021-02-23 20:03:20.306","caller":"client/client.go:83","msg":"register peer task result:true[200] for taskId:adf62a86f001e17037eedeaaba3393f3519b80ce,peerIp:10.15.233.91,securityDomain:,idc:,scheduler:127.0.0.1:8002","peerId":"10.15.233.91-65000-43096-1614081800301788000","errMsg":null} +```shell +{ + "level": "info", + "ts": "2021-02-23 20:03:20.306", + "caller": "client/client.go:83", + "msg": "register peer task result:true[200]", + "peerId": "10.15.233.91-65000-43096-1614081800301788000", + "errMsg": null +} ``` diff --git a/docs/zh-CN/runtime-integration/containerd/proxy.md b/docs/zh-CN/runtime-integration/containerd/proxy.md index 26d8a595d68..efedc257d95 100644 --- a/docs/zh-CN/runtime-integration/containerd/proxy.md +++ b/docs/zh-CN/runtime-integration/containerd/proxy.md @@ -1,6 +1,7 @@ # 使用 dfget daemon 作为 containerd 的 http 代理 -目前 containerd 的 `ctr` 命令不支持带有 registry-mirrors 的私有注册表，为此，我们需要为 containerd 使用 HTTP 代理。 +目前 containerd 的 `ctr` 命令不支持带有 +registry-mirrors 的私有注册表，为此我们需要为 containerd 使用 HTTP 代理。 ## 快速开始 @@ -16,31 +17,31 @@ openssl genrsa -out ca.key 2048 ```text [ req ] -#default_bits = 2048 -#default_md = sha256 -#default_keyfile = privkey.pem -distinguished_name = req_distinguished_name -attributes = req_attributes +#default_bits = 2048 +#default_md = sha256 +#default_keyfile = privkey.pem +distinguished_name = req_distinguished_name +attributes = req_attributes extensions = v3_ca req_extensions = v3_ca [ req_distinguished_name ] -countryName = Country Name (2 letter code) -countryName_min = 2 -countryName_max = 2 -stateOrProvinceName = State or Province Name (full name) -localityName = Locality Name (eg, city) -0.organizationName = Organization Name (eg, company) -organizationalUnitName = Organizational Unit Name (eg, section) -commonName = Common Name (eg, fully qualified host name) -commonName_max = 64 -emailAddress = Email Address -emailAddress_max = 64 +countryName = Country Name (2 letter code) +countryName_min = 2 +countryName_max = 2 +stateOrProvinceName = State or Province Name (full name) +localityName = Locality Name (eg, city) +0.organizationName = Organization Name (eg, company) +organizationalUnitName = Organizational Unit Name (eg, section) +commonName = Common Name (eg, fully qualified host name) +commonName_max = 64 +emailAddress = Email Address +emailAddress_max = 64 [ req_attributes ] -challengePassword = A challenge password -challengePassword_min = 4 -challengePassword_max = 20 +challengePassword = A challenge password +challengePassword_min = 4 +challengePassword_max = 20 [ v3_ca ] basicConstraints = CA:TRUE @@ -50,7 +51,8 @@ basicConstraints = CA:TRUE ```bash openssl req -new -key ca.key -nodes -out ca.csr -config openssl.conf -openssl x509 -req -days 36500 -extfile openssl.conf -extensions v3_ca -in ca.csr -signkey ca.key -out ca.crt +openssl x509 -req -days 36500 -extfile openssl.conf \ + -extensions v3_ca -in ca.csr -signkey ca.key -out ca.crt ``` ### 第二步：配置 dfget daemon @@ -77,10 +79,10 @@ proxy: ### 第三步：配置 containerd -在 `/etc/systemd/system/containerd.service.d/http-proxy.conf` 设置 dfdaemon 为 docker daemon 的 -`HTTP_PROXY` 和 `HTTPS_PROXY` 代理： +在 `/etc/systemd/system/containerd.service.d/http-proxy.conf` +设置 dfdaemon 为 docker daemon 的 `HTTP_PROXY` 和 `HTTPS_PROXY` 代理： -``` +```toml [Service] Environment="HTTP_PROXY=http://127.0.0.1:65001" Environment="HTTPS_PROXY=http://127.0.0.1:65001" @@ -142,6 +144,7 @@ proxy: 您能使用以下命令获取您服务器的证书： -``` -openssl x509 -in <(openssl s_client -showcerts -servername xxx -connect xxx:443 -prexit 2>/dev/null) +```bash +openssl x509 -in <(openssl s_client -showcerts \ + -servername your.domain.com -connect your.domain.com:443 -prexit 2>/dev/null) ``` diff --git a/docs/zh-CN/runtime-integration/cri-o.md b/docs/zh-CN/runtime-integration/cri-o.md index cf10e514f95..ecdcd6cefbf 100644 --- a/docs/zh-CN/runtime-integration/cri-o.md +++ b/docs/zh-CN/runtime-integration/cri-o.md @@ -33,17 +33,18 @@ location = "docker.io" ## Step 3: 重启 CRI-O -``` +```shell systemctl restart crio ``` -如果遇到如下错误 `mixing sysregistry v1/v2 is not supported` 或 `registry must be in v2 format but is in v1`, 请将您的镜像仓库配置为 v2。 +如果遇到如下错误 `mixing sysregistry v1/v2 is not supported` 或 +`registry must be in v2 format but is in v1`, 请将您的镜像仓库配置为 v2。 ## Step 4: 拉取镜像 使用以下命令拉取镜像: -``` +```shell crictl pull docker.io/library/busybox ``` @@ -51,12 +52,19 @@ crictl pull docker.io/library/busybox 可以查看日志，判断 busybox 镜像正常拉取。 -```bash +```shell grep 'register peer task result' /var/log/dragonfly/daemon/*.log ``` 如果正常日志输出如下: -``` -{"level":"info","ts":"2021-02-23 20:03:20.306","caller":"client/client.go:83","msg":"register peer task result:true[200] for taskId:adf62a86f001e17037eedeaaba3393f3519b80ce,peerIp:10.15.233.91,securityDomain:,idc:,scheduler:127.0.0.1:8002","peerId":"10.15.233.91-65000-43096-1614081800301788000","errMsg":null} +```shell +{ + "level": "info", + "ts": "2021-02-23 20:03:20.306", + "caller": "client/client.go:83", + "msg": "register peer task result:true[200], + "peerId": "10.15.233.91-65000-43096-1614081800301788000", + "errMsg": null +} ``` diff --git a/docs/zh-CN/runtime-integration/docker.md b/docs/zh-CN/runtime-integration/docker.md index 7297669f138..bb4f9b5e502 100644 --- a/docs/zh-CN/runtime-integration/docker.md +++ b/docs/zh-CN/runtime-integration/docker.md @@ -16,29 +16,29 @@ openssl genrsa -out ca.key 2048 ```text [ req ] -#default_bits = 2048 -#default_md = sha256 -#default_keyfile = privkey.pem -distinguished_name = req_distinguished_name -attributes = req_attributes +#default_bits = 2048 +#default_md = sha256 +#default_keyfile = privkey.pem +distinguished_name = req_distinguished_name +attributes = req_attributes extensions = v3_ca req_extensions = v3_ca [ req_distinguished_name ] -countryName = Country Name (2 letter code) -countryName_min = 2 -countryName_max = 2 -stateOrProvinceName = State or Province Name (full name) -localityName = Locality Name (eg, city) -0.organizationName = Organization Name (eg, company) -organizationalUnitName = Organizational Unit Name (eg, section) -commonName = Common Name (eg, fully qualified host name) -commonName_max = 64 -emailAddress = Email Address -emailAddress_max = 64 +countryName = Country Name (2 letter code) +countryName_min = 2 +countryName_max = 2 +stateOrProvinceName = State or Province Name (full name) +localityName = Locality Name (eg, city) +0.organizationName = Organization Name (eg, company) +organizationalUnitName = Organizational Unit Name (eg, section) +commonName = Common Name (eg, fully qualified host name) +commonName_max = 64 +emailAddress = Email Address +emailAddress_max = 64 [ req_attributes ] -challengePassword = A challenge password -challengePassword_min = 4 -challengePassword_max = 20 +challengePassword = A challenge password +challengePassword_min = 4 +challengePassword_max = 20 [ v3_ca ] basicConstraints = CA:TRUE ``` @@ -47,7 +47,8 @@ basicConstraints = CA:TRUE ```bash openssl req -new -key ca.key -nodes -out ca.csr -config openssl.conf -openssl x509 -req -days 36500 -extfile openssl.conf -extensions v3_ca -in ca.csr -signkey ca.key -out ca.crt +openssl x509 -req -days 36500 -extfile openssl.conf \ + -extensions v3_ca -in ca.csr -signkey ca.key -out ca.crt ``` ### 第二步：配置 dfget daemon @@ -74,7 +75,7 @@ proxy: ### 第三步：配置 Docker daemon -为了忽略您的证书错误，您需要在 +为了忽略您的证书错误，您需要在 `/etc/docker/daemon.json` 中把 `insecure-registries` 设置为您的私有代理： ```json @@ -85,10 +86,10 @@ proxy: ### 第四步：继续配置 Docker daemon -在 `/etc/systemd/system/docker.service.d/http-proxy.conf` 设置 dfdaemon 为 docker daemon 的 - `HTTP_PROXY` 和 `HTTPS_PROXY` 代理： +在 `/etc/systemd/system/docker.service.d/http-proxy.conf` 设置 dfdaemon 为 +docker daemon 的 `HTTP_PROXY` 和 `HTTPS_PROXY` 代理： -``` +```toml [Service] Environment="HTTP_PROXY=http://127.0.0.1:65001" Environment="HTTPS_PROXY=http://127.0.0.1:65001" @@ -150,6 +151,7 @@ proxy: 您能使用以下命令获取您服务器的证书： -``` -openssl x509 -in <(openssl s_client -showcerts -servername xxx -connect xxx:443 -prexit 2>/dev/null) +```shell +openssl x509 -in <(openssl s_client -showcerts \ + -servername your.domain.com -connect your.domain.com:443 -prexit 2>/dev/null) ``` diff --git a/docs/zh-CN/test-guide/test-guide.md b/docs/zh-CN/test-guide/test-guide.md index ac794b74b24..908677cba20 100644 --- a/docs/zh-CN/test-guide/test-guide.md +++ b/docs/zh-CN/test-guide/test-guide.md @@ -9,13 +9,13 @@ Dragonfly 包括单元测试和 E2E 测试。 ### 运行单元测试 ```bash -$ make test +make test ``` ### 运行单元测试并产出测试报告 ```bash -$ make test-coverage +make test-coverage ``` ## E2E 测试 @@ -25,17 +25,17 @@ E2E 测试代码在 `test/e2e` 目录中。 ### 运行 E2E 测试 ```bash -$ make e2e-test +make e2e-test ``` ### 运行 E2E 测试并产出测试报告 ```bash -$ make e2e-test-coverage +make e2e-test-coverage ``` ### 清理 E2E 测试 ```bash -$ make clean-e2e-test +make clean-e2e-test ``` diff --git a/docs/zh-CN/troubleshooting/README.md b/docs/zh-CN/troubleshooting/README.md index 4eccce71a2e..18f37673b51 100644 --- a/docs/zh-CN/troubleshooting/README.md +++ b/docs/zh-CN/troubleshooting/README.md @@ -4,27 +4,29 @@ 1. 确认限速值是否合适 [dfget.yaml](https://github.com/dragonflyoss/Dragonfly2/blob/main/docs/zh-CN/config/dfget.yaml#L61) -```yaml -download: - # 总下载限速 - totalRateLimit: 200Mi - # 单个任务下载限速 - perPeerRateLimit: 100Mi # 为了兼容极限环境下，默认值为 20Mi，可以按需调整 -upload: - # 上传限速 - rateLimit: 100Mi -``` + ```yaml + download: + # 总下载限速 + totalRateLimit: 200Mi + # 单个任务下载限速 + perPeerRateLimit: 100Mi # 为了兼容极限环境下，默认值为 20Mi，可以按需调整 + upload: + # 上传限速 + rateLimit: 100Mi + ``` 2. 确认回源速度是否正常 ## 500 Internal Server Error 1. 检查日志 /var/log/dragonfly/daemon/core.log + 2. 检查源站可连接行(DNS 错误 or 证书) -示例: -```shell -curl https://example.harbor.local/ -``` + 示例: + + ```shell + curl https://example.harbor.local/ + ``` -如果`curl`有报错，请查看具体错误 + 如果`curl`有报错，请查看具体错误 diff --git a/test/README.md b/test/README.md index e25bea1feb6..02925426768 100644 --- a/test/README.md +++ b/test/README.md @@ -9,13 +9,13 @@ Unit tests is in the project directory. ### Running unit tests ```bash -$ make test +make test ``` ### Runing uint tests with coverage reports ```bash -$ make test-coverage +make test-coverage ``` ## E2E tests @@ -25,17 +25,17 @@ E2E tests is in `test/e2e` path. ### Running E2E tests ```bash -$ make e2e-test +make e2e-test ``` ### Runing E2E tests with coverage reports ```bash -$ make e2e-test-coverage +make e2e-test-coverage ``` ### Clean E2E tests environment ```bash -$ make clean-e2e-test +make clean-e2e-test ``` diff --git a/test/tools/stress/README.md b/test/tools/stress/README.md index d95273574d3..b7b9579f94a 100644 --- a/test/tools/stress/README.md +++ b/test/tools/stress/README.md @@ -5,46 +5,49 @@ ### Build and Run 1. Build tool: -```shell -go build -o bin/stress test/stress/main.go -``` + + ```shell + go build -o bin/stress test/stress/main.go + ``` 2. Run stress: -```shell -bin/stress -connections 100 -duration 1s -proxy http://127.0.0.1:65001 \ - --url http://localhost/misc/d7y-test/blobs/sha256/128K -``` -Example output: -``` -Latency - avg 17.286522ms - min 617.801µs - max 84.201941ms -Latency Distribution - 50% 11.39049ms - 75% 18.308966ms - 90% 49.052485ms - 95% 55.886513ms - 99% 65.013042ms -HTTP codes - 200 5849 -Throughput 731.1MB -Request 5849/s -``` + ```shell + bin/stress -connections 100 -duration 1s -proxy http://127.0.0.1:65001 \ + --url http://localhost/misc/d7y-test/blobs/sha256/128K + ``` + + Example output: + + ```text + Latency + avg 17.286522ms + min 617.801µs + max 84.201941ms + Latency Distribution + 50% 11.39049ms + 75% 18.308966ms + 90% 49.052485ms + 95% 55.886513ms + 99% 65.013042ms + HTTP codes + 200 5849 + Throughput 731.1MB + Request 5849/s + ``` ### CLI Reference -``` +```text Usage of ./stress: -connections int - concurrency count of connections (default 100) + concurrency count of connections (default 100) -duration duration - testing duration (default 1m40s) + testing duration (default 1m40s) -output string - all request statistics (default "/tmp/statistics.txt") + all request statistics (default "/tmp/statistics.txt") -proxy string - target proxy for downloading, example: http://127.0.0.1:65001 + target proxy for downloading, example: http://127.0.0.1:65001 -url string - target url for stress testing, example: http://localhost + target url for stress testing, example: http://localhost ```