Skip to content

Commit

Permalink
Rework tool/doc for updating reference docs (#10123)
Browse files Browse the repository at this point in the history 
* WIP - rework update imported docs

* Rework tool/doc for updating reference docs

This PR reimplement the reference docs generator in Python and fixes
some outdated docs and data:

- Do docs import using Python because the GoLang version of tool has
  some following drawbacks:
  * its not convenient for handling YAML config files
  * it has to be compiled to binaries to run on different platforms
  * for every tiny changes you need to compile a new version and check in
- The reference docs we use in website are actually not coming directly
  from `kubernetes/kubernetes`. Most of them come from the `reference-docs`
  project. The configuration files are thus changed to avoid confusion.
- We have changed the location of generated docs so the default configuration
  files and the docs are updated.
  • Loading branch information
@tengqm @k8s-ci-robot
tengqm authored and k8s-ci-robot committed Sep 21, 2018
1 parent 7b33ce4 commit 26f0a81
Show file tree
Hide file tree
Showing 8 changed files with 288 additions and 552 deletions.
182 changes: 61 additions & 121 deletions content/en/docs/contribute/generate-ref-docs/kubernetes-components.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -116,136 +116,83 @@ need to work with someone who can set the label and milestone for you.

## Overview of update-imported-docs

The `update-imported-docs` tool performs these steps:

1. Clone the `kubernetes/kubernetes` repository.
1. Run several scripts under `kubernetes/kubernetes/hack`. These scripts
generate Markdown files and place the files under `kubernetes/kubernetes/docs`.
1. Copy the generated Markdown files to a local clone of the `kubernetes/website`
repository under `kubernetes/website/docs/reference/generated`.
1. Clone the `kubernetes/federation` repository.
1. Run several scripts under `kubernetes/federation/hack`. These scripts
generate Markdown files and place the files under `kubernetes/federation/docs`.
1. Copy the generated Markdown files to a local clone of the `kubernetes/website`
repository under `kubernetes/website/docs/reference/generated`.

After the Markdown files are in your local clone of the `kubernetes/website`
The website repository contains a `update-imported-docs` tool under the
`kubernetes/website/update-imported-docs/` directory that performs the
following steps:

1. Clones the related repositories specified in a configuration file. For the
purpose of generating reference docs, the repositories that are cloned by
default are `kubernetes-incubator/reference-docs` and `kubernetes/federation`.
1. Runs commands under the cloned repositories to prepare the docs generator and
then generates the Markdown files.
1. Copies the generated Markdown files to a local clone of the `kubernetes/website`
repository under locations specified in the configuration file.

When the Markdown files are in your local clone of the `kubernetes/website`
repository, you can submit them in a
[pull request](https://kubernetes.io/docs/home/contribute/create-pull-request/)
to `kubernetes/website`.

## Setting the branch
## Customizing the config file

Open `<web-base>/update-imported-docs/config.yaml` for editing.

Set the value of `branch` to the Kubernetes release that you want to document.
For example, if you want to generate docs for the Kubernetes 1.9 release,
set `branch` to `release-1.9`.
Open `<web-base>/update-imported-docs/reference.yaml` for editing.
Do not change the content for the `generate-command` entry unless you undertand
what it is doing and need to change the specified release branch.

```shell
repos:
- name: kubernetes
remote: https://github.com/kubernetes/kubernetes.git
branch: release-1.9
- name: reference-docs
remote: https://github.com/kubernetes-incubator/reference-docs.git
# This and the generate-command below needs a change when reference-docs has
# branches properly defined
branch: master
generate-command: |
cd $GOPATH
git clone https://github.com/kubernetes/kubernetes.git src/k8s.io/kubernetes
cd src/k8s.io/kubernetes
git checkout release-1.11
make generated_files
cp -L -R vendor $GOPATH/src
rm -r vendor
cd $GOPATH
go get -v github.com/kubernetes-incubator/reference-docs/gen-compdocs
cd src/github.com/kubernetes-incubator/reference-docs/
make comp
```

## Setting sources and destinations

The `update-imported-docs` tool uses `src` and `dst` fields
in `config.yaml` to know which files to copy from the `kubernetes/kubernetes`
repository and where to place those files in the `kubernetes/website`
repository.
The `update-imported-docs` tool uses `src` and `dst` fields in a configuration
to decide the source and target location for doc files to be copied.
For example:

For example, suppose you want the tool to copy the `kube-apiserver.md` file
from the `docs/admin` directory of the `kubernetes/kubernetes` repository
to the `docs/reference/generated/` directory of the `kubernetes/website`
repository. Then you would include a `src` and `dst` in your `config.yaml`
file like this:

```shell
```yaml
repos:
- name: kubernetes
remote: https://github.com/kubernetes/kubernetes.git
branch: release-1.9
- name: reference-docs
remote: https://github.com/kubernetes-incubator/reference-docs.git
files:
- src: docs/admin/kube-apiserver.md
dst: docs/reference/generated/kube-apiserver.md
- src: gen-compdocs/build/kube-apiserver.md
dst: content/en/docs/reference/command-line-tools-reference/kube-apiserver.md
...
```

The configuration is similar for files in the `kubernetes/federation`
repository. Here's an example that configures the tool to copy `kubefed_init.md`
from the `docs/admin` directory of the `kubernetes/federation` repository
to the `docs/reference/generated` directory of the `kubernetes/website` repository:
Note that when there are many files to be copied from the same source directory
to the same destination directory, you can use wildcards in the value given to
`src` and you can just provide the directory name as the value for `dst`.
For example:

```shell
- name: federation
remote: https://github.com/kubernetes/federation.git
# # Change this to a release branch when federation has release branches.
branch: master
files:
- src: docs/admin/kubefed_init.md
dst: docs/reference/generated/kubefed_init.md
...
- src: gen-compdocs/build/kubeadm*.md
dst: content/en/docs/reference/setup-tools/kubeadm/generated/
```

Here's an example a `config.yaml` file that shows the sources and
destinations of all the Markdown files that were generated and copied
by the `update-imported-docs` tool at the beginning of the Kubernetes
1.9 release.

```shell
repos:
- name: kubernetes
remote: https://github.com/kubernetes/kubernetes.git
branch: release-1.9
files:
- src: docs/admin/cloud-controller-manager.md
dst: docs/reference/generated/cloud-controller-manager.md
- src: docs/admin/kube-apiserver.md
dst: docs/reference/generated/kube-apiserver.md
- src: docs/admin/kube-controller-manager.md
dst: docs/reference/generated/kube-controller-manager.md
- src: docs/admin/kubelet.md
dst: docs/reference/generated/kubelet.md
- src: docs/admin/kube-proxy.md
dst: docs/reference/generated/kube-proxy.md
- src: docs/admin/kube-scheduler.md
dst: docs/reference/generated/kube-scheduler.md
- src: docs/user-guide/kubectl/kubectl.md
dst: docs/reference/generated/kubectl/kubectl.md
- name: federation
remote: https://github.com/kubernetes/federation.git
# # Change this to a release branch when federation has release branches.
branch: master
files:
- src: docs/admin/federation-apiserver.md
dst: docs/reference/generated/federation-apiserver.md
- src: docs/admin/federation-controller-manager.md
dst: docs/reference/generated/federation-controller-manager.md
- src: docs/admin/kubefed_init.md
dst: docs/reference/generated/kubefed_init.md
- src: docs/admin/kubefed_join.md
dst: docs/reference/generated/kubefed_join.md
- src: docs/admin/kubefed.md
dst: docs/reference/generated/kubefed.md
- src: docs/admin/kubefed_options.md
dst: docs/reference/generated/kubefed_options.md
- src: docs/admin/kubefed_unjoin.md
dst: docs/reference/generated/kubefed_unjoin.md
- src: docs/admin/kubefed_version.md
dst: docs/reference/generated/kubefed_version.md
```

## Running the update-imported-docs tool

Now that your `config.yaml` file contains your sources and destinations,
you can run the `update-imported-docs` tool:
After having reviewed and/or customized the `reference.yaml` file, you can run
the `update-imported-docs` tool:

```shell
cd <web-base>
go get ./update-imported-docs
go run update-imported-docs/update-imported-docs.go
cd <web-base>/update-imported-docs
./update-imported-docs reference.yml
```

## Adding and committing changes in kubernetes/website
Expand All @@ -263,21 +210,15 @@ might look like this:

```shell
...
modified: docs/reference/generated/cloud-controller-manager.md
modified: docs/reference/generated/federation-apiserver.md
modified: docs/reference/generated/federation-controller-manager.md
modified: docs/reference/generated/kube-apiserver.md
modified: docs/reference/generated/kube-controller-manager.md
modified: docs/reference/generated/kube-proxy.md
modified: docs/reference/generated/kube-scheduler.md
modified: docs/reference/generated/kubectl/kubectl.md
modified: docs/reference/generated/kubefed.md
modified: docs/reference/generated/kubefed_init.md
modified: docs/reference/generated/kubefed_join.md
modified: docs/reference/generated/kubefed_options.md
modified: docs/reference/generated/kubefed_unjoin.md
modified: docs/reference/generated/kubefed_version.md
modified: docs/reference/generated/kubelet.md

modified: content/en/docs/reference/command-line-tools-reference/cloud-controller-manager.md
modified: content/en/docs/reference/command-line-tools-reference/federation-apiserver.md
modified: content/en/docs/reference/command-line-tools-reference/federation-controller-manager.md
modified: content/en/docs/reference/command-line-tools-reference/kube-apiserver.md
modified: content/en/docs/reference/command-line-tools-reference/kube-controller-manager.md
modified: content/en/docs/reference/command-line-tools-reference/kube-proxy.md
modified: content/en/docs/reference/command-line-tools-reference/kube-scheduler.md
...
```

Run `git add` and `git commit` to commit the files.
Expand All @@ -303,4 +244,3 @@ topics will be visible in the
{{% /capture %}}



2 changes: 1 addition & 1 deletion content/en/docs/contribute/start.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ The Kubernetes documentation is written in Markdown and processed and deployed
using Hugo. The source is in Github at
[https://github.com/kubernetes/website](https://github.com/kubernetes/website).
Most of the documentation source is stored in `/content/en/docs/`. Some of the
reference documentation is automatically generated from scripts, mostly in the
reference documentation is automatically generated from scripts in the
`update-imported-docs/` directory.

You can file issues, edit content, and review changes from others, all from the
Expand Down
59 changes: 26 additions & 33 deletions update-imported-docs/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,12 +1,15 @@
# Update imported docs

This script updates the target files generated from other repos listed in the <config.yml> file, which is specified as the command line argument.
This script updates the docs files that are generated from other repos.
It accepts a YAML file name as its input which can be customized on a per-repo
basis.

## Requirements

Imported docs must follow these guidelines:

1. Be listed somewhere in the `/_data/imported.yml` table of contents file.
1. Adhere to the [Documentation Style Guide](/docs/home/contribute/style-guide/).

1. Have `title` defined in the front matter. For example:

```
Expand All @@ -16,49 +19,36 @@ Imported docs must follow these guidelines:
Rest of the .md file...
```
1. Be listed somewhere in a file under the `data` subdirectory, for example,
the `data/imported.yml` file.

1. Adhere to the [Documentation Style Guide](/docs/home/contribute/style-guide/).

## Usage

From within this directory, run the following command:
1. Make sure the `PyYAML` package is installed:

```
+./update-imported-docs-[linux|macos] <config.yaml>
sudo apt-get install python-pip
pip install PyYAML
```

The output should look similar to the following:

```
Website root directory: /Users/someuser/git/kubernetes-website
* * *
Cloning repo "community"...
## Usage

* * *
From within this directory, run the following command:

Docs imported! Run 'git add .' 'git commit -m <comment>' and 'git push' to upload them.
```
+./update-imported-docs <CONFIG-FILE>
```

## Config file format
where `<CONFIG-FILE>` can be any YAML configuration file in this directory.

Each config file may contain multiple repos, which will be imported together. You should modify the corresponding `update-imported-docs/<config.yml>` file to reflect the desired `src` and `dst` paths.
## Configuration file format

You may also create new config files for different groups of documents to import. The following is an example of the YAML file format:
Each config file may contain multiple repos that will be imported together.
When necessary, you can customize the configuration file by manually editing
it. You may create new config files for importing other groups of documents.
The following is an example of the YAML configuration file:

```
repos:
- name: kubernetes #tmp directory name
remote: https://github.com/kubernetes/kubernetes.git
branch: release-1.9
generate-command: hack/generate-docs.sh #optional command to run
files:
- src: docs/admin/cloud-controller-manager.md
dst: docs/reference/generated/cloud-controller-manager.md
- src: docs/admin/kube-apiserver.md
dst: docs/reference/generated/kube-apiserver.md
- name: community #tmp directory name
- name: community
remote: https://github.com/kubernetes/community.git
branch: master
files:
Expand All @@ -68,8 +58,11 @@ repos:
dst: docs/imported/community/guide.md
```

Note: `generate-command` is an optional entry, which can be used to run a given command to auto-generate the docs from within that repo.
Note: `generate-command` is an optional entry, which can be used to run a
given command or a short script to generate the docs from within a repo.

## Fixing Links

To fix relative links within your imported files, set the repo config's `gen-absolute-links` value to `true`. You can see an example of this in [`community.yml`](community.yml).
To fix relative links within your imported files, set the repo config's
`gen-absolute-links` property to `true`. You can find an example of this in
[`community.yml`](community.yml).
Loading

0 comments on commit 26f0a81

Please sign in to comment.