docs: various changes and improvements (#296)

docs/README.md

@@ -64,7 +64,7 @@ to install some of the tools manually:
 
 - [git] is required
 - [helm] is only needed for rendering Helm charts
-- [ytt] and [vendir] are now built into myks, no need to install separately. 
+- [ytt] and [vendir] are now built into myks, no need to install separately.
 
 At the moment, we do not track compatibility between versions of these tools and
 myks. Fairly recent versions should work fine.
@@ -136,7 +136,10 @@ The `all` command runs the both stages sequentially for convenience.
 
 These commands (`sync`, `render`, `all`) accept two optional arguments:
 environments and applications to process. When no arguments are provided, myks
-will use the [Smart Mode](SMARTMODE.md) to detect what to process.
+will use the [Smart Mode](/docs/smart-mode.md) to detect what to process.
+
+> [!TIP]  
+> Check the [optimizations](/docs/optimizations.md) page to get most of myks.
 
 ### Examples
 

docs/optimizations.md

@@ -0,0 +1,116 @@
+# Myks Optimizations
+
+This page lists myks' features that help to optimize performance and resource
+usage. Some of those features are enabled by default, while others require
+additional configuration. There are features that impreve rendering speed, but
+the most useful are those that reduce syncing activities.
+
+## Parallelism
+
+Myks can process multiple applications in parallel. It is applied to both
+syncing and rendering operations. The number of parallel operations can be
+configured with the `--async` flag, with the `MYKS_ASYNC` environment variable,
+or with the `async` configuration key. It is set to `0` by default, which means
+that there is no limit.
+
+## Selective Processing
+
+### Smart Mode
+
+Myks can detect what applications and environments to process based on the
+changes in the source files. This allows to reduce the number of applications
+and environments to process, which in turn reduces the time needed for syncing
+and rendering. See the [Smart Mode](/docs/smart-mode.md) page for more details.
+
+### Manual Selection
+
+Sometimes, the Smart Mode may not be able to detect the changes correctly. In
+this case, you can manually specify the applications and environments to
+process. For example:
+
+```shell
+myks all foo-env,bar-env baz-app
+```
+
+This command will sync and render the `baz-app` application in the `foo-env` and
+`bar-env` environments only.
+
+See `myks help` for more details.
+
+## Skip Sync
+
+### Vendir's `lazy` mode
+
+Vendir has a `lazy` mode that skips syncing of the sources that were synced
+earlier and haven't changed their vendir configuration. See the
+[`vendir.yml` Reference](https://carvel.dev/vendir/docs/v0.40.x/vendir-spec/)
+for more details.
+
+It is advised to enable the `lazy` mode in the `vendir.yml` configuration file
+if the corresponding sources are defined using immutable[^1] references (e.g.,
+commit hashes or image digests). For example:
+
+```yaml
+apiVersion: vendir.k14s.io/v1alpha1
+kind: Config
+directories:
+  - path: charts/cert-manager
+    contents:
+      - path: .
+        lazy: true # <--
+        helmChart:
+          name: cert-manager
+          version: v1.14.5
+          repository:
+            url: https://charts.jetstack.io
+```
+
+Here, we trust maintainers of the cert-manager chart to not move the `v1.14.5`
+tag and ask vendir to skip syncing this source if it was downloaded earlier.
+
+[^1]:
+    Some references aren't strictly immutable, but can be treated as such in
+    practice. For example, a git tag can be moved, but a user can decide to
+    treat it as immutable.
+
+### Centralized Cache
+
+Using the vendir's `lazy` mode is a good way to avoid unnecessary syncing of
+unchanged sources. However, it doesn't help when the same source is used in
+multiple applications. In this case, the source is synced multiple times for
+multiple applications.
+
+To overcome this limitation, myks implements a centralized cache and ensures
+that only one sync is performed for each source. The same source is made
+available to multiple applications by symlink creation.
+
+Cache entries are stored in the `.myks/vendir-cache` directory and identified by
+a hash of the corresponding `contents` section in the `vendir.yml` file. The
+cache is invalidated when the vendir configuration changes.
+
+Cache entries that are not used by any application can be removed with the
+`cleanup` command:
+
+```shell
+myks cleanup --cache
+```
+
+### Speed Up CI/CD Pipelines
+
+Persisting the cache between CI jobs can significantly reduce the time needed
+for syncing. Generally, there are two ways to achieve this:
+
+- utilize a shared cache between CI jobs that is provided by your CI platform.
+- check-in the cache directory to the repository by adjusting the gitignore
+  rules as follows:
+
+  ```diff
+  -.myks
+  +.myks/*
+  +!.myks/vendir-cache
+  ```
+
+  Note, that this approach may introduce a lot of changes and increase the
+  repository size. If you commit changes after syncing (in CI, as well as
+  locally), make sure to run the `cleanup` command before. This will reduce
+  diffs and make them cleaner.

examples/README.md

@@ -0,0 +1,9 @@
+# Examples
+
+This directory contains example myks projects with some comments and the
+rendered output.
+
+- [simple](simple/README.md) example with two environments and one helm chart.
+- [default](default/README.md) example created with `myks init`
+
+The rendered output is also used in integration tests.

examples/default/readme.md → examples/default/README.md

@@ -1,24 +1,28 @@
-# default example
+# Default myks project
 
-This repository was created using `myks init`.
+This directory contents is populated using the `myks init` command.
 
 Included are two prototypes:
 
-- Argocd: using the prerendered`manifests/ha/install.yaml` from argoCD github repo
-- httpbingo: helm chart of small demo application
-  and one environment (envs/mykso/dev).
+- Argocd: using the prerendered`manifests/ha/install.yaml` from argoCD github
+  repo
+- httpbingo: helm chart of small demo application and one environment
+  (envs/mykso/dev).
 
 ## features
 
 This example repository contains several aspects of overwrites myks can handle.
 
 ### application schema and configuration
 
-ArgoCD is installed to the cluster from a plain manifest. The manifest is changed during the rendering process.
+ArgoCD is installed to the cluster from a plain manifest. The manifest is
+changed during the rendering process.
 
 - `prototypes/argocd/app-data.ytt.yaml` defines defaults and a schema
-- The value of `gcpServiceAccountEmail` is overwritten in `envs/mykso/dev/_apps/argocd/app-data.ytt.yaml` only for the cluster `dev`
-- the value is applied in `prototypes/argocd/ytt/argocd-vault-plugin.ytt.yaml` to add an annotation to the argoCD serviceAccount using ytt overlays
+- The value of `gcpServiceAccountEmail` is overwritten in
+  `envs/mykso/dev/_apps/argocd/app-data.ytt.yaml` only for the cluster `dev`
+- the value is applied in `prototypes/argocd/ytt/argocd-vault-plugin.ytt.yaml`
+  to add an annotation to the argoCD serviceAccount using ytt overlays
 
 ### general overlays
 
@@ -31,14 +35,18 @@ Overlays can happen on every level:
 
 Example overlays:
 
-- `envs/_env/argocd/annotations.overlay.ytt.yaml`: Overlay to apply to all argoCD resources. Adds annotations.
-- `envs/_env/argocd/secret.overlay.ytt.yaml`: Extend the argoCD cluster definition (server URL and connect config)
+- `envs/_env/argocd/annotations.overlay.ytt.yaml`: Overlay to apply to all
+  argoCD resources. Adds annotations.
+- `envs/_env/argocd/secret.overlay.ytt.yaml`: Extend the argoCD cluster
+  definition (server URL and connect config)
 - `envs/_env/ytt/common.ytt.yaml`: Overlay applied on all kubernetes resources
 
 ### multiple configurations levels
 
-The httpbingo prototype defines it's own defaults (`prototypes/httpbingo/helm/httpbingo.yaml`) on top of the helm chart defaults.
-The replicaCount is overwritten for the dev cluster (`envs/mykso/dev/_apps/httpbingo/helm/httpbingo.yaml`) with ytt support.
+The httpbingo prototype defines it's own defaults
+(`prototypes/httpbingo/helm/httpbingo.yaml`) on top of the helm chart defaults.
+The replicaCount is overwritten for the dev cluster
+(`envs/mykso/dev/_apps/httpbingo/helm/httpbingo.yaml`) with ytt support.
 
 ## tree
 

examples/simple/readme.md → examples/simple/README.md

@@ -1,12 +1,18 @@
-# helm example
+# Simple example
 
-Simple example with two environments dev & prod with one helm chart.
+This example project consists of two environments (`dev` and `prod`) with a
+single helm chart.
 
-- httpbingo helm chart version 0.1.0 is defined in the prototype
-- dev environments overwrites the helm chart version (vendir config) (`envs/dev/_apps/httpbingo/vendir/vendir-data.ytt.yaml`)
-- prod environments overwrites replica count (helm chart value) (`envs/prod/_apps/httpbingo/helm/httpbingo.yaml`)
+- the single helm chart `httpbingo` of version `0.1.0` is defined in the
+  `prototypes` directory
+- the `dev` environment overwrites the version of the helm chart via the vendir
+  data values file in
+  [`envs/dev/_apps/httpbingo/vendir/vendir-data.ytt.yaml`](envs/dev/_apps/httpbingo/vendir/vendir-data.ytt.yaml)
+- the `prod` environment overwrites the replica count via the helm chart values
+  file in
+  [`envs/prod/_apps/httpbingo/helm/httpbingo.yaml`](envs/prod/_apps/httpbingo/helm/httpbingo.yaml)
 
-## tree
+## File tree
 
 ```python
 .