merge(kontemplate): Integrate kontemplate at //depot/ops/kontemplate r/251

4 files changed, 506 insertions, 0 deletions

diff --git a/ops/kontemplate/docs/cluster-config.md b/ops/kontemplate/docs/cluster-config.md
new file mode 100644
index 000000000000..fee1f576347c
--- /dev/null
+++ b/ops/kontemplate/docs/cluster-config.md
@@ -0,0 +1,106 @@
+Cluster configuration
+==========================
+
+Every cluster (or "environment") that requires individual configuration is specified in
+a very simple YAML file in Kontemplate.
+
+An example file for a hypothetical test environment could look like this:
+
+```yaml
+---
+context: k8s.test.mydomain.com
+global:
+  clusterName: test-cluster
+  defaultReplicas: 2
+import:
+  - test-secrets.yaml
+include:
+  - name: gateway
+    path: tools/nginx
+    values:
+      tlsDomains:
+        - test.oslo.pub
+        - test.tazj.in
+  - path: backend
+    values:
+      env: test
+    include:
+      - name: blog
+        values:
+          url: test.tazj.in
+      - name: pub-service
+```
+
+<!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-refresh-toc -->
+**Table of Contents**
+
+- [Cluster configuration](#cluster-configuration)
+    - [Fields](#fields)
+        - [`context`](#context)
+        - [`global`](#global)
+        - [`import`](#import)
+        - [`include`](#include)
+    - [External variables](#external-variables)
+
+<!-- markdown-toc end -->
+
+## Fields
+
+This is documentation for the individual fields in a cluster context file.
+
+### `context`
+
+The `context` field contains the name of the kubectl-context. You can list context names with
+'kubectl config get-contexts'.
+
+This must be set here so that Kontemplate can use the correct context when calling kubectl.
+
+This field is **required** for `kubectl`-wrapping commands. It can be left out if only the `template`-command is used.
+
+### `global`
+
+The `global` field contains a key/value map of variables that should be available to all resource
+sets in the cluster.
+
+This field is **optional**.
+
+### `import`
+
+The `import` field contains the file names of additional YAML or JSON files from which global
+variables should be loaded. Using this field makes it possible to keep certain configuration that
+is the same for some, but not all, clusters in a common place.
+
+This field is **optional**.
+
+### `include`
+
+The `include` field contains the actual resource sets to be included in the cluster.
+
+Information about the structure of resource sets can be found in the [resource set documentation][].
+
+This field is **required**.
+
+## External variables
+
+As mentioned above, extra variables can be loaded from additional YAML or JSON files. Assuming you
+have a file called `test-secrets.yaml` which contains variables that should be shared between a `test`
+and `dev` cluster, you could include it in your context as such:
+
+```yaml
+# test-secrets.yaml:
+mySecretVar: foo-bar-12345
+
+# test-cluster.yaml:
+context: k8s.test.mydomain.com
+include:
+  - test-secrets.yaml
+
+# dev-cluster.yaml:
+context: k8s.dev.mydomain.com
+include:
+  - test-secrets.yaml
+```
+
+The variable `mySecretVar` is then available as a global variable.
+
+[resource set documentation]: resource-sets.md
diff --git a/ops/kontemplate/docs/resource-sets.md b/ops/kontemplate/docs/resource-sets.md
new file mode 100644
index 000000000000..1444dd491249
--- /dev/null
+++ b/ops/kontemplate/docs/resource-sets.md
@@ -0,0 +1,170 @@
+Resource Sets
+================
+
+Resource sets are collections of Kubernetes resources that should be passed to `kubectl` together.
+
+Technically a resource set is simply a folder with a few YAML and/or JSON templates in it.
+
+<!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-refresh-toc -->
+**Table of Contents**
+
+- [Resource Sets](#resource-sets)
+- [Creating resource sets](#creating-resource-sets)
+    - [Default variables](#default-variables)
+- [Including resource sets](#including-resource-sets)
+    - [Fields](#fields)
+        - [`name`](#name)
+        - [`path`](#path)
+        - [`values`](#values)
+        - [`args`](#args)
+        - [`include`](#include)
+    - [Multiple includes](#multiple-includes)
+    - [Nesting resource sets](#nesting-resource-sets)
+        - [Caveats](#caveats)
+
+<!-- markdown-toc end -->
+
+# Creating resource sets
+
+Simply create a folder in your Kontemplate repository and place a YAML or JSON file in it. These
+files get interpreted as [templates][] during Kontemplate runs and variables (as well as template
+logic or functions) will be interpolated.
+
+Refer to the template documentation for information on how to write templates.
+
+## Default variables
+
+Sometimes it is useful to specify default values for variables that should be interpolated during
+a run if the [cluster configuration][] does not specify a variable explicitly.
+
+This can be done simply by placing a `default.yaml` or `default.json` file in the resource set
+folder and filling it with key/value pairs of the intended default variables.
+
+Kontemplate will error during interpolation if any variables are left unspecified.
+
+# Including resource sets
+
+Under the cluster configuration `include` key resource sets are included and required variables
+are specified. For example:
+
+```yaml
+include:
+  - name: some-api
+    values:
+      version: 1.2-SNAPSHOT
+```
+
+This will include a resource set from a folder called `some-api` and set the specified `version` variable.
+
+## Fields
+
+The available fields when including a resource set are these:
+
+### `name`
+
+The `name` field contains the name of the resource set. This name can be used to refer to the resource set
+when specifying explicit includes or excludes during a run.
+
+By default it is assumed that the `name` is the path to the resource set folder, but this can be overridden.
+
+This field is **required**.
+
+### `path`
+
+The `path` field specifies an explicit path to a resource set folder in the case that it should differ from
+the resource set's `name`.
+
+This field is **optional**.
+
+### `values`
+
+The `values` field specifies key/values pairs of variables that should be available during templating.
+
+This field is **optional**.
+
+### `args`
+
+The `args` field specifies a list of arguments that should be passed to `kubectl`.
+
+This field is **optional**.
+
+### `include`
+
+The `include` field specifies additional resource sets that should be included and that should inherit the
+variables of this resource set.
+
+The fully qualified names of "nested" resource sets are set to `${PARENT_NAME}/${CHILD_NAME}` and paths are
+merged in the same way.
+
+This makes it easy to organise different resource sets as "groups" to include / exclude them collectively
+during runs.
+
+This field is **optional**.
+
+## Multiple includes
+
+Resource sets can be included multiple times with different configurations. In this case it is recommended
+to set the `path` and `name` fields explicitly. For example:
+
+```yaml
+include:
+  - name: forwarder-europe
+    path: tools/forwarder
+    values:
+      source: europe
+  - name: forwarder-asia
+    path: tools/forwarder
+    values:
+      source: asia
+```
+
+The two different configurations can be referred to by their set names, but will use the same resource
+templates with different configurations.
+
+## Nesting resource sets
+
+As mentioned above for the `include` field, resource sets can be nested. This lets users group resource
+sets in logical ways using simple folder structures.
+
+Assuming a folder structure like:
+
+```
+├── backend
+│   ├── auth-api
+│   ├── message-api
+│   └── order-api
+└── frontend
+    ├── app-page
+    └── login-page
+```
+
+With each of these folders being a resource set, they could be included in a cluster configuration like so:
+
+```yaml
+include:
+  - name: backend
+    include:
+      - name: auth-api
+      - name: message-api
+      - name: order-api
+  - name: frontend:
+    include:
+      - name: app-page
+      - name: login-page
+```
+
+Kontemplate could then be run with, for example, `--include backend` to only include the resource sets nested
+in the backend group. Specific resource sets can also be targeted, for example as `--include backend/order-api`.
+
+Variables specified in the parent resource set are inherited by the children.
+
+### Caveats
+
+Two caveats apply that users should be aware of:
+
+1. The parent resource set can not contain any resource templates itself.
+
+2. Only one level of nesting is supported. Specifying `include` again on a nested resource set will be ignored.
+
+[templates]: templates.md
+[cluster configuration]: cluster-config.md
diff --git a/ops/kontemplate/docs/templates.md b/ops/kontemplate/docs/templates.md
new file mode 100644
index 000000000000..32da205108de
--- /dev/null
+++ b/ops/kontemplate/docs/templates.md
@@ -0,0 +1,153 @@
+Kontemplate templates
+=====================
+
+The template file format is based on Go's [templating engine][] in combination
+with a small extension library called [sprig][] that adds additional template
+functions.
+
+Go templates can either simply display variables or build more complicated
+*pipelines* in which variables are passed to functions for further processing,
+or in which conditionals are evaluated for more complex template logic.
+
+It is recommended that you check out the Golang [documentation][] for the templating
+engine in addition to the cherry-picked features listed here.
+
+<!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-refresh-toc -->
+**Table of Contents**
+
+- [Kontemplate templates](#kontemplate-templates)
+    - [Basic variable interpolation](#basic-variable-interpolation)
+        - [Example:](#example)
+    - [Template functions](#template-functions)
+    - [Examples:](#examples)
+    - [Conditionals & ranges](#conditionals--ranges)
+    - [Caveats](#caveats)
+
+<!-- markdown-toc end -->
+
+## Basic variable interpolation
+
+The basic template format uses `{{ .variableName }}` as the interpolation format.
+
+### Example:
+
+Assuming that you include a resource set as such:
+
+```
+- name: api-gateway
+  values:
+    internalHost: http://my-internal-host/
+```
+
+And the api-gateway resource set includes a ConfigMap (some fields left out for
+the example):
+
+```
+# api-gateway/configmap.yaml:
+---
+kind: ConfigMap
+metadata:
+  name: api-gateway-config
+data:
+  internalHost: {{ .internalHost }}
+```
+
+The resulting output will be:
+
+```
+
+---
+kind: ConfigMap
+metadata:
+  name: api-gateway-config
+data:
+  internalHost: http://my-internal-host/
+```
+
+## Template functions
+
+Go templates support template functions which you can think of as a sort of
+shell-like pipeline where text flows through transformations from left to
+right.
+
+Some template functions come from Go's standard library and are listed in the
+[Go documentation][]. In addition the functions declared by [sprig][] are
+available in kontemplate, as well as five custom functions:
+
+* `json`: Encodes any supplied data structure as JSON.
+* `gitHEAD`: Retrieves the commit hash at Git `HEAD`.
+* `passLookup`: Looks up the supplied key in [pass][].
+* `insertFile`: Insert the contents of the given file in the resource
+  set folder as a string.
+* `insertTemplate`: Insert the contents of the given template in the resource
+  set folder as a string.
+
+## Examples:
+
+```
+# With the following values:
+name: Donald
+certKeyPath: my-website/cert-key
+
+# The following interpolations are possible:
+
+{{ .name | upper }}
+-> DONALD
+
+{{ .name | upper | repeat 2 }}
+-> DONALD DONALD
+
+{{ .certKeyPath | passLookup }}
+-> Returns content of 'my-website/cert-key' from pass
+
+{{ gitHEAD }}
+-> Returns the Git commit hash at HEAD.
+```
+
+## Conditionals & ranges
+
+Some logic is supported in Golang templates and can be used in Kontemplate, too.
+
+With the following values:
+
+```
+useKube2IAM: true
+servicePorts:
+  - 8080
+  - 9090
+```
+
+The following interpolations are possible:
+
+```
+# Conditionally insert something in the template:
+metadata:
+  annotations:
+    foo: bar
+    {{ if .useKube2IAM -}} iam.amazonaws.com/role: my-api {{- end }}
+```
+
+```
+# Iterate over a list of values
+ports:
+  {{ range .servicePorts }}
+  - port: {{ . }}
+  {{ end }}
+```
+
+Check out the Golang documentation (linked above) for more information about template logic.
+
+## Caveats
+
+Kontemplate does not by itself parse any of the content of the templates, which
+means that it does not validate whether the resources you supply are valid YAML
+or JSON.
+
+You can perform some validation by using `kontemplate apply --dry-run` which
+will make use of the Dry-Run functionality in `kubectl`.
+
+[templating engine]: https://golang.org/pkg/text/template/
+[documentation]: https://golang.org/pkg/text/template/
+[sprig]: http://masterminds.github.io/sprig/
+[Go documentation]: https://golang.org/pkg/text/template/#hdr-Functions
+[pass]: https://www.passwordstore.org/
diff --git a/ops/kontemplate/docs/tips-and-tricks.md b/ops/kontemplate/docs/tips-and-tricks.md
new file mode 100644
index 000000000000..5401ac91e5a3
--- /dev/null
+++ b/ops/kontemplate/docs/tips-and-tricks.md
@@ -0,0 +1,77 @@
+Kontemplate tips & tricks
+=========================
+
+<!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-refresh-toc -->
+**Table of Contents**
+
+- [Kontemplate tips & tricks](#kontemplate-tips--tricks)
+    - [Update Deployments when ConfigMaps change](#update-deployments-when-configmaps-change)
+    - [direnv & pass](#direnv--pass)
+
+<!-- markdown-toc end -->
+
+## Update Deployments when ConfigMaps change
+
+Kubernetes does [not currently][] have the ability to perform rolling updates
+of Deployments and other resource types when `ConfigMap` or `Secret` objects
+are updated.
+
+It is possible to make use of annotations and templating functions in
+Kontemplate to force updates to these resources anyways.
+ 
+For example:
+
+```yaml
+# A ConfigMap that contains some configuration for your app
+---
+kind: ConfigMap
+metadata:
+  name: app-config
+data:
+  app.conf: |
+    name: {{ .appName }}
+    foo: bar
+```
+
+Now whenever the `appName` variable changes or we make an edit to the
+`ConfigMap` we would like to update the `Deployment` making use of it, too. We
+can do this by adding a hash of the parsed template to the annotations of the
+created `Pod` objects:
+
+```yaml
+
+---
+kind: Deployment
+metadata:
+  name: app
+spec:
+  template:
+    metadata:
+      annotations:
+        configHash: {{ insertTemplate "app-config.yaml" | sha256sum }}
+    spec:
+      containers:
+        - name: app
+          # Some details omitted ... 
+          volumeMounts:
+            - name: config
+              mountPath: /etc/app/
+      volumes:
+        - name: config
+          configMap:
+            name: app-config
+```
+
+Now any change to the `ConfigMap` - either by directly editing the yaml file or
+via a changed template variable - will cause the annotation to change,
+triggering a rolling update of all relevant pods.
+
+## direnv & pass
+
+Users of `pass` may have multiple different password stores on their machines.
+Assuming that `kontemplate` configuration exists somewhere on the filesystem
+per project, it is easy to use [direnv][] to switch to the correct
+`PASSWORD_STORE_DIR` variable when entering the folder.
+
+[not currently]: https://github.com/kubernetes/kubernetes/issues/22368
+[direnv]: https://direnv.net/