Restore templates

scripts/templating/README.md

@@ -0,0 +1,88 @@
+This folder contains the templates and a home-brewed templating system called
+Batesian for creating the spec. Batesian uses the templating system Jinja2 in
+Python.
+
+Installation
+------------
+```
+ $ pip install Jinja2
+```
+
+Running
+-------
+To pass arbitrary files (not limited to RST) through the templating system:
+```
+ $ python build.py -i matrix_templates /random/file/path/here.rst
+```
+
+The template output can be found at ``out/here.rst``. For a full list of
+options, type ``python build.py --help``.
+
+Developing
+----------
+
+### Sections and Units
+Batesian is built around the concept of Sections and Units. Sections are strings
+which will be inserted into the provided document. Every section has a unique
+key name which is the template variable that it represents. Units are arbitrary
+python data. They are also represented by unique key names.
+
+### Adding template variables
+If you want to add a new template variable e.g. `{{foo_bar}}` which is replaced
+with the text `foobar`, you need to add a new Section:
+
+ - Open `matrix_templates/sections.py`.
+ - Add a new function to `MatrixSections` called `render_foo_bar`. The function
+   name after `render_` determines the template variable name, and the return
+   value of this function determines what will be inserted.
+
+   ```python
+   def render_foo_bar(self):
+       return "foobar"
+   ```
+ - Run `build.py` with a file which has `{{foo_bar}}` in it, and it will be
+   replaced with `foobar`.
+
+### Adding data for template variables
+If you want to expose arbitrary data which can be used by `MatrixSections`, you
+need to add a new Unit:
+
+ - Open `matrix_templates/units.py`.
+ - Add a new function to `MatrixUnits` called `load_some_data`. Similar to
+   sections, the function name after `load_` determines the unit name, and the
+   return value of this function determines the value of the unit.
+
+   ```python
+   def load_some_data(self):
+       return {
+          "data": "this could be JSON from file from json.loads()",
+          "processed_data": "this data may have helper keys added",
+          "types": "it doesn't even need to be a dict. Whatever you want!"
+       }
+   ```
+ - In `MatrixSections`, you can now call `self.units.get("some_data")` to
+   retrieve the value you returned.
+
+### Using Jinja templates
+Sections can use Jinja templates to return text. Batesian will attempt to load
+all templates from `matrix_templates/templates/`. These can be accessed in
+Section code via `template = self.env.get_template("name_of_template.tmpl")`. At
+this point, the `template` is just a standard `jinja2.Template`. In fact,
+`self.env` is just a `jinja2.Environment`.
+
+### Debugging
+If you don't know why your template isn't behaving as you'd expect, or you just
+want to add some informative logging, use `self.log` in either the Sections
+class or Units class. You'll need to add `-v` to `build.py` for these lines to
+show.
+
+About
+-----
+
+Batesian was designed to be extremely simple and just use Python as its language
+rather than another intermediary language like some other templating systems.
+This provides a **lot** of flexibility since you aren't contrained by a
+templating language. Batesian provides a thin abstraction over Jinja which is
+very useful when you want to do random bits of processing then dump the output
+into a Jinja template. Its name is derived from Batesian mimicry due to how the
+templating system uses Python as its language, but in a harmless way.

scripts/templating/batesian/__init__.py

@@ -0,0 +1,38 @@
+# Copyright 2016 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+class AccessKeyStore(object):
+    """Storage for arbitrary data. Monitors get calls so we know if they
+    were used or not."""
+
+    def __init__(self, existing_data=None):
+        if not existing_data:
+            existing_data = {}
+        self.data = existing_data
+        self.accessed_set = set()
+
+    def keys(self):
+        return self.data.keys()
+
+    def add(self, key, unit_dict):
+        self.data[key] = unit_dict
+
+    def get(self, key):
+        self.accessed_set.add(key)
+        return self.data[key]
+
+    def get_unaccessed_set(self):
+        data_list = set(self.data.keys())
+        return data_list - self.accessed_set

scripts/templating/batesian/sections.py

@@ -0,0 +1,77 @@
+# Copyright 2016 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Parent class for writing sections."""
+import inspect
+import os
+
+
+class Sections(object):
+    """A class which creates sections for each method starting with "render_".
+    The key for the section is the text after "render_"
+    e.g. "render_room_events" has the section key "room_events"
+    """
+
+    def __init__(self, env, units, debug=False):
+        self.env = env
+        self.units = units
+        self.debug = debug
+
+    def log(self, text):
+        if self.debug:
+            print("batesian:sections: %s" % text)
+
+    def get_sections(self):
+        render_list = inspect.getmembers(self, predicate=inspect.ismethod)
+        section_dict = {}
+        for (func_name, func) in render_list:
+            if not func_name.startswith("render_"):
+                continue
+            section_key = func_name[len("render_"):]
+            self.log("Generating section '%s'" % section_key)
+            section = func()
+            if isinstance(section, str):
+                if section_key in section_dict:
+                    raise Exception(
+                        ("%s : Section %s already exists. It must have been " +
+                        "generated dynamically. Check which render_ methods " +
+                        "return a dict.") %
+                        (func_name, section_key)
+                    )
+                section_dict[section_key] = section
+                self.log(
+                    "  Generated. Snippet => %s" % section[:60].replace("\n","")
+                )
+            elif isinstance(section, dict):
+                self.log("  Generated multiple sections:")
+                for (k, v) in section.items():
+                    if not isinstance(k, str) or not isinstance(v, str):
+                        raise Exception(
+                            ("Method %s returned multiple sections as a dict but " +
+                            "expected the dict elements to be strings but they aren't.") %
+                            (func_name, )
+                        )
+                    if k in section_dict:
+                        raise Exception(
+                            "%s tried to produce section %s which already exists." %
+                            (func_name, k)
+                        )
+                    section_dict[k] = v
+                    self.log(
+                        "  %s  =>  %s" % (k, v[:60].replace("\n",""))
+                    )
+            else:
+                raise Exception(
+                    "Section function '%s' didn't return a string/dict!" % func_name
+                )
+        return section_dict

scripts/templating/batesian/units.py

@@ -0,0 +1,59 @@
+# Copyright 2016 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Parent class for writing units."""
+import inspect
+
+
+class Units(object):
+
+    @staticmethod
+    def prop(obj, path):
+        # Helper method to extract nested property values
+        nested_keys = path.split("/")
+        val = obj
+        for key in nested_keys:
+            val = val.get(key, {})
+        return val
+
+
+    def __init__(self, debug=False, substitutions=None):
+        self.debug = debug
+
+        if substitutions is None:
+            self.substitutions = {}
+        else:
+            self.substitutions = substitutions
+
+    def log(self, text):
+        if self.debug:
+            func_name = ""
+            trace = inspect.stack()
+            if len(trace) > 1 and len(trace[1]) > 2:
+                func_name = trace[1][3] + ":"
+            print("batesian:units:%s %s" % (func_name, text))
+
+    def get_units(self, debug=False):
+        unit_list = inspect.getmembers(self, predicate=inspect.ismethod)
+        unit_dict = {}
+        for (func_name, func) in unit_list:
+            if not func_name.startswith("load_"):
+                continue
+            unit_key = func_name[len("load_"):]
+            if len(inspect.getargs(func.__code__).args) > 1:
+                unit_dict[unit_key] = func(self.substitutions)
+            else:
+                unit_dict[unit_key] = func()
+            self.log("Generated unit '%s'" % unit_key)
+
+        return unit_dict