From bd9609e071f7cae21dcfa67c246102f5396e8111 Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@igalia.com>
Date: Mon, 11 Dec 2017 09:41:42 +0100
Subject: [PATCH 01/31] Fix bug in cpuset:release()

Previously, release() would always fail.
---
 src/lib/cpuset.lua | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/lib/cpuset.lua b/src/lib/cpuset.lua
index e92d0143b1..afb9c1e8a6 100644
--- a/src/lib/cpuset.lua
+++ b/src/lib/cpuset.lua
@@ -78,8 +78,8 @@ end
 function CPUSet:release(cpu)
    local node = numa.cpu_get_numa_node(cpu)
    assert(node ~= nil, 'Failed to get NUMA node for CPU: '..cpu)
-   for cpu, avail in pairs(self.by_node[node]) do
-      if avail then
+   for x, avail in pairs(self.by_node[node]) do
+      if x == cpu then
          assert(self.by_node[node][cpu] == false)
          self.by_node[node][cpu] = true
          return

From fde343465b017528d86f5e66f5eb4620867aecec Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@igalia.com>
Date: Wed, 6 Dec 2017 14:29:12 +0100
Subject: [PATCH 02/31] Copy app.config to lib.ptree ("process tree")

This is the first step in refactoring the "app.config" mechanism to be
something that's more of a library and less like magical apps.
---
 src/lib/ptree/README.md                     | 145 +++
 src/lib/ptree/action_codec.lua              | 234 +++++
 src/lib/ptree/alarm_codec.lua               | 314 +++++++
 src/lib/ptree/channel.lua                   | 231 +++++
 src/lib/ptree/follower.lua                  |  96 ++
 src/lib/ptree/leader.lua                    | 956 ++++++++++++++++++++
 src/lib/ptree/support.lua                   | 231 +++++
 src/lib/ptree/support/snabb-softwire-v2.lua | 706 +++++++++++++++
 8 files changed, 2913 insertions(+)
 create mode 100644 src/lib/ptree/README.md
 create mode 100644 src/lib/ptree/action_codec.lua
 create mode 100644 src/lib/ptree/alarm_codec.lua
 create mode 100644 src/lib/ptree/channel.lua
 create mode 100644 src/lib/ptree/follower.lua
 create mode 100644 src/lib/ptree/leader.lua
 create mode 100644 src/lib/ptree/support.lua
 create mode 100644 src/lib/ptree/support/snabb-softwire-v2.lua

diff --git a/src/lib/ptree/README.md b/src/lib/ptree/README.md
new file mode 100644
index 0000000000..cb1d37dc8c
--- /dev/null
+++ b/src/lib/ptree/README.md
@@ -0,0 +1,145 @@
+# Config leader and follower
+
+Sometimes you want to query the state or configuration of a running
+Snabb data plane, or reload its configuration, or incrementally update
+that configuration.  However, you want to minimize the impact of
+configuration query and update on data plane performance.  The
+`Leader` and `Follower` apps are here to fulfill this need, while
+minimizing performance overhead.
+
+The high-level design is that a `Leader` app is responsible for
+knowing the state and configuration of a data plane.  The leader
+offers an interface to allow the outside world to query the
+configuration and state, and to request configuration updates.  To
+avoid data-plane overhead, the `Leader` app should be deployed in a
+separate process.  Because it knows the data-plane state, it can
+respond to queries directly, without involving the data plane.  It
+processes update requests into a form that the data plane can handle,
+and feeds those requests to the data plane via a high-performance
+back-channel.
+
+The data plane runs a `Follower` app that reads and applies update
+messages sent to it from the leader.  Checking for update availability
+requires just a memory access, not a system call, so the overhead of
+including a follower in the data plane is very low.
+
+## Two protocols
+
+The leader communicates with its followers using a private protocol.
+Because the leader and the follower are from the same Snabb version,
+the details of this protocol are subject to change.  The private
+protocol's only design constraint is that it should cause the lowest
+overhead for the data plane.
+
+The leader communicates with the world via a public protocol.  The
+"snabb config" command-line tool speaks this protocol.  "snabb config
+get foo /bar" will find the local Snabb instance named "foo", open the
+UNIX socket that the "foo" instance is listening on, issue a request,
+then read the response, then close the socket.
+
+## Public protocol
+
+The design constraint on the public protocol is that it be expressive
+and future-proof.  We also want to enable the leader to talk to more
+than one "snabb config" at a time.  In particular someone should be
+able to have a long-lived "snabb config listen" session open, and that
+shouldn't impede someone else from doing a "snabb config get" to read
+state.
+
+To this end the public protocol container is very simple:
+
+```
+Message = Length "\n" RPC*
+```
+
+Length is a base-10 string of characters indicating the length of the
+message.  There may be a maximum length restriction.  This requires
+that "snabb config" build up the whole message as a string and measure
+its length, but that's OK.  Knowing the length ahead of time allows
+"snabb config" to use nonblocking operations to slurp up the whole
+message as a string.  A partial read can be resumed later.  The
+message can then be parsed without fear of blocking the main process.
+
+The RPC is an RPC request or response for the
+[`snabb-config-leader-v1` YANG
+schema](../../lib/yang/snabb-config-leader-v1.yang), expressed in the
+Snabb [textual data format for YANG data](../../lib/yang/README.md).
+For example the `snabb-config-leader-v1` schema supports a
+`get-config` RPC defined like this in the schema:
+
+```yang
+rpc get-config {
+  input {
+    leaf schema { type string; mandatory true; }
+    leaf revision { type string; }
+    leaf path { type string; default "/"; }
+  }
+  output {
+    leaf config { type string; }
+  }
+}
+```
+
+A request to this RPC might look like:
+
+```yang
+get-config {
+  schema snabb-softwire-v1;
+  path "/foo";
+}
+```
+
+As you can see, non-mandatory inputs can be left out.  A response
+might look like:
+
+```yang
+get-config {
+  config "blah blah blah";
+}
+```
+
+Responses are prefixed by the RPC name.  One message can include a
+number of RPCs; the RPCs will be made in order.  See the
+[`snabb-config-leader-v1` YANG
+schema](../../lib/yang/snabb-config-leader-v1.yang) for full details
+of available RPCs.
+
+## Private protocol
+
+The leader maintains a configuration for the program as a whole.  As
+it gets requests, it computes the set of changes to app graphs that
+would be needed to apply that configuration.  These changes are then
+passed through the private protocol to the follower.  No response from
+the follower is necessary.
+
+In some remote or perhaps not so remote future, all Snabb apps will
+have associated YANG schemas describing their individual
+configurations.  In this happy future, the generic way to ship
+configurations from the leader to a follower is by the binary
+serialization of YANG data, implemented already in the YANG modules.
+Until then however, there is also generic Lua data without a schema.
+The private protocol supports both kinds of information transfer.
+
+In the meantime, the way to indicate that an app's configuration data
+conforms to a YANG schema is to set the `schema_name` property on the
+app's class.
+
+The private protocol consists of binary messages passed over a ring
+buffer.  A follower's leader writes to the buffer, and the follower
+reads from it.  There are no other readers or writers.  Given that a
+message may in general be unbounded in size, whereas a ring buffer is
+naturally fixed, messages which may include arbtrary-sized data may be
+forced to put that data in the filesystem, and refer to it from the
+messages in the ring buffer.  Since this file system is backed by
+`tmpfs`, stalls will be minimal.
+
+## User interface
+
+The above sections document how the leader and follower apps are
+implemented so that a data-plane developer can understand the overhead
+of run-time (re)configuration.  End users won't be typing at a UNIX
+socket though; we include the `snabb config` program as a command-line
+interface to this functionality.
+
+See [the `snabb config` documentation](../../program/config/README.md)
+for full details.
diff --git a/src/lib/ptree/action_codec.lua b/src/lib/ptree/action_codec.lua
new file mode 100644
index 0000000000..305be9ed91
--- /dev/null
+++ b/src/lib/ptree/action_codec.lua
@@ -0,0 +1,234 @@
+-- Use of this source code is governed by the Apache 2.0 license; see COPYING.
+
+module(...,package.seeall)
+
+local S = require("syscall")
+local lib = require("core.lib")
+local ffi = require("ffi")
+local yang = require("lib.yang.yang")
+local binary = require("lib.yang.binary")
+local shm = require("core.shm")
+
+local action_names = { 'unlink_output', 'unlink_input', 'free_link',
+                       'new_link', 'link_output', 'link_input', 'stop_app',
+                       'start_app', 'reconfig_app',
+                       'call_app_method_with_blob', 'commit', 'shutdown' }
+local action_codes = {}
+for i, name in ipairs(action_names) do action_codes[name] = i end
+
+local actions = {}
+
+function actions.unlink_output (codec, appname, linkname)
+   local appname = codec:string(appname)
+   local linkname = codec:string(linkname)
+   return codec:finish(appname, linkname)
+end
+function actions.unlink_input (codec, appname, linkname)
+   local appname = codec:string(appname)
+   local linkname = codec:string(linkname)
+   return codec:finish(appname, linkname)
+end
+function actions.free_link (codec, linkspec)
+   local linkspec = codec:string(linkspec)
+   return codec:finish(linkspec)
+end
+function actions.new_link (codec, linkspec)
+   local linkspec = codec:string(linkspec)
+   return codec:finish(linkspec)
+end
+function actions.link_output (codec, appname, linkname, linkspec)
+   local appname = codec:string(appname)
+   local linkname = codec:string(linkname)
+   local linkspec = codec:string(linkspec)
+   return codec:finish(appname, linkname, linkspec)
+end
+function actions.link_input (codec, appname, linkname, linkspec)
+   local appname = codec:string(appname)
+   local linkname = codec:string(linkname)
+   local linkspec = codec:string(linkspec)
+   return codec:finish(appname, linkname, linkspec)
+end
+function actions.stop_app (codec, appname)
+   local appname = codec:string(appname)
+   return codec:finish(appname)
+end
+function actions.start_app (codec, appname, class, arg)
+   local appname = codec:string(appname)
+   local _class = codec:class(class)
+   local config = codec:config(class, arg)
+   return codec:finish(appname, _class, config)
+end
+function actions.reconfig_app (codec, appname, class, arg)
+   local appname = codec:string(appname)
+   local _class = codec:class(class)
+   local config = codec:config(class, arg)
+   return codec:finish(appname, _class, config)
+end
+function actions.call_app_method_with_blob (codec, appname, methodname, blob)
+   local appname = codec:string(appname)
+   local methodname = codec:string(methodname)
+   local blob = codec:blob(blob)
+   return codec:finish(appname, methodname, blob)
+end
+function actions.commit (codec)
+   return codec:finish()
+end
+function actions.shutdown (codec)
+   return codec:finish()
+end
+
+local public_names = {}
+local function find_public_name(obj)
+   if public_names[obj] then return unpack(public_names[obj]) end
+   for modname, mod in pairs(package.loaded) do
+      if type(mod) == 'table' then
+         for name, val in pairs(mod) do
+            if val == obj then
+               if type(val) == 'table' and type(val.new) == 'function' then
+                  public_names[obj] = { modname, name }
+                  return modname, name
+               end
+            end
+         end
+      end
+   end
+   error('could not determine public name for object: '..tostring(obj))
+end
+
+local function random_file_name()
+   local basename = 'app-conf-'..lib.random_printable_string(160)
+   return shm.root..'/'..shm.resolve(basename)
+end
+
+local function encoder()
+   local encoder = { out = {} }
+   function encoder:uint32(len)
+      table.insert(self.out, ffi.new('uint32_t[1]', len))
+   end
+   function encoder:string(str)
+      self:uint32(#str)
+      local buf = ffi.new('uint8_t[?]', #str)
+      ffi.copy(buf, str, #str)
+      table.insert(self.out, buf)
+   end
+   function encoder:blob(blob)
+      self:uint32(ffi.sizeof(blob))
+      table.insert(self.out, blob)
+   end
+   function encoder:class(class)
+      local require_path, name = find_public_name(class)
+      self:string(require_path)
+      self:string(name)
+   end
+   function encoder:config(class, arg)
+      local file_name = random_file_name()
+      if class.yang_schema then
+         yang.compile_config_for_schema_by_name(class.yang_schema, arg,
+                                                file_name)
+      else
+         if arg == nil then arg = {} end
+         binary.compile_ad_hoc_lua_data_to_file(file_name, arg)
+      end
+      self:string(file_name)
+   end
+   function encoder:finish()
+      local size = 0
+      for _,src in ipairs(self.out) do size = size + ffi.sizeof(src) end
+      local dst = ffi.new('uint8_t[?]', size)
+      local pos = 0
+      for _,src in ipairs(self.out) do
+         ffi.copy(dst + pos, src, ffi.sizeof(src))
+         pos = pos + ffi.sizeof(src)
+      end
+      return dst, size
+   end
+   return encoder
+end
+
+function encode(action)
+   local name, args = unpack(action)
+   local codec = encoder()
+   codec:uint32(assert(action_codes[name], name))
+   return assert(actions[name], name)(codec, unpack(args))
+end
+
+local uint32_ptr_t = ffi.typeof('uint32_t*')
+local function decoder(buf, len)
+   local decoder = { buf=buf, len=len, pos=0 }
+   function decoder:read(count)
+      local ret = self.buf + self.pos
+      self.pos = self.pos + count
+      assert(self.pos <= self.len)
+      return ret
+   end
+   function decoder:uint32()
+      return ffi.cast(uint32_ptr_t, self:read(4))[0]
+   end
+   function decoder:string()
+      local len = self:uint32()
+      return ffi.string(self:read(len), len)
+   end
+   function decoder:blob()
+      local len = self:uint32()
+      local blob = ffi.new('uint8_t[?]', len)
+      ffi.copy(blob, self:read(len), len)
+      return blob
+   end
+   function decoder:class()
+      local require_path, name = self:string(), self:string()
+      return assert(require(require_path)[name])
+   end
+   function decoder:config()
+      return binary.load_compiled_data_file(self:string()).data
+   end
+   function decoder:finish(...)
+      return { ... }
+   end
+   return decoder
+end
+
+function decode(buf, len)
+   local codec = decoder(buf, len)
+   local name = assert(action_names[codec:uint32()])
+   return { name, assert(actions[name], name)(codec) }
+end
+
+function selftest ()
+   print('selftest: lib.ptree.action_codec')
+   local function serialize(data)
+      local tmp = random_file_name()
+      print('serializing to:', tmp)
+      binary.compile_ad_hoc_lua_data_to_file(tmp, data)
+      local loaded = binary.load_compiled_data_file(tmp)
+      assert(loaded.schema_name == '')
+      assert(lib.equal(data, loaded.data))
+      os.remove(tmp)
+   end
+   serialize('foo')
+   serialize({foo='bar'})
+   serialize({foo={qux='baz'}})
+   serialize(1)
+   serialize(1LL)
+   local function test_action(action)
+      local encoded, len = encode(action)
+      local decoded = decode(encoded, len)
+      assert(lib.equal(action, decoded))
+   end
+   local appname, linkname, linkspec = 'foo', 'bar', 'foo.a -> bar.q'
+   local class, arg = require('apps.basic.basic_apps').Tee, {}
+   -- Because lib.equal only returns true when comparing cdata of
+   -- exactly the same type, here we have to use uint8_t[?].
+   local methodname, blob = 'zog', ffi.new('uint8_t[?]', 3, 1, 2, 3)
+   test_action({'unlink_output', {appname, linkname}})
+   test_action({'unlink_input', {appname, linkname}})
+   test_action({'free_link', {linkspec}})
+   test_action({'new_link', {linkspec}})
+   test_action({'link_output', {appname, linkname, linkspec}})
+   test_action({'link_input', {appname, linkname, linkspec}})
+   test_action({'stop_app', {appname}})
+   test_action({'start_app', {appname, class, arg}})
+   test_action({'reconfig_app', {appname, class, arg}})
+   test_action({'call_app_method_with_blob', {appname, methodname, blob}})
+   test_action({'commit', {}})
+   print('selftest: ok')
+end
diff --git a/src/lib/ptree/alarm_codec.lua b/src/lib/ptree/alarm_codec.lua
new file mode 100644
index 0000000000..7b7c598751
--- /dev/null
+++ b/src/lib/ptree/alarm_codec.lua
@@ -0,0 +1,314 @@
+-- Use of this source code is governed by the Apache 2.0 license; see COPYING.
+
+module(...,package.seeall)
+
+local S = require("syscall")
+local channel = require("lib.ptree.channel")
+local ffi = require("ffi")
+
+local UINT32_MAX = 0xffffffff
+
+local alarm_names = { 'raise_alarm', 'clear_alarm', 'add_to_inventory', 'declare_alarm' }
+local alarm_codes = {}
+for i, name in ipairs(alarm_names) do alarm_codes[name] = i end
+
+local alarms = {}
+
+function alarms.raise_alarm (codec, resource, alarm_type_id, alarm_type_qualifier,
+   perceived_severity, alarm_text)
+
+   local resource = codec:string(resource)
+   local alarm_type_id = codec:string(alarm_type_id)
+   local alarm_type_qualifier = codec:string(alarm_type_qualifier)
+
+   local perceived_severity = codec:maybe_string(perceived_severity)
+   local alarm_text = codec:maybe_string(alarm_text)
+
+   return codec:finish(resource, alarm_type_id, alarm_type_qualifier,
+                       perceived_severity, alarm_text)
+end
+function alarms.clear_alarm (codec, resource, alarm_type_id, alarm_type_qualifier)
+   local resource = codec:string(resource)
+   local alarm_type_id = codec:string(alarm_type_id)
+   local alarm_type_qualifier = codec:string(alarm_type_qualifier)
+
+   return codec:finish(resource, alarm_type_id, alarm_type_qualifier)
+end
+function alarms.add_to_inventory (codec, alarm_type_id, alarm_type_qualifier,
+   resource, has_clear, description)
+
+   local alarm_type_id = codec:string(alarm_type_id)
+   local alarm_type_qualifier = codec:maybe_string(alarm_type_qualifier)
+
+   local resource = codec:string(resource)
+   local has_clear = codec:string((has_clear and "true" or "false"))
+   local description = codec:maybe_string(description)
+
+   return codec:finish(alarm_type_id, alarm_type_qualifier,
+                       resource, has_clear, description)
+end
+function alarms.declare_alarm (codec, resource, alarm_type_id, alarm_type_qualifier,
+   perceived_severity, alarm_text)
+
+   local resource = codec:string(resource)
+   local alarm_type_id = codec:string(alarm_type_id)
+   local alarm_type_qualifier = codec:maybe_string(alarm_type_qualifier)
+
+   local perceived_severity = codec:maybe_string(perceived_severity)
+   local alarm_text = codec:maybe_string(alarm_text)
+
+   return codec:finish(resource, alarm_type_id, alarm_type_qualifier,
+                       perceived_severity, alarm_text)
+end
+
+local function encoder()
+   local encoder = { out = {} }
+   function encoder:uint32(len)
+      table.insert(self.out, ffi.new('uint32_t[1]', len))
+   end
+   function encoder:string(str)
+      self:uint32(#str)
+      local buf = ffi.new('uint8_t[?]', #str)
+      ffi.copy(buf, str, #str)
+      table.insert(self.out, buf)
+   end
+   function encoder:maybe_string(str)
+      if str == nil then
+         self:uint32(UINT32_MAX)
+      else
+         self:string(str)
+      end
+   end
+   function encoder:finish()
+      local size = 0
+      for _,src in ipairs(self.out) do size = size + ffi.sizeof(src) end
+      local dst = ffi.new('uint8_t[?]', size)
+      local pos = 0
+      for _,src in ipairs(self.out) do
+         ffi.copy(dst + pos, src, ffi.sizeof(src))
+         pos = pos + ffi.sizeof(src)
+      end
+      return dst, size
+   end
+   return encoder
+end
+
+function encode_raise_alarm (...)
+   local codec = encoder()
+   codec:uint32(assert(alarm_codes['raise_alarm']))
+   return assert(alarms['raise_alarm'])(codec, ...)
+end
+
+function encode_clear_alarm (...)
+   local codec = encoder()
+   codec:uint32(assert(alarm_codes['clear_alarm']))
+   return assert(alarms['clear_alarm'])(codec, ...)
+end
+
+function encode_add_to_inventory (...)
+   local codec = encoder()
+   codec:uint32(assert(alarm_codes['add_to_inventory']))
+   return assert(alarms['add_to_inventory'])(codec, ...)
+end
+
+function encode_declare_alarm (...)
+   local codec = encoder()
+   codec:uint32(assert(alarm_codes['declare_alarm']))
+   return assert(alarms['declare_alarm'])(codec, ...)
+end
+
+local uint32_ptr_t = ffi.typeof('uint32_t*')
+local function decoder(buf, len)
+   local decoder = { buf=buf, len=len, pos=0 }
+   function decoder:read(count)
+      local ret = self.buf + self.pos
+      self.pos = self.pos + count
+      assert(self.pos <= self.len)
+      return ret
+   end
+   function decoder:uint32()
+      return ffi.cast(uint32_ptr_t, self:read(4))[0]
+   end
+   function decoder:string()
+      local len = self:uint32()
+      return ffi.string(self:read(len), len)
+   end
+   function decoder:maybe_string()
+      local len = self:uint32()
+      if len == UINT32_MAX then return nil end
+      return ffi.string(self:read(len), len)
+   end
+   function decoder:finish(...)
+      return { ... }
+   end
+   return decoder
+end
+
+function decode(buf, len)
+   local codec = decoder(buf, len)
+   local name = assert(alarm_names[codec:uint32()])
+   return { name, assert(alarms[name], name)(codec) }
+end
+
+---
+
+local alarms_channel
+
+function get_channel()
+   if alarms_channel then return alarms_channel end
+   local name = '/'..S.getpid()..'/alarms-follower-channel'
+   local success, value = pcall(channel.open, name)
+   if success then
+      alarms_channel = value
+   else
+      alarms_channel = channel.create('alarms-follower-channel', 1e6)
+   end
+   return alarms_channel
+end
+
+local function normalize (t, attrs)
+   t = t or {}
+   local ret = {}
+   for i, k in ipairs(attrs) do ret[i] = t[k] end
+   return unpack(ret)
+end
+
+local alarm = {
+   key_attrs = {'resource', 'alarm_type_id', 'alarm_type_qualifier'},
+   args_attrs = {'perceived_severity', 'alarm_text'},
+}
+function alarm:normalize_key (t)
+   return normalize(t, self.key_attrs)
+end
+function alarm:normalize_args (t)
+   return normalize(t, self.args_attrs)
+end
+
+-- To be used by the leader to group args into key and args.
+function to_alarm (args)
+   local key = {
+      resource = args[1],
+      alarm_type_id = args[2],
+      alarm_type_qualifier = args[3],
+   }
+   local args = {
+      perceived_severity = args[4],
+      alarm_text = args[5],
+   }
+   return key, args
+end
+
+local alarm_type = {
+   key_attrs = {'alarm_type_id', 'alarm_type_qualifier'},
+   args_attrs = {'resource', 'has_clear', 'description'},
+}
+function alarm_type:normalize_key (t)
+   return normalize(t, self.key_attrs)
+end
+function alarm_type:normalize_args (t)
+   return normalize(t, self.args_attrs)
+end
+
+function to_alarm_type (args)
+   local alarm_type_id, alarm_type_qualifier, resource, has_clear, description = unpack(args)
+   local key = {
+      alarm_type_id = args[1],
+      alarm_type_qualifier = args[2],
+   }
+   local args = {
+      resource = args[3],
+      has_clear = args[4],
+      description = args[5],
+   }
+   return key, args
+end
+
+function raise_alarm (key, args)
+   local channel = get_channel()
+   if channel then
+      local resource, alarm_type_id, alarm_type_qualifier = alarm:normalize_key(key)
+      local perceived_severity, alarm_text = alarm:normalize_args(args)
+      local buf, len = encode_raise_alarm(
+         resource, alarm_type_id, alarm_type_qualifier,
+         perceived_severity, alarm_text
+      )
+      channel:put_message(buf, len)
+   end
+end
+
+function clear_alarm (key)
+   local channel = get_channel()
+   if channel then
+      local resource, alarm_type_id, alarm_type_qualifier = alarm:normalize_key(key)
+      local buf, len = encode_clear_alarm(resource, alarm_type_id, alarm_type_qualifier)
+      channel:put_message(buf, len)
+   end
+end
+
+function add_to_inventory (key, args)
+   local channel = get_channel()
+   if channel then
+      local alarm_type_id, alarm_type_qualifier = alarm_type:normalize_key(key)
+      local resource, has_clear, description = alarm_type:normalize_args(args)
+      local buf, len = encode_add_to_inventory(
+         alarm_type_id, alarm_type_qualifier,
+         resource, has_clear, description
+      )
+      channel:put_message(buf, len)
+   end
+end
+
+function declare_alarm (key, args)
+   local channel = get_channel()
+   if channel then
+      local resource, alarm_type_id, alarm_type_qualifier = alarm:normalize_key(key)
+      local perceived_severity, alarm_text = alarm:normalize_args(args)
+      local buf, len = encode_declare_alarm(
+         resource, alarm_type_id, alarm_type_qualifier,
+         perceived_severity, alarm_text
+      )
+      channel:put_message(buf, len)
+   end
+end
+
+function selftest ()
+   print('selftest: lib.ptree.alarm_codec')
+   local lib = require("core.lib")
+   local function test_alarm (name, args)
+      local encoded, len
+      if name == 'raise_alarm' then
+         encoded, len = encode_raise_alarm(unpack(args))
+      elseif name == 'clear_alarm' then
+         encoded, len = encode_clear_alarm(unpack(args))
+      else
+         error('not valid alarm name: '..alarm)
+      end
+      local decoded = decode(encoded, len)
+      assert(lib.equal({name, args}, decoded))
+   end
+   local function test_raise_alarm ()
+      local key = {resource='res1', alarm_type_id='type1', alarm_type_qualifier=''}
+      local args = {perceived_severity='critical'}
+
+      local resource, alarm_type_id, alarm_type_qualifier = alarm:normalize_key(key)
+      local perceived_severity, alarm_text = alarm:normalize_args(args)
+      local alarm = {resource, alarm_type_id, alarm_type_qualifier,
+                     perceived_severity, alarm_text}
+
+      test_alarm('raise_alarm', alarm)
+   end
+   local function test_clear_alarm ()
+      local key = {resource='res1', alarm_type_id='type1', alarm_type_qualifier=''}
+      local resource, alarm_type_id, alarm_type_qualifier = alarm:normalize_key(key)
+      local alarm = {resource, alarm_type_id, alarm_type_qualifier}
+      test_alarm('clear_alarm', alarm)
+   end
+
+   test_raise_alarm()
+   test_clear_alarm()
+
+   local a, b = normalize({b='foo'}, {'a', 'b'})
+   assert(a == nil and b == 'foo')
+
+   print('selftest: ok')
+end
diff --git a/src/lib/ptree/channel.lua b/src/lib/ptree/channel.lua
new file mode 100644
index 0000000000..f78926397e
--- /dev/null
+++ b/src/lib/ptree/channel.lua
@@ -0,0 +1,231 @@
+-- Use of this source code is governed by the Apache 2.0 license; see COPYING.
+
+module(...,package.seeall)
+
+-- A channel is a ring buffer used by the config leader app to send
+-- updates to a follower.  Each follower has its own ring buffer and is
+-- the only reader to the buffer.  The config leader is the only writer
+-- to these buffers also.  The ring buffer is just bytes; putting a
+-- message onto the buffer will write a header indicating the message
+-- size, then the bytes of the message.  The channel ring buffer is
+-- mapped into shared memory.  Access to a channel will never block or
+-- cause a system call.
+
+local ffi = require('ffi')
+local S = require("syscall")
+local lib = require('core.lib')
+local shm = require('core.shm')
+
+local ring_buffer_t = ffi.typeof([[struct {
+   uint32_t read;
+   uint32_t write;
+   uint32_t size;
+   uint8_t buf[?];
+}]])
+
+-- Q: Why not just use shm.map?
+-- A: We need a variable-sized mapping.
+local function create_ring_buffer (name, size)
+   local path = shm.resolve(name)
+   shm.mkdir(lib.dirname(path))
+   path = shm.root..'/'..path
+   local fd, err = S.open(path, "creat, rdwr, excl", '0664')
+   if not fd then
+      err = tostring(err or "unknown error")
+      error('error creating file "'..path..'": '..err)
+   end
+   local len = ffi.sizeof(ring_buffer_t, size)
+   assert(fd:ftruncate(len), "ring buffer: ftruncate failed")
+   local mem, err = S.mmap(nil, len, "read, write", "shared", fd, 0)
+   fd:close()
+   if mem == nil then error("mmap failed: " .. tostring(err)) end
+   mem = ffi.cast(ffi.typeof("$*", ring_buffer_t), mem)
+   ffi.gc(mem, function (ptr) S.munmap(ptr, len) end)
+   mem.size = size
+   return mem
+end
+
+local function open_ring_buffer (name)
+   local path = shm.resolve(name)
+   path = shm.root..'/'..path
+   local fd, err = S.open(path, "rdwr")
+   if not fd then
+      err = tostring(err or "unknown error")
+      error('error opening file "'..path..'": '..err)
+   end
+   local stat = S.fstat(fd)
+   local len = stat and stat.size
+   if len < ffi.sizeof(ring_buffer_t, 0) then
+      error("unexpected size for ring buffer")
+   end
+   local mem, err = S.mmap(nil, len, "read, write", "shared", fd, 0)
+   fd:close()
+   if mem == nil then error("mmap failed: " .. tostring(err)) end
+   mem = ffi.cast(ffi.typeof("$*", ring_buffer_t), mem)
+   ffi.gc(mem, function (ptr) S.munmap(ptr, len) end)
+   if len ~= ffi.sizeof(ring_buffer_t, mem.size) then
+      error("unexpected ring buffer size: "..tostring(len))
+   end
+   return mem
+end
+
+local function to_uint32 (num)
+   local buf = ffi.new('uint32_t[1]')
+   buf[0] = num
+   return buf[0]
+end
+
+local function read_avail (ring)
+   lib.compiler_barrier()
+   return to_uint32(ring.write - ring.read)
+end
+
+local function write_avail (ring)
+   return ring.size - read_avail(ring)
+end
+
+Channel = {}
+
+-- Messages typically encode up to 3 or 4 strings like app names, link
+-- names, module names, or the like.  All of that and the length headers
+-- probably fits within 256 bytes per message certainly.  So make room
+-- for around 4K messages, why not.
+local default_buffer_size = 1024*1024
+function create(name, size)
+   local ret = {}
+   size = size or default_buffer_size
+   ret.ring_buffer = create_ring_buffer(name, size)
+   return setmetatable(ret, {__index=Channel})
+end
+
+function open(name)
+   local ret = {}
+   ret.ring_buffer = open_ring_buffer(name)
+   return setmetatable(ret, {__index=Channel})
+end
+
+-- The coordination needed between the reader and the writer is that:
+--
+--  1. If the reader sees a a bumped write pointer, that the data written
+--     to the ring buffer will be available to the reader, i.e. the writer
+--     has done whatever is needed to synchronize the data.
+--
+--  2. It should be possible for the reader to update the read pointer
+--     without stompling other memory, notably the write pointer.
+--
+--  3. It should be possible for the writer to update the write pointer
+--     without stompling other memory, notably the read pointer.
+--
+--  4. Updating a write pointer or a read pointer should eventually be
+--     visible to the reader or writer, respectively.
+--
+-- The full memory barrier after updates to the read or write pointer
+-- ensures (1).  The x86 memory model, and the memory model of C11,
+-- guarantee (2) and (3).  For (4), the memory barrier on the writer
+-- side ensures that updates to the read or write pointers are
+-- eventually visible to other CPUs, but we also have to insert a
+-- compiler barrier before reading them to prevent LuaJIT from caching
+-- their value somewhere else, like in a register.  See
+-- https://www.kernel.org/doc/Documentation/memory-barriers.txt for more
+-- discussion on memory models, and
+-- http://www.freelists.org/post/luajit/Compiler-loadstore-barrier-volatile-pointer-barriers-in-general,3
+-- for more on compiler barriers in LuaJIT.
+--
+-- If there are multiple readers or writers, they should serialize their
+-- accesses through some other mechanism.
+--
+
+-- Put some bytes onto the channel, but without updating the write
+-- pointer.  Precondition: the caller has checked that COUNT bytes are
+-- indeed available for writing.
+function Channel:put_bytes(bytes, count, offset)
+   offset = offset or 0
+   local ring = self.ring_buffer
+   local start = (ring.write + offset) % ring.size
+   if start + count > ring.size then
+      local head = ring.size - start
+      ffi.copy(ring.buf + start, bytes, head)
+      ffi.copy(ring.buf, bytes + head, count - head)
+   else
+      ffi.copy(ring.buf + start, bytes, count)
+   end
+end
+
+-- Peek some bytes into the channel.  If the COUNT bytes are contiguous,
+-- return a pointer into the channel.  Otherwise allocate a buffer for
+-- those bytes and return that.  Precondition: the caller has checked
+-- that COUNT bytes are indeed available for reading.
+function Channel:peek_bytes(count, offset)
+   offset = offset or 0
+   local ring = self.ring_buffer
+   local start = (ring.read + offset) % ring.size
+   local len
+   if start + count > ring.size then
+      local buf = ffi.new('uint8_t[?]', count)
+      local head_count = ring.size - start
+      local tail_count = count - head_count
+      ffi.copy(buf, ring.buf + start, head_count)
+      ffi.copy(buf + head_count, ring.buf, tail_count)
+      return buf
+   else
+      return ring.buf + start
+   end
+end
+
+function Channel:put_message(bytes, count)
+   local ring = self.ring_buffer
+   if write_avail(ring) < count + 4 then return false end
+   self:put_bytes(ffi.cast('uint8_t*', ffi.new('uint32_t[1]', count)), 4)
+   self:put_bytes(bytes, count, 4)
+   ring.write = ring.write + count + 4
+   ffi.C.full_memory_barrier()
+   return true;
+end
+
+function Channel:peek_payload_len()
+   local ring = self.ring_buffer
+   local avail = read_avail(ring)
+   local count = 4
+   if avail < count then return nil end
+   local len = ffi.cast('uint32_t*', self:peek_bytes(4))[0]
+   if avail < count + len then return nil end
+   return len
+end
+
+function Channel:peek_message()
+   local payload_len = self:peek_payload_len()
+   if not payload_len then return nil, nil end
+   return self:peek_bytes(payload_len, 4), payload_len
+end
+
+function Channel:discard_message(payload_len)
+   local ring = self.ring_buffer
+   ring.read = ring.read + payload_len + 4
+   ffi.C.full_memory_barrier()
+end
+
+function selftest()
+   print('selftest: lib.ptree.channel')
+   local msg_t = ffi.typeof('struct { uint8_t a; uint8_t b; }')
+   local ch = create('test/config-channel', (4+2)*16 + 1)
+   local function put(i)
+      return ch:put_message(ffi.new('uint8_t[2]', {i, i+16}), 2)
+   end
+   for _=1,4 do
+      for i=1,16 do assert(put(i)) end
+      assert(not put(17))
+      local function assert_pop(i)
+         local msg, len = ch:peek_message()
+         assert(msg)
+         assert(len == 2)
+         assert(msg[0] == i)
+         assert(msg[1] == i + 16)
+         ch:discard_message(len)
+      end
+      assert_pop(1)
+      assert(put(17))
+      for i=2,17 do assert_pop(i) end
+      assert(not ch:peek_message())
+   end
+   print('selftest: channel ok')
+end
diff --git a/src/lib/ptree/follower.lua b/src/lib/ptree/follower.lua
new file mode 100644
index 0000000000..497d1bed95
--- /dev/null
+++ b/src/lib/ptree/follower.lua
@@ -0,0 +1,96 @@
+-- Use of this source code is governed by the Apache 2.0 license; see COPYING.
+
+module(...,package.seeall)
+
+local S = require("syscall")
+local ffi = require("ffi")
+local yang = require("lib.yang.yang")
+local rpc = require("lib.yang.rpc")
+local app = require("core.app")
+local shm = require("core.shm")
+local app_graph = require("core.config")
+local channel = require("lib.ptree.channel")
+local action_codec = require("lib.ptree.action_codec")
+local alarm_codec = require("lib.ptree.alarm_codec")
+
+Follower = {
+   config = {
+      Hz = {default=1000},
+   }
+}
+
+function Follower:new (conf)
+   local ret = setmetatable({}, {__index=Follower})
+   ret.period = 1/conf.Hz
+   ret.next_time = app.now()
+   ret.channel = channel.create('config-follower-channel', 1e6)
+   ret.alarms_channel = alarm_codec.get_channel()
+   ret.pending_actions = {}
+   return ret
+end
+
+function Follower:shutdown()
+   -- This will shutdown everything.
+   engine.configure(app_graph.new())
+
+   -- Now we can exit.
+   S.exit(0)
+end
+
+function Follower:commit_pending_actions()
+   local to_apply = {}
+   local should_flush = false
+   for _,action in ipairs(self.pending_actions) do
+      local name, args = unpack(action)
+      if name == 'call_app_method_with_blob' then
+         if #to_apply > 0 then
+            app.apply_config_actions(to_apply)
+            to_apply = {}
+         end
+         local callee, method, blob = unpack(args)
+         local obj = assert(app.app_table[callee])
+         assert(obj[method])(obj, blob)
+      elseif name == "shutdown" then
+         self:shutdown()
+      else
+         if name == 'start_app' or name == 'reconfig_app' then
+            should_flush = true
+         end
+         table.insert(to_apply, action)
+      end
+   end
+   if #to_apply > 0 then app.apply_config_actions(to_apply) end
+   self.pending_actions = {}
+   if should_flush then require('jit').flush() end
+end
+
+function Follower:handle_actions_from_leader()
+   local channel = self.channel
+   for i=1,4 do
+      local buf, len = channel:peek_message()
+      if not buf then break end
+      local action = action_codec.decode(buf, len)
+      if action[1] == 'commit' then
+         self:commit_pending_actions()
+      else
+         table.insert(self.pending_actions, action)
+      end
+      channel:discard_message(len)
+   end
+end
+
+function Follower:pull ()
+   if app.now() < self.next_time then return end
+   self.next_time = app.now() + self.period
+   self:handle_actions_from_leader()
+end
+
+function selftest ()
+   print('selftest: lib.ptree.follower')
+   local c = config.new()
+   config.app(c, "follower", Follower, {})
+   engine.configure(c)
+   engine.main({ duration = 0.0001, report = {showapps=true,showlinks=true}})
+   engine.configure(config.new())
+   print('selftest: ok')
+end
diff --git a/src/lib/ptree/leader.lua b/src/lib/ptree/leader.lua
new file mode 100644
index 0000000000..63581b4307
--- /dev/null
+++ b/src/lib/ptree/leader.lua
@@ -0,0 +1,956 @@
+-- Use of this source code is governed by the Apache 2.0 license; see COPYING.
+
+module(...,package.seeall)
+
+local S = require("syscall")
+local ffi = require("ffi")
+local lib = require("core.lib")
+local cltable = require("lib.cltable")
+local cpuset = require("lib.cpuset")
+local yang = require("lib.yang.yang")
+local data = require("lib.yang.data")
+local util = require("lib.yang.util")
+local schema = require("lib.yang.schema")
+local rpc = require("lib.yang.rpc")
+local state = require("lib.yang.state")
+local path_mod = require("lib.yang.path")
+local app = require("core.app")
+local shm = require("core.shm")
+local worker = require("core.worker")
+local app_graph = require("core.config")
+local action_codec = require("lib.ptree.action_codec")
+local alarm_codec = require("lib.ptree.alarm_codec")
+local support = require("lib.ptree.support")
+local channel = require("lib.ptree.channel")
+local alarms = require("lib.yang.alarms")
+
+Leader = {
+   config = {
+      socket_file_name = {default='config-leader-socket'},
+      setup_fn = {required=true},
+      -- Could relax this requirement.
+      initial_configuration = {required=true},
+      schema_name = {required=true},
+      worker_start_code = {required=true},
+      default_schema = {},
+      cpuset = {default=cpuset.global_cpuset()},
+      Hz = {default=100},
+   }
+}
+
+local function open_socket (file)
+   S.signal('pipe', 'ign')
+   local socket = assert(S.socket("unix", "stream, nonblock"))
+   S.unlink(file) --unlink to avoid EINVAL on bind()
+   local sa = S.t.sockaddr_un(file)
+   assert(socket:bind(sa))
+   assert(socket:listen())
+   return socket
+end
+
+function Leader:new (conf)
+   local ret = setmetatable({}, {__index=Leader})
+   ret.cpuset = conf.cpuset
+   ret.socket_file_name = conf.socket_file_name
+   if not ret.socket_file_name:match('^/') then
+      local instance_dir = shm.root..'/'..tostring(S.getpid())
+      ret.socket_file_name = instance_dir..'/'..ret.socket_file_name
+   end
+   ret.schema_name = conf.schema_name
+   ret.default_schema = conf.default_schema or conf.schema_name
+   ret.support = support.load_schema_config_support(conf.schema_name)
+   ret.socket = open_socket(ret.socket_file_name)
+   ret.peers = {}
+   ret.setup_fn = conf.setup_fn
+   ret.period = 1/conf.Hz
+   ret.next_time = app.now()
+   ret.worker_start_code = conf.worker_start_code
+   ret.followers = {}
+   ret.rpc_callee = rpc.prepare_callee('snabb-config-leader-v1')
+   ret.rpc_handler = rpc.dispatch_handler(ret, 'rpc_')
+
+   ret:set_initial_configuration(conf.initial_configuration)
+
+   return ret
+end
+
+function Leader:set_initial_configuration (configuration)
+   self.current_configuration = configuration
+   self.current_in_place_dependencies = {}
+
+   -- Start the followers and configure them.
+   local worker_app_graphs = self.setup_fn(configuration)
+
+   -- Calculate the dependences
+   self.current_in_place_dependencies =
+      self.support.update_mutable_objects_embedded_in_app_initargs (
+	    {}, worker_app_graphs, self.schema_name, 'load',
+            '/', self.current_configuration)
+
+   -- Iterate over followers starting the workers and queuing up actions.
+   for id, worker_app_graph in pairs(worker_app_graphs) do
+      self:start_follower_for_graph(id, worker_app_graph)
+   end
+end
+
+function Leader:start_worker(cpu)
+   local start_code = { self.worker_start_code }
+   if cpu then
+      table.insert(start_code, 1, "print('Bound data plane to CPU:',"..cpu..")")
+      table.insert(start_code, 1, "require('lib.numa').bind_to_cpu("..cpu..")")
+   end
+   return worker.start("follower", table.concat(start_code, "\n"))
+end
+
+function Leader:stop_worker(id)
+   -- Tell the worker to terminate
+   local stop_actions = {{'shutdown', {}}, {'commit', {}}}
+   self:enqueue_config_actions_for_follower(id, stop_actions)
+   self:send_messages_to_followers()
+   self.followers[id].shutting_down = true
+end
+
+function Leader:remove_stale_followers()
+   local stale = {}
+   for id, follower in pairs(self.followers) do
+      if follower.shutting_down then
+	 if S.waitpid(follower.pid, S.c.W["NOHANG"]) ~= 0 then
+	    stale[#stale + 1] = id
+	 end
+      end
+   end
+   for _, id in ipairs(stale) do
+      if self.followers[id].cpu then
+	 self.cpuset:release(self.followers[id].cpu)
+      end
+      self.followers[id] = nil
+
+   end
+end
+
+function Leader:acquire_cpu_for_follower(id, app_graph)
+   local pci_addresses = {}
+   -- Grovel through app initargs for keys named "pciaddr".  Hacky!
+   for name, init in pairs(app_graph.apps) do
+      if type(init.arg) == 'table' then
+         for k, v in pairs(init.arg) do
+            if k == 'pciaddr' then table.insert(pci_addresses, v) end
+         end
+      end
+   end
+   return self.cpuset:acquire_for_pci_addresses(pci_addresses)
+end
+
+function Leader:start_follower_for_graph(id, graph)
+   local cpu = self:acquire_cpu_for_follower(id, graph)
+   self.followers[id] = { cpu=cpu, pid=self:start_worker(cpu), queue={},
+                          graph=graph }
+   local actions = self.support.compute_config_actions(
+      app_graph.new(), self.followers[id].graph, {}, 'load')
+   self:enqueue_config_actions_for_follower(id, actions)
+   return self.followers[id]
+end
+
+function Leader:take_follower_message_queue ()
+   local actions = self.config_action_queue
+   self.config_action_queue = nil
+   return actions
+end
+
+local verbose = os.getenv('SNABB_LEADER_VERBOSE') and true
+
+function Leader:enqueue_config_actions_for_follower(follower, actions)
+   for _,action in ipairs(actions) do
+      if verbose then print('encode', action[1], unpack(action[2])) end
+      local buf, len = action_codec.encode(action)
+      table.insert(self.followers[follower].queue, { buf=buf, len=len })
+   end
+end
+
+function Leader:enqueue_config_actions (actions)
+   for id,_ in pairs(self.followers) do
+      self.enqueue_config_actions_for_follower(id, actions)
+   end
+end
+
+function Leader:rpc_describe (args)
+   local alternate_schemas = {}
+   for schema_name, translator in pairs(self.support.translators) do
+      table.insert(alternate_schemas, schema_name)
+   end
+   return { native_schema = self.schema_name,
+	    default_schema = self.default_schema,
+            alternate_schema = alternate_schemas,
+            capability = schema.get_default_capabilities() }
+end
+
+local function path_printer_for_grammar(grammar, path, format, print_default)
+   local getter, subgrammar = path_mod.resolver(grammar, path)
+   local printer
+   if format == "xpath" then
+      printer = data.xpath_printer_from_grammar(subgrammar, print_default, path)
+   else
+      printer = data.data_printer_from_grammar(subgrammar, print_default)
+   end
+   return function(data, file)
+      return printer(getter(data), file)
+   end
+end
+
+local function path_printer_for_schema(schema, path, is_config,
+                                       format, print_default)
+   local grammar = data.data_grammar_from_schema(schema, is_config)
+   return path_printer_for_grammar(grammar, path, format, print_default)
+end
+
+local function path_printer_for_schema_by_name(schema_name, path, is_config,
+                                               format, print_default)
+   local schema = yang.load_schema_by_name(schema_name)
+   return path_printer_for_schema(schema, path, is_config, format,
+                                  print_default)
+end
+
+function Leader:rpc_get_config (args)
+   local function getter()
+      if args.schema ~= self.schema_name then
+         return self:foreign_rpc_get_config(
+            args.schema, args.path, args.format, args.print_default)
+      end
+      local printer = path_printer_for_schema_by_name(
+         args.schema, args.path, true, args.format, args.print_default)
+      local config = printer(self.current_configuration, yang.string_output_file())
+      return { config = config }
+   end
+   local success, response = pcall(getter)
+   if success then return response else return {status=1, error=response} end
+end
+
+function Leader:rpc_set_alarm_operator_state (args)
+   local function getter()
+      if args.schema ~= self.schema_name then
+         return false, ("Set-operator-state operation not supported in"..
+                        "'%s' schema"):format(args.schema)
+      end
+      local key = {resource=args.resource, alarm_type_id=args.alarm_type_id,
+                   alarm_type_qualifier=args.alarm_type_qualifier}
+      local params = {state=args.state, text=args.text}
+      return { success = alarms.set_operator_state(key, params) }
+   end
+   local success, response = pcall(getter)
+   if success then return response else return {status=1, error=response} end
+end
+
+function Leader:rpc_purge_alarms (args)
+   local function purge()
+      if args.schema ~= self.schema_name then
+         return false, ("Purge-alarms operation not supported in"..
+                        "'%s' schema"):format(args.schema)
+      end
+      return { purged_alarms = alarms.purge_alarms(args) }
+   end
+   local success, response = pcall(purge)
+   if success then return response else return {status=1, error=response} end
+end
+
+function Leader:rpc_compress_alarms (args)
+   local function compress()
+      if args.schema ~= self.schema_name then
+         return false, ("Compress-alarms operation not supported in"..
+                        "'%s' schema"):format(args.schema)
+      end
+      return { compressed_alarms = alarms.compress_alarms(args) }
+   end
+   local success, response = pcall(compress)
+   if success then return response else return {status=1, error=response} end
+end
+
+
+local function path_parser_for_grammar(grammar, path)
+   local getter, subgrammar = path_mod.resolver(grammar, path)
+   return data.data_parser_from_grammar(subgrammar)
+end
+
+local function path_parser_for_schema(schema, path)
+   local grammar = data.config_grammar_from_schema(schema)
+   return path_parser_for_grammar(grammar, path)
+end
+
+local function path_parser_for_schema_by_name(schema_name, path)
+   return path_parser_for_schema(yang.load_schema_by_name(schema_name), path)
+end
+
+local function path_setter_for_grammar(grammar, path)
+   if path == "/" then
+      return function(config, subconfig) return subconfig end
+   end
+   local head, tail = lib.dirname(path), lib.basename(path)
+   local tail_path = path_mod.parse_path(tail)
+   local tail_name, query = tail_path[1].name, tail_path[1].query
+   if lib.equal(query, {}) then
+      -- No query; the simple case.
+      local getter, grammar = path_mod.resolver(grammar, head)
+      assert(grammar.type == 'struct')
+      local tail_id = data.normalize_id(tail_name)
+      return function(config, subconfig)
+         getter(config)[tail_id] = subconfig
+         return config
+      end
+   end
+
+   -- Otherwise the path ends in a query; it must denote an array or
+   -- table item.
+   local getter, grammar = path_mod.resolver(grammar, head..'/'..tail_name)
+   if grammar.type == 'array' then
+      local idx = path_mod.prepare_array_lookup(query)
+      return function(config, subconfig)
+         local array = getter(config)
+         assert(idx <= #array)
+         array[idx] = subconfig
+         return config
+      end
+   elseif grammar.type == 'table' then
+      local key = path_mod.prepare_table_lookup(grammar.keys,
+                                                grammar.key_ctype, query)
+      if grammar.string_key then
+         key = key[data.normalize_id(grammar.string_key)]
+         return function(config, subconfig)
+            local tab = getter(config)
+            assert(tab[key] ~= nil)
+            tab[key] = subconfig
+            return config
+         end
+      elseif grammar.key_ctype and grammar.value_ctype then
+         return function(config, subconfig)
+            getter(config):update(key, subconfig)
+            return config
+         end
+      elseif grammar.key_ctype then
+         return function(config, subconfig)
+            local tab = getter(config)
+            assert(tab[key] ~= nil)
+            tab[key] = subconfig
+            return config
+         end
+      else
+         return function(config, subconfig)
+            local tab = getter(config)
+            for k,v in pairs(tab) do
+               if lib.equal(k, key) then
+                  tab[k] = subconfig
+                  return config
+               end
+            end
+            error("Not found")
+         end
+      end
+   else
+      error('Query parameters only allowed on arrays and tables')
+   end
+end
+
+local function path_setter_for_schema(schema, path)
+   local grammar = data.config_grammar_from_schema(schema)
+   return path_setter_for_grammar(grammar, path)
+end
+
+function compute_set_config_fn (schema_name, path)
+   return path_setter_for_schema(yang.load_schema_by_name(schema_name), path)
+end
+
+local function path_adder_for_grammar(grammar, path)
+   local top_grammar = grammar
+   local getter, grammar = path_mod.resolver(grammar, path)
+   if grammar.type == 'array' then
+      if grammar.ctype then
+         -- It's an FFI array; have to create a fresh one, sadly.
+         local setter = path_setter_for_grammar(top_grammar, path)
+         local elt_t = data.typeof(grammar.ctype)
+         local array_t = ffi.typeof('$[?]', elt_t)
+         return function(config, subconfig)
+            local cur = getter(config)
+            local new = array_t(#cur + #subconfig)
+            local i = 1
+            for _,elt in ipairs(cur) do new[i-1] = elt; i = i + 1 end
+            for _,elt in ipairs(subconfig) do new[i-1] = elt; i = i + 1 end
+            return setter(config, util.ffi_array(new, elt_t))
+         end
+      end
+      -- Otherwise we can add entries in place.
+      return function(config, subconfig)
+         local cur = getter(config)
+         for _,elt in ipairs(subconfig) do table.insert(cur, elt) end
+         return config
+      end
+   elseif grammar.type == 'table' then
+      -- Invariant: either all entries in the new subconfig are added,
+      -- or none are.
+      if grammar.key_ctype and grammar.value_ctype then
+         -- ctable.
+         return function(config, subconfig)
+            local ctab = getter(config)
+            for entry in subconfig:iterate() do
+               if ctab:lookup_ptr(entry.key) ~= nil then
+                  error('already-existing entry')
+               end
+            end
+            for entry in subconfig:iterate() do
+               ctab:add(entry.key, entry.value)
+            end
+            return config
+         end
+      elseif grammar.string_key or grammar.key_ctype then
+         -- cltable or string-keyed table.
+         local pairs = grammar.key_ctype and cltable.pairs or pairs
+         return function(config, subconfig)
+            local tab = getter(config)
+            for k,_ in pairs(subconfig) do
+               if tab[k] ~= nil then error('already-existing entry') end
+            end
+            for k,v in pairs(subconfig) do tab[k] = v end
+            return config
+         end
+      else
+         -- Sad quadratic loop.
+         return function(config, subconfig)
+            local tab = getter(config)
+            for key,val in pairs(tab) do
+               for k,_ in pairs(subconfig) do
+                  if lib.equal(key, k) then
+                     error('already-existing entry', key)
+                  end
+               end
+            end
+            for k,v in pairs(subconfig) do tab[k] = v end
+            return config
+         end
+      end
+   else
+      error('Add only allowed on arrays and tables')
+   end
+end
+
+local function path_adder_for_schema(schema, path)
+   local grammar = data.config_grammar_from_schema(schema)
+   return path_adder_for_grammar(grammar, path)
+end
+
+function compute_add_config_fn (schema_name, path)
+   return path_adder_for_schema(yang.load_schema_by_name(schema_name), path)
+end
+compute_add_config_fn = util.memoize(compute_add_config_fn)
+
+local function path_remover_for_grammar(grammar, path)
+   local top_grammar = grammar
+   local head, tail = lib.dirname(path), lib.basename(path)
+   local tail_path = path_mod.parse_path(tail)
+   local tail_name, query = tail_path[1].name, tail_path[1].query
+   local head_and_tail_name = head..'/'..tail_name
+   local getter, grammar = path_mod.resolver(grammar, head_and_tail_name)
+   if grammar.type == 'array' then
+      if grammar.ctype then
+         -- It's an FFI array; have to create a fresh one, sadly.
+         local idx = path_mod.prepare_array_lookup(query)
+         local setter = path_setter_for_grammar(top_grammar, head_and_tail_name)
+         local elt_t = data.typeof(grammar.ctype)
+         local array_t = ffi.typeof('$[?]', elt_t)
+         return function(config)
+            local cur = getter(config)
+            assert(idx <= #cur)
+            local new = array_t(#cur - 1)
+            for i,elt in ipairs(cur) do
+               if i < idx then new[i-1] = elt end
+               if i > idx then new[i-2] = elt end
+            end
+            return setter(config, util.ffi_array(new, elt_t))
+         end
+      end
+      -- Otherwise we can remove the entry in place.
+      return function(config)
+         local cur = getter(config)
+         assert(i <= #cur)
+         table.remove(cur, i)
+         return config
+      end
+   elseif grammar.type == 'table' then
+      local key = path_mod.prepare_table_lookup(grammar.keys,
+                                                grammar.key_ctype, query)
+      if grammar.string_key then
+         key = key[data.normalize_id(grammar.string_key)]
+         return function(config)
+            local tab = getter(config)
+            assert(tab[key] ~= nil)
+            tab[key] = nil
+            return config
+         end
+      elseif grammar.key_ctype and grammar.value_ctype then
+         return function(config)
+            getter(config):remove(key)
+            return config
+         end
+      elseif grammar.key_ctype then
+         return function(config)
+            local tab = getter(config)
+            assert(tab[key] ~= nil)
+            tab[key] = nil
+            return config
+         end
+      else
+         return function(config)
+            local tab = getter(config)
+            for k,v in pairs(tab) do
+               if lib.equal(k, key) then
+                  tab[k] = nil
+                  return config
+               end
+            end
+            error("Not found")
+         end
+      end
+   else
+      error('Remove only allowed on arrays and tables')
+   end
+end
+
+local function path_remover_for_schema(schema, path)
+   local grammar = data.config_grammar_from_schema(schema)
+   return path_remover_for_grammar(grammar, path)
+end
+
+function compute_remove_config_fn (schema_name, path)
+   return path_remover_for_schema(yang.load_schema_by_name(schema_name), path)
+end
+
+function Leader:notify_pre_update (config, verb, path, ...)
+   for _,translator in pairs(self.support.translators) do
+      translator.pre_update(config, verb, path, ...)
+   end
+end
+
+function Leader:update_configuration (update_fn, verb, path, ...)
+   self:notify_pre_update(self.current_configuration, verb, path, ...)
+   local to_restart =
+      self.support.compute_apps_to_restart_after_configuration_update (
+         self.schema_name, self.current_configuration, verb, path,
+         self.current_in_place_dependencies, ...)
+   local new_config = update_fn(self.current_configuration, ...)
+   local new_graphs = self.setup_fn(new_config, ...)
+   for id, graph in pairs(new_graphs) do
+      if self.followers[id] == nil then
+	 self:start_follower_for_graph(id, graph)
+      end
+   end
+
+   for id, follower in pairs(self.followers) do
+      if new_graphs[id] == nil then
+         self:stop_worker(id)
+      else
+	 local actions = self.support.compute_config_actions(
+	    follower.graph, new_graphs[id], to_restart, verb, path, ...)
+	 self:enqueue_config_actions_for_follower(id, actions)
+	 follower.graph = new_graphs[id]
+      end
+   end
+   self.current_configuration = new_config
+   self.current_in_place_dependencies =
+      self.support.update_mutable_objects_embedded_in_app_initargs (
+         self.current_in_place_dependencies, new_graphs, verb, path, ...)
+end
+
+function Leader:handle_rpc_update_config (args, verb, compute_update_fn)
+   local path = path_mod.normalize_path(args.path)
+   local parser = path_parser_for_schema_by_name(args.schema, path)
+   self:update_configuration(compute_update_fn(args.schema, path),
+                             verb, path, parser(args.config))
+   return {}
+end
+
+function Leader:get_native_state ()
+   local states = {}
+   local state_reader = self.support.compute_state_reader(self.schema_name)
+   for _, follower in pairs(self.followers) do
+      local follower_config = self.support.configuration_for_follower(
+         follower, self.current_configuration)
+      table.insert(states, state_reader(follower.pid, follower_config))
+   end
+   return self.support.process_states(states)
+end
+
+function Leader:get_translator (schema_name)
+   local translator = self.support.translators[schema_name]
+   if translator then return translator end
+   error('unsupported schema: '..schema_name)
+end
+function Leader:apply_translated_rpc_updates (updates)
+   for _,update in ipairs(updates) do
+      local verb, args = unpack(update)
+      local method = assert(self['rpc_'..verb..'_config'])
+      method(self, args)
+   end
+   return {}
+end
+function Leader:foreign_rpc_get_config (schema_name, path, format,
+                                        print_default)
+   path = path_mod.normalize_path(path)
+   local translate = self:get_translator(schema_name)
+   local foreign_config = translate.get_config(self.current_configuration)
+   local printer = path_printer_for_schema_by_name(
+      schema_name, path, true, format, print_default)
+   local config = printer(foreign_config, yang.string_output_file())
+   return { config = config }
+end
+function Leader:foreign_rpc_get_state (schema_name, path, format,
+                                       print_default)
+   path = path_mod.normalize_path(path)
+   local translate = self:get_translator(schema_name)
+   local foreign_state = translate.get_state(self:get_native_state())
+   local printer = path_printer_for_schema_by_name(
+      schema_name, path, false, format, print_default)
+   local state = printer(foreign_state, yang.string_output_file())
+   return { state = state }
+end
+function Leader:foreign_rpc_set_config (schema_name, path, config_str)
+   path = path_mod.normalize_path(path)
+   local translate = self:get_translator(schema_name)
+   local parser = path_parser_for_schema_by_name(schema_name, path)
+   local updates = translate.set_config(self.current_configuration, path,
+                                        parser(config_str))
+   return self:apply_translated_rpc_updates(updates)
+end
+function Leader:foreign_rpc_add_config (schema_name, path, config_str)
+   path = path_mod.normalize_path(path)
+   local translate = self:get_translator(schema_name)
+   local parser = path_parser_for_schema_by_name(schema_name, path)
+   local updates = translate.add_config(self.current_configuration, path,
+                                        parser(config_str))
+   return self:apply_translated_rpc_updates(updates)
+end
+function Leader:foreign_rpc_remove_config (schema_name, path)
+   path = path_mod.normalize_path(path)
+   local translate = self:get_translator(schema_name)
+   local updates = translate.remove_config(self.current_configuration, path)
+   return self:apply_translated_rpc_updates(updates)
+end
+
+function Leader:rpc_set_config (args)
+   local function setter()
+      if self.listen_peer ~= nil and self.listen_peer ~= self.rpc_peer then
+         error('Attempt to modify configuration while listener attached')
+      end
+      if args.schema ~= self.schema_name then
+         return self:foreign_rpc_set_config(args.schema, args.path, args.config)
+      end
+      return self:handle_rpc_update_config(args, 'set', compute_set_config_fn)
+   end
+   local success, response = pcall(setter)
+   if success then return response else return {status=1, error=response} end
+end
+
+function Leader:rpc_add_config (args)
+   local function adder()
+      if self.listen_peer ~= nil and self.listen_peer ~= self.rpc_peer then
+         error('Attempt to modify configuration while listener attached')
+      end
+      if args.schema ~= self.schema_name then
+         return self:foreign_rpc_add_config(args.schema, args.path, args.config)
+      end
+      return self:handle_rpc_update_config(args, 'add', compute_add_config_fn)
+   end
+   local success, response = pcall(adder)
+   if success then return response else return {status=1, error=response} end
+end
+
+function Leader:rpc_remove_config (args)
+   local function remover()
+      if self.listen_peer ~= nil and self.listen_peer ~= self.rpc_peer then
+         error('Attempt to modify configuration while listener attached')
+      end
+      if args.schema ~= self.schema_name then
+         return self:foreign_rpc_remove_config(args.schema, args.path)
+      end
+      local path = path_mod.normalize_path(args.path)
+      self:update_configuration(compute_remove_config_fn(args.schema, path),
+                              'remove', path)
+      return {}
+   end
+   local success, response = pcall(remover)
+   if success then return response else return {status=1, error=response} end
+end
+
+function Leader:rpc_attach_listener (args)
+   local function attacher()
+      if self.listen_peer ~= nil then error('Listener already attached') end
+      self.listen_peer = self.rpc_peer
+      return {}
+   end
+   local success, response = pcall(attacher)
+   if success then return response else return {status=1, error=response} end
+end
+
+function Leader:rpc_get_state (args)
+   local function getter()
+      if args.schema ~= self.schema_name then
+         return self:foreign_rpc_get_state(args.schema, args.path,
+                                           args.format, args.print_default)
+      end
+      local state = self:get_native_state()
+      local printer = path_printer_for_schema_by_name(
+         self.schema_name, args.path, false, args.format, args.print_default)
+      return { state = printer(state, yang.string_output_file()) }
+   end
+   local success, response = pcall(getter)
+   if success then return response else return {status=1, error=response} end
+end
+
+function Leader:rpc_get_alarms_state (args)
+   local function getter()
+      assert(args.schema == "ietf-alarms")
+      local printer = path_printer_for_schema_by_name(
+         args.schema, args.path, false, args.format, args.print_default)
+      local state = {
+         alarms = alarms.get_state()
+      }
+      state = printer(state, yang.string_output_file())
+      return { state = state }
+   end
+   local success, response = pcall(getter)
+   if success then return response else return {status=1, error=response} end
+end
+
+function Leader:handle (payload)
+   return rpc.handle_calls(self.rpc_callee, payload, self.rpc_handler)
+end
+
+local dummy_unix_sockaddr = S.t.sockaddr_un()
+
+function Leader:handle_calls_from_peers()
+   local peers = self.peers
+   while true do
+      local fd, err = self.socket:accept(dummy_unix_sockaddr)
+      if not fd then
+         if err.AGAIN then break end
+         assert(nil, err)
+      end
+      fd:nonblock()
+      table.insert(peers, { state='length', len=0, fd=fd })
+   end
+   local i = 1
+   while i <= #peers do
+      local peer = peers[i]
+      local visit_peer_again = false
+      while peer.state == 'length' do
+         local ch, err = peer.fd:read(nil, 1)
+         if not ch then
+            if err.AGAIN then break end
+            peer.state = 'error'
+            peer.msg = tostring(err)
+         elseif ch == '\n' then
+            peer.pos = 0
+            peer.buf = ffi.new('uint8_t[?]', peer.len)
+            peer.state = 'payload'
+         elseif tonumber(ch) then
+            peer.len = peer.len * 10 + tonumber(ch)
+            if peer.len > 1e8 then
+               peer.state = 'error'
+               peer.msg = 'length too long: '..peer.len
+            end
+         elseif ch == '' then
+            if peer.len == 0 then
+               peer.state = 'done'
+            else
+               peer.state = 'error'
+               peer.msg = 'unexpected EOF'
+            end
+         else
+            peer.state = 'error'
+            peer.msg = 'unexpected character: '..ch
+         end
+      end
+      while peer.state == 'payload' do
+         if peer.pos == peer.len then
+            peer.state = 'ready'
+            peer.payload = ffi.string(peer.buf, peer.len)
+            peer.buf, peer.len = nil, nil
+         else
+            local count, err = peer.fd:read(peer.buf + peer.pos,
+                                            peer.len - peer.pos)
+            if not count then
+               if err.AGAIN then break end
+               peer.state = 'error'
+               peer.msg = tostring(err)
+            elseif count == 0 then
+               peer.state = 'error'
+               peer.msg = 'short read'
+            else
+               peer.pos = peer.pos + count
+               assert(peer.pos <= peer.len)
+            end
+         end
+      end
+      while peer.state == 'ready' do
+         -- Uncomment to get backtraces.
+         self.rpc_peer = peer
+         -- local success, reply = true, self:handle(peer.payload)
+         local success, reply = pcall(self.handle, self, peer.payload)
+         self.rpc_peer = nil
+         peer.payload = nil
+         if success then
+            assert(type(reply) == 'string')
+            reply = #reply..'\n'..reply
+            peer.state = 'reply'
+            peer.buf = ffi.new('uint8_t[?]', #reply+1, reply)
+            peer.pos = 0
+            peer.len = #reply
+         else
+            peer.state = 'error'
+            peer.msg = reply
+         end
+      end
+      while peer.state == 'reply' do
+         if peer.pos == peer.len then
+            visit_peer_again = true
+            peer.state = 'length'
+            peer.buf, peer.pos = nil, nil
+            peer.len = 0
+         else
+            local count, err = peer.fd:write(peer.buf + peer.pos,
+                                             peer.len - peer.pos)
+            if not count then
+               if err.AGAIN then break end
+               peer.state = 'error'
+               peer.msg = tostring(err)
+            elseif count == 0 then
+               peer.state = 'error'
+               peer.msg = 'short write'
+            else
+               peer.pos = peer.pos + count
+               assert(peer.pos <= peer.len)
+            end
+         end
+      end
+      if peer.state == 'done' or peer.state == 'error' then
+         if peer.state == 'error' then print('error: '..peer.msg) end
+         peer.fd:close()
+         table.remove(peers, i)
+         if self.listen_peer == peer then self.listen_peer = nil end
+      elseif not visit_peer_again then
+         i = i + 1
+      end
+   end
+end
+
+function Leader:send_messages_to_followers()
+   for _,follower in pairs(self.followers) do
+      if not follower.channel then
+         local name = '/'..tostring(follower.pid)..'/config-follower-channel'
+         local success, channel = pcall(channel.open, name)
+         if success then follower.channel = channel end
+      end
+      local channel = follower.channel
+      if channel then
+         local queue = follower.queue
+         follower.queue = {}
+         local requeue = false
+         for _,msg in ipairs(queue) do
+            if not requeue then
+               requeue = not channel:put_message(msg.buf, msg.len)
+            end
+            if requeue then table.insert(follower.queue, msg) end
+         end
+      end
+   end
+end
+
+function Leader:pull ()
+   if app.now() < self.next_time then return end
+   self.next_time = app.now() + self.period
+   self:remove_stale_followers()
+   self:handle_calls_from_peers()
+   self:send_messages_to_followers()
+   self:receive_alarms_from_followers()
+end
+
+function Leader:receive_alarms_from_followers ()
+   for _,follower in pairs(self.followers) do
+      self:receive_alarms_from_follower(follower)
+   end
+end
+
+function Leader:receive_alarms_from_follower (follower)
+   if not follower.alarms_channel then
+      local name = '/'..tostring(follower.pid)..'/alarms-follower-channel'
+      local success, channel = pcall(channel.open, name)
+      if not success then return end
+      follower.alarms_channel = channel
+   end
+   local channel = follower.alarms_channel
+   while true do
+      local buf, len = channel:peek_message()
+      if not buf then break end
+      local alarm = alarm_codec.decode(buf, len)
+      self:handle_alarm(follower, alarm)
+      channel:discard_message(len)
+   end
+end
+
+function Leader:handle_alarm (follower, alarm)
+   local fn, args = unpack(alarm)
+   if fn == 'raise_alarm' then
+      local key, args = alarm_codec.to_alarm(args)
+      alarms.raise_alarm(key, args)
+   end
+   if fn == 'clear_alarm' then
+      local key = alarm_codec.to_alarm(args)
+      alarms.clear_alarm(key)
+   end
+   if fn == 'add_to_inventory' then
+      local key, args = alarm_codec.to_alarm_type(args)
+      alarms.do_add_to_inventory(key, args)
+   end
+   if fn == 'declare_alarm' then
+      local key, args = alarm_codec.to_alarm(args)
+      alarms.do_declare_alarm(key, args)
+   end
+end
+
+function Leader:stop ()
+   for _,peer in ipairs(self.peers) do peer.fd:close() end
+   self.peers = {}
+   self.socket:close()
+   S.unlink(self.socket_file_name)
+end
+
+function test_worker()
+   local follower = require("lib.ptree.follower")
+   local myconf = config.new()
+   config.app(myconf, "follower", follower.Follower, {})
+   app.configure(myconf)
+   app.busywait = true
+   app.main({})
+end
+
+function selftest ()
+   print('selftest: lib.ptree.leader')
+   local graph = app_graph.new()
+   local function setup_fn(cfg)
+      local graph = app_graph.new()
+      local basic_apps = require('apps.basic.basic_apps')
+      app_graph.app(graph, "source", basic_apps.Source, {})
+      app_graph.app(graph, "sink", basic_apps.Sink, {})
+      app_graph.link(graph, "source.foo -> sink.bar")
+      return {graph}
+   end
+   local worker_start_code = "require('lib.ptree.leader').test_worker()"
+   app_graph.app(graph, "leader", Leader,
+                 {setup_fn=setup_fn, worker_start_code=worker_start_code,
+                  -- Use a schema with no data nodes, just for
+                  -- testing.
+                  schema_name='ietf-inet-types', initial_configuration={}})
+   engine.configure(graph)
+   engine.main({ duration = 0.05, report = {showapps=true,showlinks=true}})
+   assert(app.app_table.leader.followers[1])
+   assert(app.app_table.leader.followers[1].graph.links)
+   assert(app.app_table.leader.followers[1].graph.links["source.foo -> sink.bar"])
+   local link = app.link_table["source.foo -> sink.bar"]
+   engine.configure(app_graph.new())
+   print('selftest: ok')
+end
diff --git a/src/lib/ptree/support.lua b/src/lib/ptree/support.lua
new file mode 100644
index 0000000000..d11a6dae21
--- /dev/null
+++ b/src/lib/ptree/support.lua
@@ -0,0 +1,231 @@
+-- Use of this source code is governed by the Apache 2.0 license; see COPYING.
+
+module(...,package.seeall)
+
+local app = require("core.app")
+local app_graph_mod = require("core.config")
+local path_mod = require("lib.yang.path")
+local yang = require("lib.yang.yang")
+local data = require("lib.yang.data")
+local cltable = require("lib.cltable")
+
+function compute_parent_paths(path)
+   local function sorted_keys(t)
+      local ret = {}
+      for k, v in pairs(t) do table.insert(ret, k) end
+      table.sort(ret)
+      return ret
+   end
+   local ret = { '/' }
+   local head = ''
+   for _,part in ipairs(path_mod.parse_path(path)) do
+      head = head..'/'..part.name
+      table.insert(ret, head)
+      local keys = sorted_keys(part.query)
+      if #keys > 0 then
+         for _,k in ipairs(keys) do
+            head = head..'['..k..'='..part.query[k]..']'
+         end
+         table.insert(ret, head)
+      end
+   end
+   return ret
+end
+
+local function add_child_objects(accum, grammar, config)
+   local visitor = {}
+   local function visit(grammar, config)
+      assert(visitor[grammar.type])(grammar, config)
+   end
+   local function visit_child(grammar, config)
+      if grammar.type == 'scalar' then return end
+      table.insert(accum, config)
+      return visit(grammar, config)
+   end
+   function visitor.table(grammar, config)
+      -- Ctables are raw data, and raw data doesn't contain children
+      -- with distinct identity.
+      if grammar.key_ctype and grammar.value_ctype then return end
+      local child_grammar = {type="struct", members=grammar.values,
+                             ctype=grammar.value_ctype}
+      if grammar.key_ctype then
+         for k, v in cltable.pairs(config) do visit_child(child_grammar, v) end
+      else
+         for k, v in pairs(config) do visit_child(child_grammar, v) end
+      end
+   end
+   function visitor.array(grammar, config)
+      -- Children are leaves; nothing to do.
+   end
+   function visitor.struct(grammar, config)
+      -- Raw data doesn't contain children with distinct identity.
+      if grammar.ctype then return end
+      for k,grammar in pairs(grammar.members) do
+         local id = data.normalize_id(k)
+         local child = config[id]
+         if child ~= nil then visit_child(grammar, child) end
+      end
+   end
+   return visit(grammar, config)
+end
+
+local function compute_objects_maybe_updated_in_place (schema_name, config,
+                                                       changed_path)
+   local schema = yang.load_schema_by_name(schema_name)
+   local grammar = data.config_grammar_from_schema(schema)
+   local objs = {}
+   local getter, subgrammar
+   for _,path in ipairs(compute_parent_paths(changed_path)) do
+      -- Calling the getter is avg O(N) in depth, so that makes the
+      -- loop O(N^2), though it is generally bounded at a shallow
+      -- level so perhaps it's OK.  path_mod.resolver is O(N) too but
+      -- memoization makes it O(1).
+      getter, subgrammar = path_mod.resolver(grammar, path)
+      -- Scalars can't be updated in place.
+      if subgrammar.type == 'scalar' then return objs end
+      table.insert(objs, getter(config))
+      -- Members of raw data can't be updated in place either.
+      if subgrammar.type == 'table' then
+         if subgrammar.key_ctype and subgrammar.value_ctype then return objs end
+      elseif subgrammar.type == 'struct' then
+         if subgrammar.ctype then return objs end
+      end
+   end
+   -- If the loop above finished normally, then it means that the
+   -- object at changed_path might contain in-place-updatable objects
+   -- inside of it, so visit its children.
+   add_child_objects(objs, subgrammar, objs[#objs])
+   return objs
+end
+
+local function record_mutable_objects_embedded_in_app_initarg (follower_id, app_name, obj, accum)
+   local function record(obj)
+      local tab = accum[obj]
+      if not tab then
+         tab = {}
+         accum[obj] = tab
+      end
+      if tab[follower_id] == nil then
+         tab[follower_id] = {app_name}
+      else
+         table.insert(tab[follower_id], app_name)
+      end
+   end
+   local function visit(obj)
+      if type(obj) == 'table' then
+         record(obj)
+         for _,v in pairs(obj) do visit(v) end
+      elseif type(obj) == 'cdata' then
+         record(obj)
+         -- Cdata contains sub-objects but they don't have identity;
+         -- it's only the cdata object itself that has identity.
+      else
+         -- Other object kinds can't be updated in place.
+      end
+   end
+   visit(obj)
+end
+
+-- Takes a table of follower ids (app_graph_map) and returns a tabl≈e which has
+-- the follower id as the key and a table listing all app names
+--   i.e. {follower_id => {app name, ...}, ...}
+local function compute_mutable_objects_embedded_in_app_initargs (app_graph_map)
+   local deps = {}
+   for id, app_graph in pairs(app_graph_map) do
+      for name, info in pairs(app_graph.apps) do
+         record_mutable_objects_embedded_in_app_initarg(id, name, info.arg, deps)
+      end
+   end
+   return deps
+end
+
+local function compute_apps_to_restart_after_configuration_update (
+      schema_name, configuration, verb, changed_path, in_place_dependencies, arg)
+   local maybe_updated = compute_objects_maybe_updated_in_place(
+      schema_name, configuration, changed_path)
+   local needs_restart = {}
+   for _,place in ipairs(maybe_updated) do
+      for _, id in ipairs(in_place_dependencies[place] or {}) do
+         if needs_restart[id] == nil then needs_restart[id] = {} end
+         for _, appname in ipairs(in_place_dependencies[place][id] or {}) do
+            needs_restart[id][appname] = true
+         end
+      end
+   end
+   return needs_restart
+end
+
+local function add_restarts(actions, app_graph, to_restart)
+   for _,action in ipairs(actions) do
+      local name, args = unpack(action)
+      if name == 'stop_app' or name == 'reconfig_app' then
+         local appname = args[1]
+         to_restart[appname] = nil
+      end
+   end
+   local to_relink = {}
+   for id, apps in pairs(to_restart) do
+      for appname, _ in pairs(apps) do
+         local info = assert(app_graph.apps[appname])
+         local class, arg = info.class, info.arg
+         if class.reconfig then
+            table.insert(actions, {'reconfig_app', {appname, class, arg}})
+         else
+            table.insert(actions, {'stop_app', {appname}})
+            table.insert(actions, {'start_app', {appname, class, arg}})
+            to_relink[appname] = true
+         end
+      end
+   end
+   for linkspec,_ in pairs(app_graph.links) do
+      local fa, fl, ta, tl = app_graph_mod.parse_link(linkspec)
+      if to_relink[fa] then
+         table.insert(actions, {'link_output', {fa, fl, linkspec}})
+      end
+      if to_relink[ta] then
+         table.insert(actions, {'link_input', {ta, tl, linkspec}})
+      end
+   end
+   table.insert(actions, {'commit', {}})
+   return actions
+end
+
+local function configuration_for_follower(follower, configuration)
+   return configuration
+end
+
+local function compute_state_reader(schema_name)
+   return function(pid)
+      local reader = state.state_reader_from_schema_by_name(schema_name)
+      return reader(state.counters_for_pid(pid))
+   end
+end
+
+local function process_states(states)
+   return states[1]
+end
+
+generic_schema_config_support = {
+   compute_config_actions = function(
+         old_graph, new_graph, to_restart, verb, path, ...)
+      return add_restarts(app.compute_config_actions(old_graph, new_graph),
+                          new_graph, to_restart)
+   end,
+   update_mutable_objects_embedded_in_app_initargs = function(
+         in_place_dependencies, app_graph, schema_name, verb, path, arg)
+      return compute_mutable_objects_embedded_in_app_initargs(app_graph)
+   end,
+   compute_state_reader = compute_state_reader,
+   configuration_for_follower = configuration_for_follower,
+   process_states = process_states,
+   compute_apps_to_restart_after_configuration_update =
+      compute_apps_to_restart_after_configuration_update,
+   translators = {}
+}
+
+function load_schema_config_support(schema_name)
+   local mod_name = 'lib.ptree.support.'..schema_name:gsub('-', '_')
+   local success, support_mod = pcall(require, mod_name)
+   if success then return support_mod.get_config_support() end
+   return generic_schema_config_support
+end
diff --git a/src/lib/ptree/support/snabb-softwire-v2.lua b/src/lib/ptree/support/snabb-softwire-v2.lua
new file mode 100644
index 0000000000..274fd80c53
--- /dev/null
+++ b/src/lib/ptree/support/snabb-softwire-v2.lua
@@ -0,0 +1,706 @@
+-- Use of this source code is governed by the Apache 2.0 license; see COPYING.
+module(..., package.seeall)
+local ffi = require('ffi')
+local app = require('core.app')
+local corelib = require('core.lib')
+local equal = require('core.lib').equal
+local dirname = require('core.lib').dirname
+local data = require('lib.yang.data')
+local state = require('lib.yang.state')
+local ipv4_ntop = require('lib.yang.util').ipv4_ntop
+local ipv6 = require('lib.protocol.ipv6')
+local yang = require('lib.yang.yang')
+local ctable = require('lib.ctable')
+local cltable = require('lib.cltable')
+local path_mod = require('lib.yang.path')
+local generic = require('lib.ptree.support').generic_schema_config_support
+local binding_table = require("apps.lwaftr.binding_table")
+
+local binding_table_instance
+local function get_binding_table_instance(conf)
+   if binding_table_instance == nil then
+      binding_table_instance = binding_table.load(conf)
+   end
+   return binding_table_instance
+end
+
+-- Packs snabb-softwire-v2 softwire entry into softwire and PSID blob
+--
+-- The data plane stores a separate table of psid maps and softwires. It
+-- requires that we give it a blob it can quickly add. These look rather
+-- similar to snabb-softwire-v1 structures however it maintains the br-address
+-- on the softwire so are subtly different.
+local function pack_softwire(app_graph, entry)
+   assert(app_graph.apps['lwaftr'])
+   assert(entry.value.port_set, "Softwire lacks port-set definition")
+   local key, value = entry.key, entry.value
+   
+   -- Get the binding table
+   local bt_conf = app_graph.apps.lwaftr.arg.softwire_config.binding_table
+   bt = get_binding_table_instance(bt_conf)
+
+   local softwire_t = bt.softwires.entry_type()
+   psid_map_t = bt.psid_map.entry_type()
+
+   -- Now lets pack the stuff!
+   local packed_softwire = ffi.new(softwire_t)
+   packed_softwire.key.ipv4 = key.ipv4
+   packed_softwire.key.psid = key.psid
+   packed_softwire.value.b4_ipv6 = value.b4_ipv6
+   packed_softwire.value.br_address = value.br_address
+
+   local packed_psid_map = ffi.new(psid_map_t)
+   packed_psid_map.key.addr = key.ipv4
+   if value.port_set.psid_length then
+      packed_psid_map.value.psid_length = value.port_set.psid_length
+   end
+
+   return packed_softwire, packed_psid_map
+end
+
+local function add_softwire_entry_actions(app_graph, entries)
+   assert(app_graph.apps['lwaftr'])
+   local bt_conf = app_graph.apps.lwaftr.arg.softwire_config.binding_table
+   local bt = get_binding_table_instance(bt_conf)
+   local ret = {}
+   for entry in entries:iterate() do
+      local psoftwire, ppsid = pack_softwire(app_graph, entry)
+      assert(bt:is_managed_ipv4_address(psoftwire.key.ipv4))
+
+      local softwire_args = {'lwaftr', 'add_softwire_entry', psoftwire}
+      table.insert(ret, {'call_app_method_with_blob', softwire_args})
+   end
+   table.insert(ret, {'commit', {}})
+   return ret
+end
+
+local softwire_grammar
+local function get_softwire_grammar()
+   if not softwire_grammar then
+      local schema = yang.load_schema_by_name('snabb-softwire-v2')
+      local grammar = data.config_grammar_from_schema(schema)
+      softwire_grammar =
+         assert(grammar.members['softwire-config'].
+                   members['binding-table'].members['softwire'])
+   end
+   return softwire_grammar
+end
+
+local function remove_softwire_entry_actions(app_graph, path)
+   assert(app_graph.apps['lwaftr'])
+   path = path_mod.parse_path(path)
+   local grammar = get_softwire_grammar()
+   local key = path_mod.prepare_table_lookup(
+      grammar.keys, grammar.key_ctype, path[#path].query)
+   local args = {'lwaftr', 'remove_softwire_entry', key}
+   -- If it's the last softwire for the corresponding psid entry, remove it.
+   -- TODO: check if last psid entry and then remove.
+   return {{'call_app_method_with_blob', args}, {'commit', {}}}
+end
+
+local function compute_config_actions(old_graph, new_graph, to_restart,
+                                      verb, path, arg)
+   -- If the binding cable changes, remove our cached version.
+   if path ~= nil and path:match("^/softwire%-config/binding%-table") then
+      binding_table_instance = nil
+   end
+
+   if verb == 'add' and path == '/softwire-config/binding-table/softwire' then
+      if to_restart == false then
+	 return add_softwire_entry_actions(new_graph, arg)
+      end
+   elseif (verb == 'remove' and
+           path:match('^/softwire%-config/binding%-table/softwire')) then
+      return remove_softwire_entry_actions(new_graph, path)
+   elseif (verb == 'set' and path == '/softwire-config/name') then
+      return {}
+   end
+   return generic.compute_config_actions(
+      old_graph, new_graph, to_restart, verb, path, arg)
+end
+
+local function update_mutable_objects_embedded_in_app_initargs(
+      in_place_dependencies, app_graph, schema_name, verb, path, arg)
+   if verb == 'add' and path == '/softwire-config/binding-table/softwire' then
+      return in_place_dependencies
+   elseif (verb == 'remove' and
+           path:match('^/softwire%-config/binding%-table/softwire')) then
+      return in_place_dependencies
+   else
+      return generic.update_mutable_objects_embedded_in_app_initargs(
+         in_place_dependencies, app_graph, schema_name, verb, path, arg)
+   end
+end
+
+local function compute_apps_to_restart_after_configuration_update(
+      schema_name, configuration, verb, path, in_place_dependencies, arg)
+   if verb == 'add' and path == '/softwire-config/binding-table/softwire' then
+      -- We need to check if the softwire defines a new port-set, if so we need to
+      -- restart unfortunately. If not we can just add the softwire.
+      local bt = get_binding_table_instance(configuration.softwire_config.binding_table)
+      local to_restart = false
+      for entry in arg:iterate() do
+	 to_restart = (bt:is_managed_ipv4_address(entry.key.ipv4) == false) or false
+      end
+      if to_restart == false then return {} end
+   elseif (verb == 'remove' and
+           path:match('^/softwire%-config/binding%-table/softwire')) then
+      return {}
+   elseif (verb == 'set' and path == '/softwire-config/name') then
+      return {}
+   end
+   return generic.compute_apps_to_restart_after_configuration_update(
+      schema_name, configuration, verb, path, in_place_dependencies, arg)
+end
+
+local function memoize1(f)
+   local memoized_arg, memoized_result
+   return function(arg)
+      if arg == memoized_arg then return memoized_result end
+      memoized_result = f(arg)
+      memoized_arg = arg
+      return memoized_result
+   end
+end
+
+local function cltable_for_grammar(grammar)
+   assert(grammar.key_ctype)
+   assert(not grammar.value_ctype)
+   local key_t = data.typeof(grammar.key_ctype)
+   return cltable.new({key_type=key_t}), key_t
+end
+
+local ietf_br_instance_grammar
+local function get_ietf_br_instance_grammar()
+   if not ietf_br_instance_grammar then
+      local schema = yang.load_schema_by_name('ietf-softwire-br')
+      local grammar = data.config_grammar_from_schema(schema)
+      grammar = assert(grammar.members['br-instances'])
+      grammar = assert(grammar.members['br-type'])
+      grammar = assert(grammar.choices['binding'].binding)
+      grammar = assert(grammar.members['br-instance'])
+      ietf_br_instance_grammar = grammar
+   end
+   return ietf_br_instance_grammar
+end
+
+local ietf_softwire_grammar
+local function get_ietf_softwire_grammar()
+   if not ietf_softwire_grammar then
+      local grammar = get_ietf_br_instance_grammar()
+      grammar = assert(grammar.values['binding-table'])
+      grammar = assert(grammar.members['binding-entry'])
+      ietf_softwire_grammar = grammar
+   end
+   return ietf_softwire_grammar
+end
+
+local function ietf_binding_table_from_native(bt)
+   local ret, key_t = cltable_for_grammar(get_ietf_softwire_grammar())
+   for softwire in bt.softwire:iterate() do
+      local k = key_t({ binding_ipv6info = softwire.value.b4_ipv6 })
+      local v = {
+         binding_ipv4_addr = softwire.key.ipv4,
+         port_set = {
+            psid_offset = softwire.value.port_set.reserved_ports_bit_count,
+            psid_len = softwire.value.port_set.psid_length,
+            psid = softwire.key.psid
+         },
+         br_ipv6_addr = softwire.value.br_address,
+      }
+      ret[k] = v
+   end
+   return ret
+end
+
+local function schema_getter(schema_name, path)
+   local schema = yang.load_schema_by_name(schema_name)
+   local grammar = data.config_grammar_from_schema(schema)
+   return path_mod.resolver(grammar, path)
+end
+
+local function snabb_softwire_getter(path)
+   return schema_getter('snabb-softwire-v2', path)
+end
+
+local function ietf_softwire_br_getter(path)
+   return schema_getter('ietf-softwire-br', path)
+end
+
+local function native_binding_table_from_ietf(ietf)
+   local _, softwire_grammar =
+      snabb_softwire_getter('/softwire-config/binding-table/softwire')
+   local softwire_key_t = data.typeof(softwire_grammar.key_ctype)
+   local softwire = cltable.new({key_type=softwire_key_t})
+   for k,v in cltable.pairs(ietf) do
+      local softwire_key =
+         softwire_key_t({ipv4=v.binding_ipv4_addr, psid=v.port_set.psid})
+      local softwire_value = {
+         br_address=v.br_ipv6_addr,
+         b4_ipv6=k.binding_ipv6info,
+         port_set={
+            psid_length=v.port_set.psid_len,
+            reserved_ports_bit_count=v.port_set.psid_offset
+         }
+      }
+      cltable.set(softwire, softwire_key, softwire_value)
+   end
+   return {softwire=softwire}
+end
+
+local function serialize_binding_table(bt)
+   local _, grammar = snabb_softwire_getter('/softwire-config/binding-table')
+   local printer = data.data_printer_from_grammar(grammar)
+   return printer(bt, yang.string_output_file())
+end
+
+local uint64_ptr_t = ffi.typeof('uint64_t*')
+function ipv6_equals(a, b)
+   local x, y = ffi.cast(uint64_ptr_t, a), ffi.cast(uint64_ptr_t, b)
+   return x[0] == y[0] and x[1] == y[1]
+end
+
+local function ietf_softwire_br_translator ()
+   local ret = {}
+   local instance_id_map = {}
+   local cached_config
+   local function instance_id_by_device(device)
+      local last
+      for id, pciaddr in ipairs(instance_id_map) do
+	 if pciaddr == device then return id end
+	 last = id
+      end
+      if last == nil then
+	 last = 1
+      else
+	 last = last + 1
+      end
+      instance_id_map[last] = device
+      return last
+   end
+   function ret.get_config(native_config)
+      if cached_config ~= nil then return cached_config end
+      local int = native_config.softwire_config.internal_interface
+      local int_err = int.error_rate_limiting
+      local ext = native_config.softwire_config.external_interface
+      local br_instance, br_instance_key_t =
+         cltable_for_grammar(get_ietf_br_instance_grammar())
+      for device, instance in pairs(native_config.softwire_config.instance) do
+	 br_instance[br_instance_key_t({id=instance_id_by_device(device)})] = {
+	    name = native_config.softwire_config.name,
+	    tunnel_payload_mtu = int.mtu,
+	    tunnel_path_mru = ext.mtu,
+	    -- FIXME: There's no equivalent of softwire-num-threshold in
+            -- snabb-softwire-v1.
+	    softwire_num_threshold = 0xffffffff,
+            enable_hairpinning = int.hairpinning,
+	    binding_table = {
+	       binding_entry = ietf_binding_table_from_native(
+		  native_config.softwire_config.binding_table)
+	    },
+            icmp_policy = {
+               icmpv4_errors = {
+                  allow_incoming_icmpv4 = ext.allow_incoming_icmp,
+                  generate_icmpv4_errors = ext.generate_icmp_errors
+               },
+               icmpv6_errors = {
+                  generate_icmpv6_errors = int.generate_icmp_errors,
+                  icmpv6_errors_rate =
+                     math.floor(int_err.packets / int_err.period)
+               }
+            }
+	 }
+      end
+      cached_config = {
+         br_instances = {
+            binding = { br_instance = br_instance }
+         }
+      }
+      return cached_config
+   end
+   function ret.get_state(native_state)
+      -- Even though this is a different br-instance node, it is a
+      -- cltable with the same key type, so just re-use the key here.
+      local br_instance, br_instance_key_t =
+         cltable_for_grammar(get_ietf_br_instance_grammar())
+      for device, instance in pairs(native_state.softwire_config.instance) do
+         local c = instance.softwire_state
+	 br_instance[br_instance_key_t({id=instance_id_by_device(device)})] = {
+            traffic_stat = {
+               sent_ipv4_packet = c.out_ipv4_packets,
+               sent_ipv4_byte = c.out_ipv4_bytes,
+               sent_ipv6_packet = c.out_ipv6_packets,
+               sent_ipv6_byte = c.out_ipv6_bytes,
+               rcvd_ipv4_packet = c.in_ipv4_packets,
+               rcvd_ipv4_byte = c.in_ipv4_bytes,
+               rcvd_ipv6_packet = c.in_ipv6_packets,
+               rcvd_ipv6_byte = c.in_ipv6_bytes,
+               dropped_ipv4_packet = c.drop_all_ipv4_iface_packets,
+               dropped_ipv4_byte = c.drop_all_ipv4_iface_bytes,
+               dropped_ipv6_packet = c.drop_all_ipv6_iface_packets,
+               dropped_ipv6_byte = c.drop_all_ipv6_iface_bytes,
+               dropped_ipv4_fragments = 0, -- FIXME
+               dropped_ipv4_bytes = 0, -- FIXME
+               ipv6_fragments_reassembled = c.in_ipv6_frag_reassembled,
+               ipv6_fragments_bytes_reassembled = 0, -- FIXME
+               out_icmpv4_error_packets = c.out_icmpv4_error_packets,
+               out_icmpv6_error_packets = c.out_icmpv6_error_packets,
+               hairpin_ipv4_bytes = c.hairpin_ipv4_bytes,
+               hairpin_ipv4_packets = c.hairpin_ipv4_packets,
+               active_softwire_num = 0, -- FIXME
+            }
+         }
+      end
+      return {
+         br_instances = {
+            binding = { br_instance = br_instance }
+         }
+      }
+   end
+   local function sets_whole_table(path, count)
+      if #path > count then return false end
+      if #path == count then
+         for k,v in pairs(path[#path].query) do return false end
+      end
+      return true
+   end
+   function ret.set_config(native_config, path_str, arg)
+      path = path_mod.parse_path(path_str)
+      local br_instance_paths = {'br-instances', 'binding', 'br-instance'}
+      local bt_paths = {'binding-table', 'binding-entry'}
+
+      -- Can't actually set the instance itself.
+      if #path <= #br_instance_paths then
+         error("Unspported path: "..path_str)
+      end
+
+      -- Handle special br attributes (tunnel-payload-mtu, tunnel-path-mru, softwire-num-threshold).
+      if #path > #br_instance_paths then
+         local maybe_leaf = path[#path].name
+         local path_tails = {
+            ['tunnel-payload-mtu'] = 'internal-interface/mtu',
+            ['tunnel-path-mtu'] = 'external-interface/mtu',
+            ['name'] = 'name',
+            ['enable-hairpinning'] = 'internal-interface/hairpinning',
+            ['allow-incoming-icmpv4'] = 'external-interface/allow-incoming-icmp',
+            ['generate-icmpv4-errors'] = 'external-interface/generate-icmp-errors',
+            ['generate-icmpv6-errors'] = 'internal-interface/generate-icmp-errors'
+         }
+         local path_tail = path_tails[maybe_leaf]
+         if path_tail then
+            return {{'set', {schema='snabb-softwire-v2',
+                             path='/softwire-config/'..path_tail,
+                             config=tostring(arg)}}}
+         elseif maybe_leaf == 'icmpv6-errors-rate' then
+            local head = '/softwire-config/internal-interface/error-rate-limiting'
+            return {
+               {'set', {schema='snabb-softwire-v2', path=head..'/packets',
+                        config=tostring(arg * 2)}},
+               {'set', {schema='snabb-softwire-v2', path=head..'/period',
+                        config='2'}}}
+         else
+            error('unrecognized leaf: '..maybe_leaf)
+         end
+      end
+
+      -- Two kinds of updates: setting the whole binding table, or
+      -- updating one entry.
+      if sets_whole_table(path, #br_instance_paths + #bt_paths) then
+         -- Setting the whole binding table.
+         if sets_whole_table(path, #br_instance_paths) then
+            for i=#path+1,#br_instance_paths do
+               arg = arg[data.normalize_id(br_instance_paths[i])]
+            end
+            local instance
+            for k,v in cltable.pairs(arg) do
+               if instance then error('multiple instances in config') end
+               if k.id ~= 1 then error('instance id not 1: '..tostring(k.id)) end
+               instance = v
+            end
+            if not instance then error('no instances in config') end
+            arg = instance
+         end
+         for i=math.max(#path-#br_instance_paths,0)+1,#bt_paths do
+            arg = arg[data.normalize_id(bt_paths[i])]
+         end
+         local bt = native_binding_table_from_ietf(arg)
+         return {{'set', {schema='snabb-softwire-v2',
+                          path='/softwire-config/binding-table',
+                          config=serialize_binding_table(bt)}}}
+      else
+         -- An update to an existing entry.  First, get the existing entry.
+         local config = ret.get_config(native_config)
+         local entry_path = path_str
+         local entry_path_len = #br_instance_paths + #bt_paths
+         for i=entry_path_len+1, #path do
+            entry_path = dirname(entry_path)
+         end
+         local old = ietf_softwire_br_getter(entry_path)(config)
+         -- Now figure out what the new entry should look like.
+         local new
+         if #path == entry_path_len then
+            new = arg
+         else
+            new = {
+               port_set = {
+                  psid_offset = old.port_set.psid_offset,
+                  psid_len = old.port_set.psid_len,
+                  psid = old.port_set.psid
+               },
+               binding_ipv4_addr = old.binding_ipv4_addr,
+               br_ipv6_addr = old.br_ipv6_addr
+            }
+            if path[entry_path_len + 1].name == 'port-set' then
+               if #path == entry_path_len + 1 then
+                  new.port_set = arg
+               else
+                  local k = data.normalize_id(path[#path].name)
+                  new.port_set[k] = arg
+               end
+            elseif path[#path].name == 'binding-ipv4-addr' then
+               new.binding_ipv4_addr = arg
+            elseif path[#path].name == 'br-ipv6-addr' then
+               new.br_ipv6_addr = arg
+            else
+               error('bad path element: '..path[#path].name)
+            end
+         end
+         -- Apply changes.  Ensure  that the port-set
+         -- changes are compatible with the existing configuration.
+         local updates = {}
+         local softwire_path = '/softwire-config/binding-table/softwire'
+
+         -- Lets remove this softwire entry and add a new one.
+         local function q(ipv4, psid)
+            return string.format('[ipv4=%s][psid=%s]', ipv4_ntop(ipv4), psid)
+         end
+         local old_query = q(old.binding_ipv4_addr, old.port_set.psid)
+         -- FIXME: This remove will succeed but the add could fail if
+         -- there's already a softwire with this IPv4 and PSID.  We need
+         -- to add a check here that the IPv4/PSID is not present in the
+         -- binding table.
+         table.insert(updates,
+                      {'remove', {schema='snabb-softwire-v2',
+                                  path=softwire_path..old_query}})
+
+         local config_str = string.format([[{
+            ipv4 %s;
+            psid %s;
+            br-address %s;
+            b4-ipv6 %s;
+            port-set {
+               psid-length %s;
+               reserved-ports-bit-count %s;
+            }
+         }]], ipv4_ntop(new.binding_ipv4_addr), new.port_set.psid,
+              ipv6:ntop(new.br_ipv6_addr),
+              path[entry_path_len].query['binding-ipv6info'],
+              new.port_set.psid_len, new.port_set.psid_offset)
+         table.insert(updates,
+                      {'add', {schema='snabb-softwire-v2',
+                               path=softwire_path,
+                               config=config_str}})
+         return updates
+      end
+   end
+   function ret.add_config(native_config, path_str, data)
+      local binding_entry_path = {'br-instances', 'binding', 'br-instance',
+                                  'binding-table', 'binding-entry'}
+      local path = path_mod.parse_path(path_str)
+
+      if #path ~= #binding_entry_path then
+         error('unsupported path: '..path)
+      end
+      local config = ret.get_config(native_config)
+      local ietf_bt = ietf_softwire_br_getter(path_str)(config)
+      local old_bt = native_config.softwire_config.binding_table
+      local new_bt = native_binding_table_from_ietf(data)
+      local updates = {}
+      local softwire_path = '/softwire-config/binding-table/softwire'
+      local psid_map_path = '/softwire-config/binding-table/psid-map'
+      -- Add softwires.
+      local additions = {}
+      for entry in new_bt.softwire:iterate() do
+         local key, value = entry.key, entry.value
+         if old_bt.softwire:lookup_ptr(key) ~= nil then
+            error('softwire already present in table: '..
+                     inet_ntop(key.ipv4)..'/'..key.psid)
+         end
+         local config_str = string.format([[{
+            ipv4 %s;
+            psid %s;
+            br-address %s;
+            b4-ipv6 %s;
+            port-set {
+               psid-length %s;
+               reserved-ports-bit-count %s;
+            }
+         }]], ipv4_ntop(key.ipv4), key.psid,
+              ipv6:ntop(value.br_address),
+              ipv6:ntop(value.b4_ipv6),
+              value.port_set.psid_length,
+              value.port_set.reserved_ports_bit_count
+         )
+         table.insert(additions, config_str)
+      end
+      table.insert(updates,
+                   {'add', {schema='snabb-softwire-v2',
+                            path=softwire_path,
+                            config=table.concat(additions, '\n')}})
+      return updates
+   end
+   function ret.remove_config(native_config, path_str)
+      local path = path_mod.parse_path(path_str)
+      local ietf_binding_table_path = {'softwire-config', 'binding', 'br',
+         'br-instances', 'br-instance', 'binding-table'}
+      local ietf_instance_path = {'softwire-config', 'binding', 'br',
+         'br-instances', 'br-instance'}
+
+      if #path == #ietf_instance_path then
+         -- Remove appropriate instance
+         local ietf_instance_id = tonumber(assert(path[5].query).id)
+         local instance_path = "/softwire-config/instance"
+
+         -- If it's not been populated in instance_id_map this is meaningless
+         -- and dangerous as they have no mapping from snabb's "device".
+         local function q(device) return
+            string.format("[device=%s]", device)
+         end
+         local device = instance_id_map[ietf_instance_id]
+         if device then
+            return {{'remove', {schema='snabb-softwire-v2',
+                                path=instance_path..q(device)}}}
+         else
+            error(string.format(
+               "Could not find '%s' in ietf instance mapping", ietf_instance_id
+            ))
+         end
+      elseif #path == #ietf_binding_table_path then
+         local softwire_path = '/softwire-config/binding-table/softwire'
+         if path:sub(-1) ~= ']' then error('unsupported path: '..path_str) end
+         local config = ret.get_config(native_config)
+         local entry = ietf_softwire_getter(path_str)(config)
+         local function q(ipv4, psid)
+            return string.format('[ipv4=%s][psid=%s]', ipv4_ntop(ipv4), psid)
+         end
+         local query = q(entry.binding_ipv4_addr, entry.port_set.psid)
+         return {{'remove', {schema='snabb-softwire-v2',
+                             path=softwire_path..query}}}
+      else
+         return error('unsupported path: '..path_str)
+      end
+   end
+   function ret.pre_update(native_config, verb, path, data)
+      -- Given the notification that the native config is about to be
+      -- updated, make our cached config follow along if possible (and
+      -- if we have one).  Otherwise throw away our cached config; we'll
+      -- re-make it next time.
+      if cached_config == nil then return end
+      local br_instance = cached_config.br_instances.binding.br_instance
+      if (verb == 'remove' and
+          path:match('^/softwire%-config/binding%-table/softwire')) then
+         -- Remove a softwire.
+         local value = snabb_softwire_getter(path)(native_config)
+         for _,instance in cltable.pairs(br_instance) do
+            local grammar = get_ietf_softwire_grammar()
+            local key = path_mod.prepare_table_lookup(
+               grammar.keys, grammar.key_ctype, {['binding-ipv6info']='::'})
+            key.binding_ipv6info = value.b4_ipv6
+            assert(instance.binding_table.binding_entry[key] ~= nil)
+            instance.binding_table.binding_entry[key] = nil
+         end
+      elseif (verb == 'add' and
+              path == '/softwire-config/binding-table/softwire') then
+         local bt = native_config.softwire_config.binding_table
+         for k,v in cltable.pairs(
+            ietf_binding_table_from_native({softwire = data})) do
+            for _,instance in cltable.pairs(br_instance) do
+               instance.binding_table.binding_entry[k] = v
+            end
+         end
+      elseif (verb == 'set' and path == "/softwire-config/name") then
+	 local br = cached_config.softwire_config.binding.br
+	 for _, instance in cltable.pairs(br_instance) do
+	    instance.name = data
+	 end
+      else
+         cached_config = nil
+      end
+   end
+   return ret
+end
+
+local function configuration_for_follower(follower, configuration)
+   return follower.graph.apps.lwaftr.arg
+end
+
+local function compute_state_reader(schema_name)
+   -- The schema has two lists which we want to look in.
+   local schema = yang.load_schema_by_name(schema_name)
+   local grammar = data.data_grammar_from_schema(schema, false)
+
+   local instance_list_gmr = grammar.members["softwire-config"].members.instance
+   local instance_state_gmr = instance_list_gmr.values["softwire-state"]
+
+   local base_reader = state.state_reader_from_grammar(grammar)
+   local instance_state_reader = state.state_reader_from_grammar(instance_state_gmr)
+
+   return function(pid, data)
+      local counters = state.counters_for_pid(pid)
+      local ret = base_reader(counters)
+      ret.softwire_config.instance = {}
+
+      for device, instance in pairs(data.softwire_config.instance) do
+         local instance_state = instance_state_reader(counters)
+         ret.softwire_config.instance[device] = {}
+         ret.softwire_config.instance[device].softwire_state = instance_state
+      end
+
+      return ret
+   end
+end
+
+local function process_states(states)
+   -- We need to create a summation of all the states as well as adding all the
+   -- instance specific state data to create a total in software-state.
+
+   local unified = {
+      softwire_config = {instance = {}},
+      softwire_state = {}
+   }
+
+   local function total_counter(name, softwire_stats, value)
+      if softwire_stats[name] == nil then
+         return value
+      else
+         return softwire_stats[name] + value
+      end
+   end
+
+   for _, inst_config in ipairs(states) do
+      local name, instance = next(inst_config.softwire_config.instance)
+      unified.softwire_config.instance[name] = instance
+
+      for name, value in pairs(instance.softwire_state) do
+         unified.softwire_state[name] = total_counter(
+            name, unified.softwire_state, value)
+      end
+   end
+
+   return unified
+end
+
+
+function get_config_support()
+   return {
+      compute_config_actions = compute_config_actions,
+      update_mutable_objects_embedded_in_app_initargs =
+         update_mutable_objects_embedded_in_app_initargs,
+      compute_apps_to_restart_after_configuration_update =
+         compute_apps_to_restart_after_configuration_update,
+      compute_state_reader = compute_state_reader,
+      process_states = process_states,
+      configuration_for_follower = configuration_for_follower,
+      translators = { ['ietf-softwire-br'] = ietf_softwire_br_translator () }
+   }
+end

From 370498708cda4fcc2639efc9f791666e61b3daa8 Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@igalia.com>
Date: Wed, 6 Dec 2017 14:56:14 +0100
Subject: [PATCH 03/31] Rename lib.ptree.leader to lib.ptree.manager

---
 src/lib/ptree/alarm_codec.lua             |   2 +-
 src/lib/ptree/channel.lua                 |  15 ++--
 src/lib/ptree/follower.lua                |   4 +-
 src/lib/ptree/{leader.lua => manager.lua} | 100 +++++++++++-----------
 4 files changed, 60 insertions(+), 61 deletions(-)
 rename src/lib/ptree/{leader.lua => manager.lua} (92%)

diff --git a/src/lib/ptree/alarm_codec.lua b/src/lib/ptree/alarm_codec.lua
index 7b7c598751..7e3c812751 100644
--- a/src/lib/ptree/alarm_codec.lua
+++ b/src/lib/ptree/alarm_codec.lua
@@ -184,7 +184,7 @@ function alarm:normalize_args (t)
    return normalize(t, self.args_attrs)
 end
 
--- To be used by the leader to group args into key and args.
+-- To be used by the manager to group args into key and args.
 function to_alarm (args)
    local key = {
       resource = args[1],
diff --git a/src/lib/ptree/channel.lua b/src/lib/ptree/channel.lua
index f78926397e..1818bb7b43 100644
--- a/src/lib/ptree/channel.lua
+++ b/src/lib/ptree/channel.lua
@@ -2,14 +2,13 @@
 
 module(...,package.seeall)
 
--- A channel is a ring buffer used by the config leader app to send
--- updates to a follower.  Each follower has its own ring buffer and is
--- the only reader to the buffer.  The config leader is the only writer
--- to these buffers also.  The ring buffer is just bytes; putting a
--- message onto the buffer will write a header indicating the message
--- size, then the bytes of the message.  The channel ring buffer is
--- mapped into shared memory.  Access to a channel will never block or
--- cause a system call.
+-- A channel is a ring buffer used by the manager to send updates to a
+-- follower.  Each follower has its own ring buffer and is the only
+-- reader to the buffer.  The manager is the only writer to these
+-- buffers also.  The ring buffer is just bytes; putting a message onto
+-- the buffer will write a header indicating the message size, then the
+-- bytes of the message.  The channel ring buffer is mapped into shared
+-- memory.  Access to a channel will never block or cause a system call.
 
 local ffi = require('ffi')
 local S = require("syscall")
diff --git a/src/lib/ptree/follower.lua b/src/lib/ptree/follower.lua
index 497d1bed95..e21c9a2a00 100644
--- a/src/lib/ptree/follower.lua
+++ b/src/lib/ptree/follower.lua
@@ -64,7 +64,7 @@ function Follower:commit_pending_actions()
    if should_flush then require('jit').flush() end
 end
 
-function Follower:handle_actions_from_leader()
+function Follower:handle_actions_from_manager()
    local channel = self.channel
    for i=1,4 do
       local buf, len = channel:peek_message()
@@ -82,7 +82,7 @@ end
 function Follower:pull ()
    if app.now() < self.next_time then return end
    self.next_time = app.now() + self.period
-   self:handle_actions_from_leader()
+   self:handle_actions_from_manager()
 end
 
 function selftest ()
diff --git a/src/lib/ptree/leader.lua b/src/lib/ptree/manager.lua
similarity index 92%
rename from src/lib/ptree/leader.lua
rename to src/lib/ptree/manager.lua
index 63581b4307..8e80906e12 100644
--- a/src/lib/ptree/leader.lua
+++ b/src/lib/ptree/manager.lua
@@ -24,9 +24,9 @@ local support = require("lib.ptree.support")
 local channel = require("lib.ptree.channel")
 local alarms = require("lib.yang.alarms")
 
-Leader = {
+Manager = {
    config = {
-      socket_file_name = {default='config-leader-socket'},
+      socket_file_name = {default='config-manager-socket'},
       setup_fn = {required=true},
       -- Could relax this requirement.
       initial_configuration = {required=true},
@@ -48,8 +48,8 @@ local function open_socket (file)
    return socket
 end
 
-function Leader:new (conf)
-   local ret = setmetatable({}, {__index=Leader})
+function Manager:new (conf)
+   local ret = setmetatable({}, {__index=Manager})
    ret.cpuset = conf.cpuset
    ret.socket_file_name = conf.socket_file_name
    if not ret.socket_file_name:match('^/') then
@@ -74,7 +74,7 @@ function Leader:new (conf)
    return ret
 end
 
-function Leader:set_initial_configuration (configuration)
+function Manager:set_initial_configuration (configuration)
    self.current_configuration = configuration
    self.current_in_place_dependencies = {}
 
@@ -93,7 +93,7 @@ function Leader:set_initial_configuration (configuration)
    end
 end
 
-function Leader:start_worker(cpu)
+function Manager:start_worker(cpu)
    local start_code = { self.worker_start_code }
    if cpu then
       table.insert(start_code, 1, "print('Bound data plane to CPU:',"..cpu..")")
@@ -102,7 +102,7 @@ function Leader:start_worker(cpu)
    return worker.start("follower", table.concat(start_code, "\n"))
 end
 
-function Leader:stop_worker(id)
+function Manager:stop_worker(id)
    -- Tell the worker to terminate
    local stop_actions = {{'shutdown', {}}, {'commit', {}}}
    self:enqueue_config_actions_for_follower(id, stop_actions)
@@ -110,7 +110,7 @@ function Leader:stop_worker(id)
    self.followers[id].shutting_down = true
 end
 
-function Leader:remove_stale_followers()
+function Manager:remove_stale_followers()
    local stale = {}
    for id, follower in pairs(self.followers) do
       if follower.shutting_down then
@@ -128,7 +128,7 @@ function Leader:remove_stale_followers()
    end
 end
 
-function Leader:acquire_cpu_for_follower(id, app_graph)
+function Manager:acquire_cpu_for_follower(id, app_graph)
    local pci_addresses = {}
    -- Grovel through app initargs for keys named "pciaddr".  Hacky!
    for name, init in pairs(app_graph.apps) do
@@ -141,7 +141,7 @@ function Leader:acquire_cpu_for_follower(id, app_graph)
    return self.cpuset:acquire_for_pci_addresses(pci_addresses)
 end
 
-function Leader:start_follower_for_graph(id, graph)
+function Manager:start_follower_for_graph(id, graph)
    local cpu = self:acquire_cpu_for_follower(id, graph)
    self.followers[id] = { cpu=cpu, pid=self:start_worker(cpu), queue={},
                           graph=graph }
@@ -151,15 +151,15 @@ function Leader:start_follower_for_graph(id, graph)
    return self.followers[id]
 end
 
-function Leader:take_follower_message_queue ()
+function Manager:take_follower_message_queue ()
    local actions = self.config_action_queue
    self.config_action_queue = nil
    return actions
 end
 
-local verbose = os.getenv('SNABB_LEADER_VERBOSE') and true
+local verbose = os.getenv('SNABB_MANAGER_VERBOSE') and true
 
-function Leader:enqueue_config_actions_for_follower(follower, actions)
+function Manager:enqueue_config_actions_for_follower(follower, actions)
    for _,action in ipairs(actions) do
       if verbose then print('encode', action[1], unpack(action[2])) end
       local buf, len = action_codec.encode(action)
@@ -167,13 +167,13 @@ function Leader:enqueue_config_actions_for_follower(follower, actions)
    end
 end
 
-function Leader:enqueue_config_actions (actions)
+function Manager:enqueue_config_actions (actions)
    for id,_ in pairs(self.followers) do
       self.enqueue_config_actions_for_follower(id, actions)
    end
 end
 
-function Leader:rpc_describe (args)
+function Manager:rpc_describe (args)
    local alternate_schemas = {}
    for schema_name, translator in pairs(self.support.translators) do
       table.insert(alternate_schemas, schema_name)
@@ -210,7 +210,7 @@ local function path_printer_for_schema_by_name(schema_name, path, is_config,
                                   print_default)
 end
 
-function Leader:rpc_get_config (args)
+function Manager:rpc_get_config (args)
    local function getter()
       if args.schema ~= self.schema_name then
          return self:foreign_rpc_get_config(
@@ -225,7 +225,7 @@ function Leader:rpc_get_config (args)
    if success then return response else return {status=1, error=response} end
 end
 
-function Leader:rpc_set_alarm_operator_state (args)
+function Manager:rpc_set_alarm_operator_state (args)
    local function getter()
       if args.schema ~= self.schema_name then
          return false, ("Set-operator-state operation not supported in"..
@@ -240,7 +240,7 @@ function Leader:rpc_set_alarm_operator_state (args)
    if success then return response else return {status=1, error=response} end
 end
 
-function Leader:rpc_purge_alarms (args)
+function Manager:rpc_purge_alarms (args)
    local function purge()
       if args.schema ~= self.schema_name then
          return false, ("Purge-alarms operation not supported in"..
@@ -252,7 +252,7 @@ function Leader:rpc_purge_alarms (args)
    if success then return response else return {status=1, error=response} end
 end
 
-function Leader:rpc_compress_alarms (args)
+function Manager:rpc_compress_alarms (args)
    local function compress()
       if args.schema ~= self.schema_name then
          return false, ("Compress-alarms operation not supported in"..
@@ -520,13 +520,13 @@ function compute_remove_config_fn (schema_name, path)
    return path_remover_for_schema(yang.load_schema_by_name(schema_name), path)
 end
 
-function Leader:notify_pre_update (config, verb, path, ...)
+function Manager:notify_pre_update (config, verb, path, ...)
    for _,translator in pairs(self.support.translators) do
       translator.pre_update(config, verb, path, ...)
    end
 end
 
-function Leader:update_configuration (update_fn, verb, path, ...)
+function Manager:update_configuration (update_fn, verb, path, ...)
    self:notify_pre_update(self.current_configuration, verb, path, ...)
    local to_restart =
       self.support.compute_apps_to_restart_after_configuration_update (
@@ -556,7 +556,7 @@ function Leader:update_configuration (update_fn, verb, path, ...)
          self.current_in_place_dependencies, new_graphs, verb, path, ...)
 end
 
-function Leader:handle_rpc_update_config (args, verb, compute_update_fn)
+function Manager:handle_rpc_update_config (args, verb, compute_update_fn)
    local path = path_mod.normalize_path(args.path)
    local parser = path_parser_for_schema_by_name(args.schema, path)
    self:update_configuration(compute_update_fn(args.schema, path),
@@ -564,7 +564,7 @@ function Leader:handle_rpc_update_config (args, verb, compute_update_fn)
    return {}
 end
 
-function Leader:get_native_state ()
+function Manager:get_native_state ()
    local states = {}
    local state_reader = self.support.compute_state_reader(self.schema_name)
    for _, follower in pairs(self.followers) do
@@ -575,12 +575,12 @@ function Leader:get_native_state ()
    return self.support.process_states(states)
 end
 
-function Leader:get_translator (schema_name)
+function Manager:get_translator (schema_name)
    local translator = self.support.translators[schema_name]
    if translator then return translator end
    error('unsupported schema: '..schema_name)
 end
-function Leader:apply_translated_rpc_updates (updates)
+function Manager:apply_translated_rpc_updates (updates)
    for _,update in ipairs(updates) do
       local verb, args = unpack(update)
       local method = assert(self['rpc_'..verb..'_config'])
@@ -588,7 +588,7 @@ function Leader:apply_translated_rpc_updates (updates)
    end
    return {}
 end
-function Leader:foreign_rpc_get_config (schema_name, path, format,
+function Manager:foreign_rpc_get_config (schema_name, path, format,
                                         print_default)
    path = path_mod.normalize_path(path)
    local translate = self:get_translator(schema_name)
@@ -598,7 +598,7 @@ function Leader:foreign_rpc_get_config (schema_name, path, format,
    local config = printer(foreign_config, yang.string_output_file())
    return { config = config }
 end
-function Leader:foreign_rpc_get_state (schema_name, path, format,
+function Manager:foreign_rpc_get_state (schema_name, path, format,
                                        print_default)
    path = path_mod.normalize_path(path)
    local translate = self:get_translator(schema_name)
@@ -608,7 +608,7 @@ function Leader:foreign_rpc_get_state (schema_name, path, format,
    local state = printer(foreign_state, yang.string_output_file())
    return { state = state }
 end
-function Leader:foreign_rpc_set_config (schema_name, path, config_str)
+function Manager:foreign_rpc_set_config (schema_name, path, config_str)
    path = path_mod.normalize_path(path)
    local translate = self:get_translator(schema_name)
    local parser = path_parser_for_schema_by_name(schema_name, path)
@@ -616,7 +616,7 @@ function Leader:foreign_rpc_set_config (schema_name, path, config_str)
                                         parser(config_str))
    return self:apply_translated_rpc_updates(updates)
 end
-function Leader:foreign_rpc_add_config (schema_name, path, config_str)
+function Manager:foreign_rpc_add_config (schema_name, path, config_str)
    path = path_mod.normalize_path(path)
    local translate = self:get_translator(schema_name)
    local parser = path_parser_for_schema_by_name(schema_name, path)
@@ -624,14 +624,14 @@ function Leader:foreign_rpc_add_config (schema_name, path, config_str)
                                         parser(config_str))
    return self:apply_translated_rpc_updates(updates)
 end
-function Leader:foreign_rpc_remove_config (schema_name, path)
+function Manager:foreign_rpc_remove_config (schema_name, path)
    path = path_mod.normalize_path(path)
    local translate = self:get_translator(schema_name)
    local updates = translate.remove_config(self.current_configuration, path)
    return self:apply_translated_rpc_updates(updates)
 end
 
-function Leader:rpc_set_config (args)
+function Manager:rpc_set_config (args)
    local function setter()
       if self.listen_peer ~= nil and self.listen_peer ~= self.rpc_peer then
          error('Attempt to modify configuration while listener attached')
@@ -645,7 +645,7 @@ function Leader:rpc_set_config (args)
    if success then return response else return {status=1, error=response} end
 end
 
-function Leader:rpc_add_config (args)
+function Manager:rpc_add_config (args)
    local function adder()
       if self.listen_peer ~= nil and self.listen_peer ~= self.rpc_peer then
          error('Attempt to modify configuration while listener attached')
@@ -659,7 +659,7 @@ function Leader:rpc_add_config (args)
    if success then return response else return {status=1, error=response} end
 end
 
-function Leader:rpc_remove_config (args)
+function Manager:rpc_remove_config (args)
    local function remover()
       if self.listen_peer ~= nil and self.listen_peer ~= self.rpc_peer then
          error('Attempt to modify configuration while listener attached')
@@ -676,7 +676,7 @@ function Leader:rpc_remove_config (args)
    if success then return response else return {status=1, error=response} end
 end
 
-function Leader:rpc_attach_listener (args)
+function Manager:rpc_attach_listener (args)
    local function attacher()
       if self.listen_peer ~= nil then error('Listener already attached') end
       self.listen_peer = self.rpc_peer
@@ -686,7 +686,7 @@ function Leader:rpc_attach_listener (args)
    if success then return response else return {status=1, error=response} end
 end
 
-function Leader:rpc_get_state (args)
+function Manager:rpc_get_state (args)
    local function getter()
       if args.schema ~= self.schema_name then
          return self:foreign_rpc_get_state(args.schema, args.path,
@@ -701,7 +701,7 @@ function Leader:rpc_get_state (args)
    if success then return response else return {status=1, error=response} end
 end
 
-function Leader:rpc_get_alarms_state (args)
+function Manager:rpc_get_alarms_state (args)
    local function getter()
       assert(args.schema == "ietf-alarms")
       local printer = path_printer_for_schema_by_name(
@@ -716,13 +716,13 @@ function Leader:rpc_get_alarms_state (args)
    if success then return response else return {status=1, error=response} end
 end
 
-function Leader:handle (payload)
+function Manager:handle (payload)
    return rpc.handle_calls(self.rpc_callee, payload, self.rpc_handler)
 end
 
 local dummy_unix_sockaddr = S.t.sockaddr_un()
 
-function Leader:handle_calls_from_peers()
+function Manager:handle_calls_from_peers()
    local peers = self.peers
    while true do
       local fd, err = self.socket:accept(dummy_unix_sockaddr)
@@ -838,7 +838,7 @@ function Leader:handle_calls_from_peers()
    end
 end
 
-function Leader:send_messages_to_followers()
+function Manager:send_messages_to_followers()
    for _,follower in pairs(self.followers) do
       if not follower.channel then
          local name = '/'..tostring(follower.pid)..'/config-follower-channel'
@@ -860,7 +860,7 @@ function Leader:send_messages_to_followers()
    end
 end
 
-function Leader:pull ()
+function Manager:pull ()
    if app.now() < self.next_time then return end
    self.next_time = app.now() + self.period
    self:remove_stale_followers()
@@ -869,13 +869,13 @@ function Leader:pull ()
    self:receive_alarms_from_followers()
 end
 
-function Leader:receive_alarms_from_followers ()
+function Manager:receive_alarms_from_followers ()
    for _,follower in pairs(self.followers) do
       self:receive_alarms_from_follower(follower)
    end
 end
 
-function Leader:receive_alarms_from_follower (follower)
+function Manager:receive_alarms_from_follower (follower)
    if not follower.alarms_channel then
       local name = '/'..tostring(follower.pid)..'/alarms-follower-channel'
       local success, channel = pcall(channel.open, name)
@@ -892,7 +892,7 @@ function Leader:receive_alarms_from_follower (follower)
    end
 end
 
-function Leader:handle_alarm (follower, alarm)
+function Manager:handle_alarm (follower, alarm)
    local fn, args = unpack(alarm)
    if fn == 'raise_alarm' then
       local key, args = alarm_codec.to_alarm(args)
@@ -912,7 +912,7 @@ function Leader:handle_alarm (follower, alarm)
    end
 end
 
-function Leader:stop ()
+function Manager:stop ()
    for _,peer in ipairs(self.peers) do peer.fd:close() end
    self.peers = {}
    self.socket:close()
@@ -929,7 +929,7 @@ function test_worker()
 end
 
 function selftest ()
-   print('selftest: lib.ptree.leader')
+   print('selftest: lib.ptree.manager')
    local graph = app_graph.new()
    local function setup_fn(cfg)
       local graph = app_graph.new()
@@ -939,17 +939,17 @@ function selftest ()
       app_graph.link(graph, "source.foo -> sink.bar")
       return {graph}
    end
-   local worker_start_code = "require('lib.ptree.leader').test_worker()"
-   app_graph.app(graph, "leader", Leader,
+   local worker_start_code = "require('lib.ptree.manager').test_worker()"
+   app_graph.app(graph, "manager", Manager,
                  {setup_fn=setup_fn, worker_start_code=worker_start_code,
                   -- Use a schema with no data nodes, just for
                   -- testing.
                   schema_name='ietf-inet-types', initial_configuration={}})
    engine.configure(graph)
    engine.main({ duration = 0.05, report = {showapps=true,showlinks=true}})
-   assert(app.app_table.leader.followers[1])
-   assert(app.app_table.leader.followers[1].graph.links)
-   assert(app.app_table.leader.followers[1].graph.links["source.foo -> sink.bar"])
+   assert(app.app_table.manager.followers[1])
+   assert(app.app_table.manager.followers[1].graph.links)
+   assert(app.app_table.manager.followers[1].graph.links["source.foo -> sink.bar"])
    local link = app.link_table["source.foo -> sink.bar"]
    engine.configure(app_graph.new())
    print('selftest: ok')

From 3619ac22a1db4a8ec09bca7f174ba8fac34ad53e Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@igalia.com>
Date: Wed, 6 Dec 2017 15:00:07 +0100
Subject: [PATCH 04/31] Rename lib.ptree.follower to lib.ptree.worker

---
 src/lib/ptree/alarm_codec.lua               |   4 +-
 src/lib/ptree/channel.lua                   |  12 +-
 src/lib/ptree/manager.lua                   | 128 ++++++++++----------
 src/lib/ptree/support.lua                   |  18 +--
 src/lib/ptree/support/snabb-softwire-v2.lua |   6 +-
 src/lib/ptree/{follower.lua => worker.lua}  |  20 +--
 6 files changed, 94 insertions(+), 94 deletions(-)
 rename src/lib/ptree/{follower.lua => worker.lua} (85%)

diff --git a/src/lib/ptree/alarm_codec.lua b/src/lib/ptree/alarm_codec.lua
index 7e3c812751..42ca02c7da 100644
--- a/src/lib/ptree/alarm_codec.lua
+++ b/src/lib/ptree/alarm_codec.lua
@@ -156,12 +156,12 @@ local alarms_channel
 
 function get_channel()
    if alarms_channel then return alarms_channel end
-   local name = '/'..S.getpid()..'/alarms-follower-channel'
+   local name = '/'..S.getpid()..'/alarms-worker-channel'
    local success, value = pcall(channel.open, name)
    if success then
       alarms_channel = value
    else
-      alarms_channel = channel.create('alarms-follower-channel', 1e6)
+      alarms_channel = channel.create('alarms-worker-channel', 1e6)
    end
    return alarms_channel
 end
diff --git a/src/lib/ptree/channel.lua b/src/lib/ptree/channel.lua
index 1818bb7b43..3554c57b06 100644
--- a/src/lib/ptree/channel.lua
+++ b/src/lib/ptree/channel.lua
@@ -3,12 +3,12 @@
 module(...,package.seeall)
 
 -- A channel is a ring buffer used by the manager to send updates to a
--- follower.  Each follower has its own ring buffer and is the only
--- reader to the buffer.  The manager is the only writer to these
--- buffers also.  The ring buffer is just bytes; putting a message onto
--- the buffer will write a header indicating the message size, then the
--- bytes of the message.  The channel ring buffer is mapped into shared
--- memory.  Access to a channel will never block or cause a system call.
+-- worker.  Each worker has its own ring buffer and is the only reader
+-- to the buffer.  The manager is the only writer to these buffers also.
+-- The ring buffer is just bytes; putting a message onto the buffer will
+-- write a header indicating the message size, then the bytes of the
+-- message.  The channel ring buffer is mapped into shared memory.
+-- Access to a channel will never block or cause a system call.
 
 local ffi = require('ffi')
 local S = require("syscall")
diff --git a/src/lib/ptree/manager.lua b/src/lib/ptree/manager.lua
index 8e80906e12..706fbee134 100644
--- a/src/lib/ptree/manager.lua
+++ b/src/lib/ptree/manager.lua
@@ -65,7 +65,7 @@ function Manager:new (conf)
    ret.period = 1/conf.Hz
    ret.next_time = app.now()
    ret.worker_start_code = conf.worker_start_code
-   ret.followers = {}
+   ret.workers = {}
    ret.rpc_callee = rpc.prepare_callee('snabb-config-leader-v1')
    ret.rpc_handler = rpc.dispatch_handler(ret, 'rpc_')
 
@@ -78,7 +78,7 @@ function Manager:set_initial_configuration (configuration)
    self.current_configuration = configuration
    self.current_in_place_dependencies = {}
 
-   -- Start the followers and configure them.
+   -- Start the workers and configure them.
    local worker_app_graphs = self.setup_fn(configuration)
 
    -- Calculate the dependences
@@ -87,9 +87,9 @@ function Manager:set_initial_configuration (configuration)
 	    {}, worker_app_graphs, self.schema_name, 'load',
             '/', self.current_configuration)
 
-   -- Iterate over followers starting the workers and queuing up actions.
+   -- Iterate over workers starting the workers and queuing up actions.
    for id, worker_app_graph in pairs(worker_app_graphs) do
-      self:start_follower_for_graph(id, worker_app_graph)
+      self:start_worker_for_graph(id, worker_app_graph)
    end
 end
 
@@ -99,36 +99,36 @@ function Manager:start_worker(cpu)
       table.insert(start_code, 1, "print('Bound data plane to CPU:',"..cpu..")")
       table.insert(start_code, 1, "require('lib.numa').bind_to_cpu("..cpu..")")
    end
-   return worker.start("follower", table.concat(start_code, "\n"))
+   return worker.start("worker", table.concat(start_code, "\n"))
 end
 
 function Manager:stop_worker(id)
    -- Tell the worker to terminate
    local stop_actions = {{'shutdown', {}}, {'commit', {}}}
-   self:enqueue_config_actions_for_follower(id, stop_actions)
-   self:send_messages_to_followers()
-   self.followers[id].shutting_down = true
+   self:enqueue_config_actions_for_worker(id, stop_actions)
+   self:send_messages_to_workers()
+   self.workers[id].shutting_down = true
 end
 
-function Manager:remove_stale_followers()
+function Manager:remove_stale_workers()
    local stale = {}
-   for id, follower in pairs(self.followers) do
-      if follower.shutting_down then
-	 if S.waitpid(follower.pid, S.c.W["NOHANG"]) ~= 0 then
+   for id, worker in pairs(self.workers) do
+      if worker.shutting_down then
+	 if S.waitpid(worker.pid, S.c.W["NOHANG"]) ~= 0 then
 	    stale[#stale + 1] = id
 	 end
       end
    end
    for _, id in ipairs(stale) do
-      if self.followers[id].cpu then
-	 self.cpuset:release(self.followers[id].cpu)
+      if self.workers[id].cpu then
+	 self.cpuset:release(self.workers[id].cpu)
       end
-      self.followers[id] = nil
+      self.workers[id] = nil
 
    end
 end
 
-function Manager:acquire_cpu_for_follower(id, app_graph)
+function Manager:acquire_cpu_for_worker(id, app_graph)
    local pci_addresses = {}
    -- Grovel through app initargs for keys named "pciaddr".  Hacky!
    for name, init in pairs(app_graph.apps) do
@@ -141,17 +141,17 @@ function Manager:acquire_cpu_for_follower(id, app_graph)
    return self.cpuset:acquire_for_pci_addresses(pci_addresses)
 end
 
-function Manager:start_follower_for_graph(id, graph)
-   local cpu = self:acquire_cpu_for_follower(id, graph)
-   self.followers[id] = { cpu=cpu, pid=self:start_worker(cpu), queue={},
+function Manager:start_worker_for_graph(id, graph)
+   local cpu = self:acquire_cpu_for_worker(id, graph)
+   self.workers[id] = { cpu=cpu, pid=self:start_worker(cpu), queue={},
                           graph=graph }
    local actions = self.support.compute_config_actions(
-      app_graph.new(), self.followers[id].graph, {}, 'load')
-   self:enqueue_config_actions_for_follower(id, actions)
-   return self.followers[id]
+      app_graph.new(), self.workers[id].graph, {}, 'load')
+   self:enqueue_config_actions_for_worker(id, actions)
+   return self.workers[id]
 end
 
-function Manager:take_follower_message_queue ()
+function Manager:take_worker_message_queue ()
    local actions = self.config_action_queue
    self.config_action_queue = nil
    return actions
@@ -159,17 +159,17 @@ end
 
 local verbose = os.getenv('SNABB_MANAGER_VERBOSE') and true
 
-function Manager:enqueue_config_actions_for_follower(follower, actions)
+function Manager:enqueue_config_actions_for_worker(worker, actions)
    for _,action in ipairs(actions) do
       if verbose then print('encode', action[1], unpack(action[2])) end
       local buf, len = action_codec.encode(action)
-      table.insert(self.followers[follower].queue, { buf=buf, len=len })
+      table.insert(self.workers[worker].queue, { buf=buf, len=len })
    end
 end
 
 function Manager:enqueue_config_actions (actions)
-   for id,_ in pairs(self.followers) do
-      self.enqueue_config_actions_for_follower(id, actions)
+   for id,_ in pairs(self.workers) do
+      self.enqueue_config_actions_for_worker(id, actions)
    end
 end
 
@@ -535,19 +535,19 @@ function Manager:update_configuration (update_fn, verb, path, ...)
    local new_config = update_fn(self.current_configuration, ...)
    local new_graphs = self.setup_fn(new_config, ...)
    for id, graph in pairs(new_graphs) do
-      if self.followers[id] == nil then
-	 self:start_follower_for_graph(id, graph)
+      if self.workers[id] == nil then
+	 self:start_worker_for_graph(id, graph)
       end
    end
 
-   for id, follower in pairs(self.followers) do
+   for id, worker in pairs(self.workers) do
       if new_graphs[id] == nil then
          self:stop_worker(id)
       else
 	 local actions = self.support.compute_config_actions(
-	    follower.graph, new_graphs[id], to_restart, verb, path, ...)
-	 self:enqueue_config_actions_for_follower(id, actions)
-	 follower.graph = new_graphs[id]
+	    worker.graph, new_graphs[id], to_restart, verb, path, ...)
+	 self:enqueue_config_actions_for_worker(id, actions)
+	 worker.graph = new_graphs[id]
       end
    end
    self.current_configuration = new_config
@@ -567,10 +567,10 @@ end
 function Manager:get_native_state ()
    local states = {}
    local state_reader = self.support.compute_state_reader(self.schema_name)
-   for _, follower in pairs(self.followers) do
-      local follower_config = self.support.configuration_for_follower(
-         follower, self.current_configuration)
-      table.insert(states, state_reader(follower.pid, follower_config))
+   for _, worker in pairs(self.workers) do
+      local worker_config = self.support.configuration_for_worker(
+         worker, self.current_configuration)
+      table.insert(states, state_reader(worker.pid, worker_config))
    end
    return self.support.process_states(states)
 end
@@ -838,23 +838,23 @@ function Manager:handle_calls_from_peers()
    end
 end
 
-function Manager:send_messages_to_followers()
-   for _,follower in pairs(self.followers) do
-      if not follower.channel then
-         local name = '/'..tostring(follower.pid)..'/config-follower-channel'
+function Manager:send_messages_to_workers()
+   for _,worker in pairs(self.workers) do
+      if not worker.channel then
+         local name = '/'..tostring(worker.pid)..'/config-worker-channel'
          local success, channel = pcall(channel.open, name)
-         if success then follower.channel = channel end
+         if success then worker.channel = channel end
       end
-      local channel = follower.channel
+      local channel = worker.channel
       if channel then
-         local queue = follower.queue
-         follower.queue = {}
+         local queue = worker.queue
+         worker.queue = {}
          local requeue = false
          for _,msg in ipairs(queue) do
             if not requeue then
                requeue = not channel:put_message(msg.buf, msg.len)
             end
-            if requeue then table.insert(follower.queue, msg) end
+            if requeue then table.insert(worker.queue, msg) end
          end
       end
    end
@@ -863,36 +863,36 @@ end
 function Manager:pull ()
    if app.now() < self.next_time then return end
    self.next_time = app.now() + self.period
-   self:remove_stale_followers()
+   self:remove_stale_workers()
    self:handle_calls_from_peers()
-   self:send_messages_to_followers()
-   self:receive_alarms_from_followers()
+   self:send_messages_to_workers()
+   self:receive_alarms_from_workers()
 end
 
-function Manager:receive_alarms_from_followers ()
-   for _,follower in pairs(self.followers) do
-      self:receive_alarms_from_follower(follower)
+function Manager:receive_alarms_from_workers ()
+   for _,worker in pairs(self.workers) do
+      self:receive_alarms_from_worker(worker)
    end
 end
 
-function Manager:receive_alarms_from_follower (follower)
-   if not follower.alarms_channel then
-      local name = '/'..tostring(follower.pid)..'/alarms-follower-channel'
+function Manager:receive_alarms_from_worker (worker)
+   if not worker.alarms_channel then
+      local name = '/'..tostring(worker.pid)..'/alarms-worker-channel'
       local success, channel = pcall(channel.open, name)
       if not success then return end
-      follower.alarms_channel = channel
+      worker.alarms_channel = channel
    end
-   local channel = follower.alarms_channel
+   local channel = worker.alarms_channel
    while true do
       local buf, len = channel:peek_message()
       if not buf then break end
       local alarm = alarm_codec.decode(buf, len)
-      self:handle_alarm(follower, alarm)
+      self:handle_alarm(worker, alarm)
       channel:discard_message(len)
    end
 end
 
-function Manager:handle_alarm (follower, alarm)
+function Manager:handle_alarm (worker, alarm)
    local fn, args = unpack(alarm)
    if fn == 'raise_alarm' then
       local key, args = alarm_codec.to_alarm(args)
@@ -920,9 +920,9 @@ function Manager:stop ()
 end
 
 function test_worker()
-   local follower = require("lib.ptree.follower")
+   local worker = require("lib.ptree.worker")
    local myconf = config.new()
-   config.app(myconf, "follower", follower.Follower, {})
+   config.app(myconf, "worker", worker.Worker, {})
    app.configure(myconf)
    app.busywait = true
    app.main({})
@@ -947,9 +947,9 @@ function selftest ()
                   schema_name='ietf-inet-types', initial_configuration={}})
    engine.configure(graph)
    engine.main({ duration = 0.05, report = {showapps=true,showlinks=true}})
-   assert(app.app_table.manager.followers[1])
-   assert(app.app_table.manager.followers[1].graph.links)
-   assert(app.app_table.manager.followers[1].graph.links["source.foo -> sink.bar"])
+   assert(app.app_table.manager.workers[1])
+   assert(app.app_table.manager.workers[1].graph.links)
+   assert(app.app_table.manager.workers[1].graph.links["source.foo -> sink.bar"])
    local link = app.link_table["source.foo -> sink.bar"]
    engine.configure(app_graph.new())
    print('selftest: ok')
diff --git a/src/lib/ptree/support.lua b/src/lib/ptree/support.lua
index d11a6dae21..a1f45ad523 100644
--- a/src/lib/ptree/support.lua
+++ b/src/lib/ptree/support.lua
@@ -98,17 +98,17 @@ local function compute_objects_maybe_updated_in_place (schema_name, config,
    return objs
 end
 
-local function record_mutable_objects_embedded_in_app_initarg (follower_id, app_name, obj, accum)
+local function record_mutable_objects_embedded_in_app_initarg (worker_id, app_name, obj, accum)
    local function record(obj)
       local tab = accum[obj]
       if not tab then
          tab = {}
          accum[obj] = tab
       end
-      if tab[follower_id] == nil then
-         tab[follower_id] = {app_name}
+      if tab[worker_id] == nil then
+         tab[worker_id] = {app_name}
       else
-         table.insert(tab[follower_id], app_name)
+         table.insert(tab[worker_id], app_name)
       end
    end
    local function visit(obj)
@@ -126,9 +126,9 @@ local function record_mutable_objects_embedded_in_app_initarg (follower_id, app_
    visit(obj)
 end
 
--- Takes a table of follower ids (app_graph_map) and returns a tabl≈e which has
--- the follower id as the key and a table listing all app names
---   i.e. {follower_id => {app name, ...}, ...}
+-- Takes a table of worker ids (app_graph_map) and returns a tabl≈e which has
+-- the worker id as the key and a table listing all app names
+--   i.e. {worker_id => {app name, ...}, ...}
 local function compute_mutable_objects_embedded_in_app_initargs (app_graph_map)
    local deps = {}
    for id, app_graph in pairs(app_graph_map) do
@@ -190,7 +190,7 @@ local function add_restarts(actions, app_graph, to_restart)
    return actions
 end
 
-local function configuration_for_follower(follower, configuration)
+local function configuration_for_worker(worker, configuration)
    return configuration
 end
 
@@ -216,7 +216,7 @@ generic_schema_config_support = {
       return compute_mutable_objects_embedded_in_app_initargs(app_graph)
    end,
    compute_state_reader = compute_state_reader,
-   configuration_for_follower = configuration_for_follower,
+   configuration_for_worker = configuration_for_worker,
    process_states = process_states,
    compute_apps_to_restart_after_configuration_update =
       compute_apps_to_restart_after_configuration_update,
diff --git a/src/lib/ptree/support/snabb-softwire-v2.lua b/src/lib/ptree/support/snabb-softwire-v2.lua
index 274fd80c53..7f83d788a4 100644
--- a/src/lib/ptree/support/snabb-softwire-v2.lua
+++ b/src/lib/ptree/support/snabb-softwire-v2.lua
@@ -630,8 +630,8 @@ local function ietf_softwire_br_translator ()
    return ret
 end
 
-local function configuration_for_follower(follower, configuration)
-   return follower.graph.apps.lwaftr.arg
+local function configuration_for_worker(worker, configuration)
+   return worker.graph.apps.lwaftr.arg
 end
 
 local function compute_state_reader(schema_name)
@@ -700,7 +700,7 @@ function get_config_support()
          compute_apps_to_restart_after_configuration_update,
       compute_state_reader = compute_state_reader,
       process_states = process_states,
-      configuration_for_follower = configuration_for_follower,
+      configuration_for_worker = configuration_for_worker,
       translators = { ['ietf-softwire-br'] = ietf_softwire_br_translator () }
    }
 end
diff --git a/src/lib/ptree/follower.lua b/src/lib/ptree/worker.lua
similarity index 85%
rename from src/lib/ptree/follower.lua
rename to src/lib/ptree/worker.lua
index e21c9a2a00..25466431b0 100644
--- a/src/lib/ptree/follower.lua
+++ b/src/lib/ptree/worker.lua
@@ -13,23 +13,23 @@ local channel = require("lib.ptree.channel")
 local action_codec = require("lib.ptree.action_codec")
 local alarm_codec = require("lib.ptree.alarm_codec")
 
-Follower = {
+Worker = {
    config = {
       Hz = {default=1000},
    }
 }
 
-function Follower:new (conf)
-   local ret = setmetatable({}, {__index=Follower})
+function Worker:new (conf)
+   local ret = setmetatable({}, {__index=Worker})
    ret.period = 1/conf.Hz
    ret.next_time = app.now()
-   ret.channel = channel.create('config-follower-channel', 1e6)
+   ret.channel = channel.create('config-worker-channel', 1e6)
    ret.alarms_channel = alarm_codec.get_channel()
    ret.pending_actions = {}
    return ret
 end
 
-function Follower:shutdown()
+function Worker:shutdown()
    -- This will shutdown everything.
    engine.configure(app_graph.new())
 
@@ -37,7 +37,7 @@ function Follower:shutdown()
    S.exit(0)
 end
 
-function Follower:commit_pending_actions()
+function Worker:commit_pending_actions()
    local to_apply = {}
    local should_flush = false
    for _,action in ipairs(self.pending_actions) do
@@ -64,7 +64,7 @@ function Follower:commit_pending_actions()
    if should_flush then require('jit').flush() end
 end
 
-function Follower:handle_actions_from_manager()
+function Worker:handle_actions_from_manager()
    local channel = self.channel
    for i=1,4 do
       local buf, len = channel:peek_message()
@@ -79,16 +79,16 @@ function Follower:handle_actions_from_manager()
    end
 end
 
-function Follower:pull ()
+function Worker:pull ()
    if app.now() < self.next_time then return end
    self.next_time = app.now() + self.period
    self:handle_actions_from_manager()
 end
 
 function selftest ()
-   print('selftest: lib.ptree.follower')
+   print('selftest: lib.ptree.worker')
    local c = config.new()
-   config.app(c, "follower", Follower, {})
+   config.app(c, "worker", Worker, {})
    engine.configure(c)
    engine.main({ duration = 0.0001, report = {showapps=true,showlinks=true}})
    engine.configure(config.new())

From 90b7c2fa7ced22b6d67fb76e3c27b71e99ff6507 Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@igalia.com>
Date: Wed, 6 Dec 2017 15:47:51 +0100
Subject: [PATCH 05/31] Add lib.scheduling

New file applies scheduling parameters for a data plane.
---
 src/lib/scheduling.lua | 105 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 105 insertions(+)
 create mode 100644 src/lib/scheduling.lua

diff --git a/src/lib/scheduling.lua b/src/lib/scheduling.lua
new file mode 100644
index 0000000000..66b43638b8
--- /dev/null
+++ b/src/lib/scheduling.lua
@@ -0,0 +1,105 @@
+-- Use of this source code is governed by the Apache 2.0 license; see COPYING.
+
+module(..., package.seeall)
+
+local S = require("syscall")
+local lib = require("core.lib")
+local numa = require("lib.numa")
+local ingress_drop_monitor = require("lib.timers.ingress_drop_monitor")
+
+local function fatal (msg)
+   print(msg)
+   main.exit(1)
+end
+
+local scheduling_opts = {
+   cpu = {},                  -- CPU index (integer).
+   real_time = {},            -- Boolean.
+   ingress_drop_monitor = {}, -- Action string: one of 'flush' or 'warn'.
+   busywait = {},             -- Boolean.
+   j = {},                    -- Profiling argument string, e.g. "p" or "v".
+   eval = {}                  -- String.
+}
+
+local sched_apply = {}
+
+function sched_apply.cpu (cpu)
+   print(string.format('Binding data plane PID %s to CPU %s.',
+                       tonumber(S.getpid()), cpu))
+   numa.bind_to_cpu(cpu)
+end
+
+function sched_apply.ingress_drop_monitor (action)
+   timer.activate(ingress_drop_monitor.new({action=action}):timer())
+end
+
+function sched_apply.real_time (real_time)
+   if real_time and not S.sched_setscheduler(0, "fifo", 1) then
+      fatal('Failed to enable real-time scheduling.  Try running as root.')
+   end
+end
+
+function sched_apply.j (arg)
+   if arg:match("^v") then
+      local file = arg:match("^v=(.*)")
+      if file == '' then file = nil end
+      require("jit.v").start(file)
+   elseif arg:match("^p") then
+      local opts, file = arg:match("^p=([^,]*),?(.*)")
+      if file == '' then file = nil end
+      local prof = require('jit.p')
+      prof.start(opts, file)
+      local function report() prof.stop(); prof.start(opts, file) end
+      timer.activate(timer.new('p', report, 10e9, 'repeating'))
+   elseif arg:match("^dump") then
+      local opts, file = arg:match("^dump=([^,]*),?(.*)")
+      if file == '' then file = nil end
+      require("jit.dump").on(opts, file)
+   elseif arg:match("^tprof") then
+      local prof = require('lib.traceprof.traceprof')
+      prof.start()
+      local function report() prof.stop(); prof.start() end
+      timer.activate(timer.new('tprof', report, 10e9, 'repeating'))
+   end
+end
+
+function sched_apply.busywait (busywait)
+   engine.busywait = busywait
+end
+
+function sched_apply.eval (str)
+   loadstring(str)()
+end
+
+function apply (opts)
+   opts = lib.parse(opts, scheduling_opts)
+   for k, v in pairs(opts) do sched_apply[k](v) end
+end
+
+local function stringify (x)
+   if type(x) == 'string' then return string.format('%q', x) end
+   if type(x) == 'number' then return tostring(x) end
+   if type(x) == 'boolean' then return x and 'true' or 'false' end
+   assert(type(x) == 'table')
+   local ret = {"{"}
+   local first = true
+   for k,v in pairs(x) do
+      if first then first = false else table.insert(ret, ",") end
+      table.insert(ret, string.format('[%s]=%s', stringify(k), stringify(v)))
+   end
+   table.insert(ret, "}")
+   return table.concat(ret)
+end
+
+function stage (opts)
+   return string.format("require('lib.scheduling').apply(%s)",
+                        stringify(lib.parse(opts, scheduling_opts)))
+end
+
+function selftest ()
+   print('selftest: lib.scheduling')
+   loadstring(stage({}))()
+   loadstring(stage({busywait=true}))()
+   loadstring(stage({eval='print("lib.scheduling: eval test")'}))()
+   print('selftest: ok')
+end

From 67e44e8f9b9ff074d11a41e637b8bfd1ab495d92 Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@igalia.com>
Date: Wed, 6 Dec 2017 16:22:32 +0100
Subject: [PATCH 06/31] Rework ptree manager to handle worker scheduling

Also add generic worker.main() export from lib.ptree.worker
---
 src/lib/ptree/manager.lua | 45 ++++++++++++++++++++++-----------------
 src/lib/ptree/worker.lua  | 12 +++++++++++
 2 files changed, 38 insertions(+), 19 deletions(-)

diff --git a/src/lib/ptree/manager.lua b/src/lib/ptree/manager.lua
index 706fbee134..0cc2f9eb22 100644
--- a/src/lib/ptree/manager.lua
+++ b/src/lib/ptree/manager.lua
@@ -7,6 +7,7 @@ local ffi = require("ffi")
 local lib = require("core.lib")
 local cltable = require("lib.cltable")
 local cpuset = require("lib.cpuset")
+local scheduling = require("lib.scheduling")
 local yang = require("lib.yang.yang")
 local data = require("lib.yang.data")
 local util = require("lib.yang.util")
@@ -31,7 +32,7 @@ Manager = {
       -- Could relax this requirement.
       initial_configuration = {required=true},
       schema_name = {required=true},
-      worker_start_code = {required=true},
+      worker_default_scheduling = {default={busywait=true}},
       default_schema = {},
       cpuset = {default=cpuset.global_cpuset()},
       Hz = {default=100},
@@ -64,7 +65,7 @@ function Manager:new (conf)
    ret.setup_fn = conf.setup_fn
    ret.period = 1/conf.Hz
    ret.next_time = app.now()
-   ret.worker_start_code = conf.worker_start_code
+   ret.worker_default_scheduling = conf.worker_default_scheduling
    ret.workers = {}
    ret.rpc_callee = rpc.prepare_callee('snabb-config-leader-v1')
    ret.rpc_handler = rpc.dispatch_handler(ret, 'rpc_')
@@ -93,13 +94,12 @@ function Manager:set_initial_configuration (configuration)
    end
 end
 
-function Manager:start_worker(cpu)
-   local start_code = { self.worker_start_code }
-   if cpu then
-      table.insert(start_code, 1, "print('Bound data plane to CPU:',"..cpu..")")
-      table.insert(start_code, 1, "require('lib.numa').bind_to_cpu("..cpu..")")
-   end
-   return worker.start("worker", table.concat(start_code, "\n"))
+function Manager:start_worker(sched_opts)
+   local code = {
+      scheduling.stage(sched_opts),
+      "require('lib.ptree.worker').main()"
+   }
+   return worker.start("worker", table.concat(code, "\n"))
 end
 
 function Manager:stop_worker(id)
@@ -120,8 +120,8 @@ function Manager:remove_stale_workers()
       end
    end
    for _, id in ipairs(stale) do
-      if self.workers[id].cpu then
-	 self.cpuset:release(self.workers[id].cpu)
+      if self.workers[id].scheduling.cpu then
+	 self.cpuset:release(self.workers[id].scheduling.cpu)
       end
       self.workers[id] = nil
 
@@ -141,10 +141,18 @@ function Manager:acquire_cpu_for_worker(id, app_graph)
    return self.cpuset:acquire_for_pci_addresses(pci_addresses)
 end
 
+function Manager:compute_scheduling_for_worker(id, app_graph)
+   local ret = {}
+   for k, v in pairs(self.worker_default_scheduling) do ret[k] = v end
+   ret.cpu = self:acquire_cpu_for_worker(id, app_graph)
+   return ret
+end
+
 function Manager:start_worker_for_graph(id, graph)
-   local cpu = self:acquire_cpu_for_worker(id, graph)
-   self.workers[id] = { cpu=cpu, pid=self:start_worker(cpu), queue={},
-                          graph=graph }
+   local scheduling = self:compute_scheduling_for_worker(id, graph)
+   self.workers[id] = { scheduling=scheduling,
+                        pid=self:start_worker(scheduling),
+                        queue={}, graph=graph }
    local actions = self.support.compute_config_actions(
       app_graph.new(), self.workers[id].graph, {}, 'load')
    self:enqueue_config_actions_for_worker(id, actions)
@@ -939,12 +947,11 @@ function selftest ()
       app_graph.link(graph, "source.foo -> sink.bar")
       return {graph}
    end
-   local worker_start_code = "require('lib.ptree.manager').test_worker()"
    app_graph.app(graph, "manager", Manager,
-                 {setup_fn=setup_fn, worker_start_code=worker_start_code,
-                  -- Use a schema with no data nodes, just for
-                  -- testing.
-                  schema_name='ietf-inet-types', initial_configuration={}})
+                 {setup_fn=setup_fn,
+                  -- Use a schema with no data nodes, just for testing.
+                  schema_name='ietf-inet-types',
+                  initial_configuration={}})
    engine.configure(graph)
    engine.main({ duration = 0.05, report = {showapps=true,showlinks=true}})
    assert(app.app_table.manager.workers[1])
diff --git a/src/lib/ptree/worker.lua b/src/lib/ptree/worker.lua
index 25466431b0..8cf959b366 100644
--- a/src/lib/ptree/worker.lua
+++ b/src/lib/ptree/worker.lua
@@ -85,6 +85,18 @@ function Worker:pull ()
    self:handle_actions_from_manager()
 end
 
+function main (opts)
+   local app_graph = require('core.config')
+   local engine = require('core.app')
+
+   if opts == nil then opts = {} end
+
+   local graph = app_graph.new()
+   app_graph.app(graph, "worker", Worker, {})
+   engine.configure(graph)
+   engine.main(opts)
+end
+
 function selftest ()
    print('selftest: lib.ptree.worker')
    local c = config.new()

From aaeac1bae4265a6f9181d4d3657caf4c39b31c57 Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@igalia.com>
Date: Wed, 6 Dec 2017 16:35:50 +0100
Subject: [PATCH 07/31] Make worker app private

* src/lib/ptree/worker.lua: Privatize "Worker".
---
 src/lib/ptree/worker.lua | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/lib/ptree/worker.lua b/src/lib/ptree/worker.lua
index 8cf959b366..b0bb4c98bb 100644
--- a/src/lib/ptree/worker.lua
+++ b/src/lib/ptree/worker.lua
@@ -13,7 +13,7 @@ local channel = require("lib.ptree.channel")
 local action_codec = require("lib.ptree.action_codec")
 local alarm_codec = require("lib.ptree.alarm_codec")
 
-Worker = {
+local Worker = {
    config = {
       Hz = {default=1000},
    }

From 49651f81a9adc2b7e1df79bba49f284ef89a6e30 Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@igalia.com>
Date: Wed, 6 Dec 2017 17:36:59 +0100
Subject: [PATCH 08/31] Rework ptree worker to be library, not app

---
 src/lib/ptree/worker.lua | 82 ++++++++++++++++++++++------------------
 1 file changed, 46 insertions(+), 36 deletions(-)

diff --git a/src/lib/ptree/worker.lua b/src/lib/ptree/worker.lua
index b0bb4c98bb..0c69c9662b 100644
--- a/src/lib/ptree/worker.lua
+++ b/src/lib/ptree/worker.lua
@@ -2,30 +2,43 @@
 
 module(...,package.seeall)
 
-local S = require("syscall")
-local ffi = require("ffi")
-local yang = require("lib.yang.yang")
-local rpc = require("lib.yang.rpc")
-local app = require("core.app")
-local shm = require("core.shm")
-local app_graph = require("core.config")
-local channel = require("lib.ptree.channel")
+local S            = require("syscall")
+local engine       = require("core.app")
+local app_graph    = require("core.config")
+local counter      = require("core.counter")
+local histogram    = require('core.histogram')
+local lib          = require('core.lib')
+local timer        = require('core.timer')
+local channel      = require("lib.ptree.channel")
 local action_codec = require("lib.ptree.action_codec")
-local alarm_codec = require("lib.ptree.alarm_codec")
+local alarm_codec  = require("lib.ptree.alarm_codec")
 
-local Worker = {
-   config = {
-      Hz = {default=1000},
-   }
+local Worker = {}
+
+local worker_config_spec = {
+   duration = {},
+   measure_latency = {default=true},
+   no_report = {default=false},
+   report = {default={showapps=true,showlinks=true}},
+   Hz = {default=1000},
 }
 
-function Worker:new (conf)
+function new_worker (conf)
+   local conf = lib.parse(conf, worker_config_spec)
    local ret = setmetatable({}, {__index=Worker})
    ret.period = 1/conf.Hz
-   ret.next_time = app.now()
+   ret.duration = conf.duration or 1/0
+   ret.no_report = conf.no_report
+   ret.next_time = engine.now()
    ret.channel = channel.create('config-worker-channel', 1e6)
    ret.alarms_channel = alarm_codec.get_channel()
    ret.pending_actions = {}
+
+   ret.breathe = engine.breathe
+   if conf.measure_latency then
+      local latency = histogram.create('engine/latency.histogram', 1e-6, 1e0)
+      ret.breathe = latency:wrap_thunk(ret.breathe, engine.now)
+   end
    return ret
 end
 
@@ -44,11 +57,11 @@ function Worker:commit_pending_actions()
       local name, args = unpack(action)
       if name == 'call_app_method_with_blob' then
          if #to_apply > 0 then
-            app.apply_config_actions(to_apply)
+            engine.apply_config_actions(to_apply)
             to_apply = {}
          end
          local callee, method, blob = unpack(args)
-         local obj = assert(app.app_table[callee])
+         local obj = assert(engine.app_table[callee])
          assert(obj[method])(obj, blob)
       elseif name == "shutdown" then
          self:shutdown()
@@ -59,7 +72,7 @@ function Worker:commit_pending_actions()
          table.insert(to_apply, action)
       end
    end
-   if #to_apply > 0 then app.apply_config_actions(to_apply) end
+   if #to_apply > 0 then engine.apply_config_actions(to_apply) end
    self.pending_actions = {}
    if should_flush then require('jit').flush() end
 end
@@ -79,30 +92,27 @@ function Worker:handle_actions_from_manager()
    end
 end
 
-function Worker:pull ()
-   if app.now() < self.next_time then return end
-   self.next_time = app.now() + self.period
-   self:handle_actions_from_manager()
+function Worker:main ()
+   local stop = engine.now() + self.duration
+   repeat
+      self.breathe()
+      if self.next_time < engine.now() then
+         self.next_time = engine.now() + self.period
+         self:handle_actions_from_manager()
+         timer.run()
+      end
+      if not engine.busywait then engine.pace_breathing() end
+   until stop < engine.now()
+   counter.commit()
+   if not self.no_report then engine.report(self.report) end
 end
 
 function main (opts)
-   local app_graph = require('core.config')
-   local engine = require('core.app')
-
-   if opts == nil then opts = {} end
-
-   local graph = app_graph.new()
-   app_graph.app(graph, "worker", Worker, {})
-   engine.configure(graph)
-   engine.main(opts)
+   return new_worker(opts):main()
 end
 
 function selftest ()
    print('selftest: lib.ptree.worker')
-   local c = config.new()
-   config.app(c, "worker", Worker, {})
-   engine.configure(c)
-   engine.main({ duration = 0.0001, report = {showapps=true,showlinks=true}})
-   engine.configure(config.new())
+   main({duration=0.0001})
    print('selftest: ok')
 end

From 7473408099c07679058cbf26ea1d9c0205b9e529 Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@igalia.com>
Date: Thu, 7 Dec 2017 10:18:13 +0100
Subject: [PATCH 09/31] Refactor ptree manager to be lib, not app

---
 src/lib/ptree/manager.lua | 92 ++++++++++++++++++++-------------------
 1 file changed, 48 insertions(+), 44 deletions(-)

diff --git a/src/lib/ptree/manager.lua b/src/lib/ptree/manager.lua
index 0cc2f9eb22..d2551c3401 100644
--- a/src/lib/ptree/manager.lua
+++ b/src/lib/ptree/manager.lua
@@ -4,6 +4,7 @@ module(...,package.seeall)
 
 local S = require("syscall")
 local ffi = require("ffi")
+local C = ffi.C
 local lib = require("core.lib")
 local cltable = require("lib.cltable")
 local cpuset = require("lib.cpuset")
@@ -15,7 +16,6 @@ local schema = require("lib.yang.schema")
 local rpc = require("lib.yang.rpc")
 local state = require("lib.yang.state")
 local path_mod = require("lib.yang.path")
-local app = require("core.app")
 local shm = require("core.shm")
 local worker = require("core.worker")
 local app_graph = require("core.config")
@@ -25,18 +25,18 @@ local support = require("lib.ptree.support")
 local channel = require("lib.ptree.channel")
 local alarms = require("lib.yang.alarms")
 
-Manager = {
-   config = {
-      socket_file_name = {default='config-manager-socket'},
-      setup_fn = {required=true},
-      -- Could relax this requirement.
-      initial_configuration = {required=true},
-      schema_name = {required=true},
-      worker_default_scheduling = {default={busywait=true}},
-      default_schema = {},
-      cpuset = {default=cpuset.global_cpuset()},
-      Hz = {default=100},
-   }
+local Manager = {}
+
+local manager_config_spec = {
+   socket_file_name = {default='config-manager-socket'},
+   setup_fn = {required=true},
+   -- Could relax this requirement.
+   initial_configuration = {required=true},
+   schema_name = {required=true},
+   worker_default_scheduling = {default={busywait=true}},
+   default_schema = {},
+   cpuset = {default=cpuset.global_cpuset()},
+   Hz = {default=100},
 }
 
 local function open_socket (file)
@@ -49,7 +49,8 @@ local function open_socket (file)
    return socket
 end
 
-function Manager:new (conf)
+function new_manager (conf)
+   local conf = lib.parse(conf, manager_config_spec)
    local ret = setmetatable({}, {__index=Manager})
    ret.cpuset = conf.cpuset
    ret.socket_file_name = conf.socket_file_name
@@ -64,7 +65,6 @@ function Manager:new (conf)
    ret.peers = {}
    ret.setup_fn = conf.setup_fn
    ret.period = 1/conf.Hz
-   ret.next_time = app.now()
    ret.worker_default_scheduling = conf.worker_default_scheduling
    ret.workers = {}
    ret.rpc_callee = rpc.prepare_callee('snabb-config-leader-v1')
@@ -868,15 +868,6 @@ function Manager:send_messages_to_workers()
    end
 end
 
-function Manager:pull ()
-   if app.now() < self.next_time then return end
-   self.next_time = app.now() + self.period
-   self:remove_stale_workers()
-   self:handle_calls_from_peers()
-   self:send_messages_to_workers()
-   self:receive_alarms_from_workers()
-end
-
 function Manager:receive_alarms_from_workers ()
    for _,worker in pairs(self.workers) do
       self:receive_alarms_from_worker(worker)
@@ -927,18 +918,31 @@ function Manager:stop ()
    S.unlink(self.socket_file_name)
 end
 
-function test_worker()
-   local worker = require("lib.ptree.worker")
-   local myconf = config.new()
-   config.app(myconf, "worker", worker.Worker, {})
-   app.configure(myconf)
-   app.busywait = true
-   app.main({})
+function Manager:main (duration)
+   local now = C.get_monotonic_time()
+   local stop = now + (duration or 1/0)
+   while stop < now do
+      next_time = now + self.period
+      self:remove_stale_workers()
+      self:handle_calls_from_peers()
+      self:send_messages_to_workers()
+      self:receive_alarms_from_workers()
+      now = C.get_monotonic_time()
+      if now < next_time then
+         C.usleep(math.floor((next_time - now) * 1e6))
+         now = C.get_monotonic_time()
+      end
+   end
+end
+
+function main (opts, duration)
+   local m = new_manager(opts)
+   m:main(duration)
+   m:stop()
 end
 
 function selftest ()
    print('selftest: lib.ptree.manager')
-   local graph = app_graph.new()
    local function setup_fn(cfg)
       local graph = app_graph.new()
       local basic_apps = require('apps.basic.basic_apps')
@@ -947,17 +951,17 @@ function selftest ()
       app_graph.link(graph, "source.foo -> sink.bar")
       return {graph}
    end
-   app_graph.app(graph, "manager", Manager,
-                 {setup_fn=setup_fn,
-                  -- Use a schema with no data nodes, just for testing.
-                  schema_name='ietf-inet-types',
-                  initial_configuration={}})
-   engine.configure(graph)
-   engine.main({ duration = 0.05, report = {showapps=true,showlinks=true}})
-   assert(app.app_table.manager.workers[1])
-   assert(app.app_table.manager.workers[1].graph.links)
-   assert(app.app_table.manager.workers[1].graph.links["source.foo -> sink.bar"])
-   local link = app.link_table["source.foo -> sink.bar"]
-   engine.configure(app_graph.new())
+   local m = new_manager({setup_fn=setup_fn,
+                          -- Use a schema with no data nodes, just for
+                          -- testing.
+                          schema_name='ietf-inet-types',
+                          initial_configuration={}})
+   m:main(0.05)
+   assert(m.workers[1])
+   assert(m.workers[1].graph.links)
+   assert(m.workers[1].graph.links["source.foo -> sink.bar"])
+   m:stop()
+   -- FIXME: Actually stop the workers.
+   -- assert(m.workers[1] == nil)
    print('selftest: ok')
 end

From d49eb7dedb7e40d699871effd26a4ec393fe18a2 Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@igalia.com>
Date: Thu, 7 Dec 2017 10:41:37 +0100
Subject: [PATCH 10/31] Ptree manager stop() method shuts down workers too

---
 src/lib/ptree/manager.lua | 27 +++++++++++++++++++++++++--
 1 file changed, 25 insertions(+), 2 deletions(-)

diff --git a/src/lib/ptree/manager.lua b/src/lib/ptree/manager.lua
index d2551c3401..390ba8a337 100644
--- a/src/lib/ptree/manager.lua
+++ b/src/lib/ptree/manager.lua
@@ -916,6 +916,30 @@ function Manager:stop ()
    self.peers = {}
    self.socket:close()
    S.unlink(self.socket_file_name)
+
+   for id, worker in pairs(self.workers) do
+      if not worker.shutting_down then
+         print(string.format('Asking worker %s to shut down.', id))
+         self:stop_worker(id)
+      end
+   end
+   -- Wait 250ms for workers to shut down nicely, polling every 5ms.
+   local start = C.get_monotonic_time()
+   local wait = 0.25
+   while C.get_monotonic_time() < start + wait do
+      self:remove_stale_workers()
+      if not next(self.workers) then break end
+      C.usleep(5000)
+   end
+   -- If that didn't work, send SIGKILL and wait indefinitely.
+   for id, worker in pairs(self.workers) do
+      print(string.format('Forcing worker %s to shut down.', id))
+      S.kill(worker.pid, "KILL")
+   end
+   while next(self.workers) do
+      self:remove_stale_workers()
+      C.usleep(5000)
+   end
 end
 
 function Manager:main (duration)
@@ -961,7 +985,6 @@ function selftest ()
    assert(m.workers[1].graph.links)
    assert(m.workers[1].graph.links["source.foo -> sink.bar"])
    m:stop()
-   -- FIXME: Actually stop the workers.
-   -- assert(m.workers[1] == nil)
+   assert(m.workers[1] == nil)
    print('selftest: ok')
 end

From 356d34e38eaeec3659749413431516b9d96ca2d6 Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@igalia.com>
Date: Thu, 7 Dec 2017 11:29:00 +0100
Subject: [PATCH 11/31] Ptree worker refactors

Store next_time locally in the main() function, and in the selftest, run
the main() function for longer than a single tick.
---
 src/lib/ptree/worker.lua | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/src/lib/ptree/worker.lua b/src/lib/ptree/worker.lua
index 0c69c9662b..86452808a1 100644
--- a/src/lib/ptree/worker.lua
+++ b/src/lib/ptree/worker.lua
@@ -29,7 +29,6 @@ function new_worker (conf)
    ret.period = 1/conf.Hz
    ret.duration = conf.duration or 1/0
    ret.no_report = conf.no_report
-   ret.next_time = engine.now()
    ret.channel = channel.create('config-worker-channel', 1e6)
    ret.alarms_channel = alarm_codec.get_channel()
    ret.pending_actions = {}
@@ -94,10 +93,11 @@ end
 
 function Worker:main ()
    local stop = engine.now() + self.duration
+   local next_time = engine.now()
    repeat
       self.breathe()
-      if self.next_time < engine.now() then
-         self.next_time = engine.now() + self.period
+      if next_time < engine.now() then
+         next_time = engine.now() + self.period
          self:handle_actions_from_manager()
          timer.run()
       end
@@ -113,6 +113,6 @@ end
 
 function selftest ()
    print('selftest: lib.ptree.worker')
-   main({duration=0.0001})
+   main({duration=0.005})
    print('selftest: ok')
 end

From 94907573be3301b9f306576af11771d4289dbee6 Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@igalia.com>
Date: Thu, 7 Dec 2017 11:30:06 +0100
Subject: [PATCH 12/31] Add ptree manager log facility; fix manager main
 function

---
 src/lib/ptree/manager.lua | 56 +++++++++++++++++++++++++++------------
 1 file changed, 39 insertions(+), 17 deletions(-)

diff --git a/src/lib/ptree/manager.lua b/src/lib/ptree/manager.lua
index 390ba8a337..e9b0493717 100644
--- a/src/lib/ptree/manager.lua
+++ b/src/lib/ptree/manager.lua
@@ -27,6 +27,10 @@ local alarms = require("lib.yang.alarms")
 
 local Manager = {}
 
+local log_levels = { DEBUG=1, INFO=2, WARN=3 }
+local default_log_level = "WARN"
+if os.getenv('SNABB_MANAGER_VERBOSE') then default_log_level = "DEBUG" end
+
 local manager_config_spec = {
    socket_file_name = {default='config-manager-socket'},
    setup_fn = {required=true},
@@ -35,6 +39,7 @@ local manager_config_spec = {
    schema_name = {required=true},
    worker_default_scheduling = {default={busywait=true}},
    default_schema = {},
+   log_level = {default=default_log_level},
    cpuset = {default=cpuset.global_cpuset()},
    Hz = {default=100},
 }
@@ -52,6 +57,7 @@ end
 function new_manager (conf)
    local conf = lib.parse(conf, manager_config_spec)
    local ret = setmetatable({}, {__index=Manager})
+   ret.log_level = assert(log_levels[conf.log_level])
    ret.cpuset = conf.cpuset
    ret.socket_file_name = conf.socket_file_name
    if not ret.socket_file_name:match('^/') then
@@ -75,6 +81,17 @@ function new_manager (conf)
    return ret
 end
 
+function Manager:log (level, fmt, ...)
+   if log_levels[level] < self.log_level then return end
+   local prefix = os.date("%F %H:%M:%S")..": "..level..': '
+   io.stderr:write(prefix..fmt:format(...)..'\n')
+   io.stderr:flush()
+end
+
+function Manager:debug(fmt, ...) self:log("DEBUG", fmt, ...) end
+function Manager:info(fmt, ...) self:log("INFO", fmt, ...) end
+function Manager:warn(fmt, ...) self:log("WARN", fmt, ...) end
+
 function Manager:set_initial_configuration (configuration)
    self.current_configuration = configuration
    self.current_in_place_dependencies = {}
@@ -103,7 +120,7 @@ function Manager:start_worker(sched_opts)
 end
 
 function Manager:stop_worker(id)
-   -- Tell the worker to terminate
+   self:info('Asking worker %s to shut down.', id)
    local stop_actions = {{'shutdown', {}}, {'commit', {}}}
    self:enqueue_config_actions_for_worker(id, stop_actions)
    self:send_messages_to_workers()
@@ -150,9 +167,11 @@ end
 
 function Manager:start_worker_for_graph(id, graph)
    local scheduling = self:compute_scheduling_for_worker(id, graph)
+   self:info('Starting worker %s.', id)
    self.workers[id] = { scheduling=scheduling,
                         pid=self:start_worker(scheduling),
                         queue={}, graph=graph }
+   self:debug('Worker %s has PID %s.', id, self.workers[id].pid)
    local actions = self.support.compute_config_actions(
       app_graph.new(), self.workers[id].graph, {}, 'load')
    self:enqueue_config_actions_for_worker(id, actions)
@@ -165,13 +184,11 @@ function Manager:take_worker_message_queue ()
    return actions
 end
 
-local verbose = os.getenv('SNABB_MANAGER_VERBOSE') and true
-
-function Manager:enqueue_config_actions_for_worker(worker, actions)
+function Manager:enqueue_config_actions_for_worker(id, actions)
    for _,action in ipairs(actions) do
-      if verbose then print('encode', action[1], unpack(action[2])) end
+      self:debug('encode %s for worker %s', action[1], id)
       local buf, len = action_codec.encode(action)
-      table.insert(self.workers[worker].queue, { buf=buf, len=len })
+      table.insert(self.workers[id].queue, { buf=buf, len=len })
    end
 end
 
@@ -836,7 +853,7 @@ function Manager:handle_calls_from_peers()
          end
       end
       if peer.state == 'done' or peer.state == 'error' then
-         if peer.state == 'error' then print('error: '..peer.msg) end
+         if peer.state == 'error' then self:warn('%s', peer.msg) end
          peer.fd:close()
          table.remove(peers, i)
          if self.listen_peer == peer then self.listen_peer = nil end
@@ -847,11 +864,14 @@ function Manager:handle_calls_from_peers()
 end
 
 function Manager:send_messages_to_workers()
-   for _,worker in pairs(self.workers) do
+   for id,worker in pairs(self.workers) do
       if not worker.channel then
          local name = '/'..tostring(worker.pid)..'/config-worker-channel'
          local success, channel = pcall(channel.open, name)
-         if success then worker.channel = channel end
+         if success then
+            worker.channel = channel
+            self:info("Worker %s has started (PID %s).", id, worker.pid)
+         end
       end
       local channel = worker.channel
       if channel then
@@ -918,10 +938,7 @@ function Manager:stop ()
    S.unlink(self.socket_file_name)
 
    for id, worker in pairs(self.workers) do
-      if not worker.shutting_down then
-         print(string.format('Asking worker %s to shut down.', id))
-         self:stop_worker(id)
-      end
+      if not worker.shutting_down then self:stop_worker(id) end
    end
    -- Wait 250ms for workers to shut down nicely, polling every 5ms.
    local start = C.get_monotonic_time()
@@ -933,19 +950,20 @@ function Manager:stop ()
    end
    -- If that didn't work, send SIGKILL and wait indefinitely.
    for id, worker in pairs(self.workers) do
-      print(string.format('Forcing worker %s to shut down.', id))
+      self:warn('Forcing worker %s to shut down.', id)
       S.kill(worker.pid, "KILL")
    end
    while next(self.workers) do
       self:remove_stale_workers()
       C.usleep(5000)
    end
+   self:info('Shutdown complete.')
 end
 
 function Manager:main (duration)
    local now = C.get_monotonic_time()
    local stop = now + (duration or 1/0)
-   while stop < now do
+   while now < stop do
       next_time = now + self.period
       self:remove_stale_workers()
       self:handle_calls_from_peers()
@@ -979,11 +997,15 @@ function selftest ()
                           -- Use a schema with no data nodes, just for
                           -- testing.
                           schema_name='ietf-inet-types',
-                          initial_configuration={}})
-   m:main(0.05)
+                          initial_configuration={},
+                          log_level="DEBUG"})
    assert(m.workers[1])
    assert(m.workers[1].graph.links)
    assert(m.workers[1].graph.links["source.foo -> sink.bar"])
+   -- Worker will be started once main loop starts to run.
+   assert(not m.workers[1].channel)
+   -- Wait for worker to start.
+   while not m.workers[1].channel do m:main(0.005) end
    m:stop()
    assert(m.workers[1] == nil)
    print('selftest: ok')

From 9511c0b0dafd1fbd1ff29c845abe3bf53d9cd01a Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@igalia.com>
Date: Thu, 7 Dec 2017 11:34:33 +0100
Subject: [PATCH 13/31] Update ptree worker comment

---
 src/lib/ptree/worker.lua | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/lib/ptree/worker.lua b/src/lib/ptree/worker.lua
index 86452808a1..b779cae0b0 100644
--- a/src/lib/ptree/worker.lua
+++ b/src/lib/ptree/worker.lua
@@ -42,7 +42,7 @@ function new_worker (conf)
 end
 
 function Worker:shutdown()
-   -- This will shutdown everything.
+   -- This will call stop() on all apps.
    engine.configure(app_graph.new())
 
    -- Now we can exit.

From 7c0d1a12014575b680e4b6ec7def809cc75840bb Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@igalia.com>
Date: Thu, 7 Dec 2017 11:52:22 +0100
Subject: [PATCH 14/31] Ptree manager runs timers

---
 src/lib/ptree/manager.lua | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/src/lib/ptree/manager.lua b/src/lib/ptree/manager.lua
index e9b0493717..d9d126f4e1 100644
--- a/src/lib/ptree/manager.lua
+++ b/src/lib/ptree/manager.lua
@@ -5,7 +5,11 @@ module(...,package.seeall)
 local S = require("syscall")
 local ffi = require("ffi")
 local C = ffi.C
+local app_graph = require("core.config")
 local lib = require("core.lib")
+local shm = require("core.shm")
+local timer = require("core.timer")
+local worker = require("core.worker")
 local cltable = require("lib.cltable")
 local cpuset = require("lib.cpuset")
 local scheduling = require("lib.scheduling")
@@ -16,9 +20,6 @@ local schema = require("lib.yang.schema")
 local rpc = require("lib.yang.rpc")
 local state = require("lib.yang.state")
 local path_mod = require("lib.yang.path")
-local shm = require("core.shm")
-local worker = require("core.worker")
-local app_graph = require("core.config")
 local action_codec = require("lib.ptree.action_codec")
 local alarm_codec = require("lib.ptree.alarm_codec")
 local support = require("lib.ptree.support")
@@ -965,6 +966,7 @@ function Manager:main (duration)
    local stop = now + (duration or 1/0)
    while now < stop do
       next_time = now + self.period
+      timer.run_to_time(now * 1e9)
       self:remove_stale_workers()
       self:handle_calls_from_peers()
       self:send_messages_to_workers()

From 3c5f169e02d1f6d394510300562ea729fae6f58a Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@igalia.com>
Date: Thu, 7 Dec 2017 15:21:39 +0100
Subject: [PATCH 15/31] Ptree manager has "state change listener" facility

---
 src/lib/ptree/manager.lua | 42 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 41 insertions(+), 1 deletion(-)

diff --git a/src/lib/ptree/manager.lua b/src/lib/ptree/manager.lua
index d9d126f4e1..e0cf2b78d8 100644
--- a/src/lib/ptree/manager.lua
+++ b/src/lib/ptree/manager.lua
@@ -74,6 +74,7 @@ function new_manager (conf)
    ret.period = 1/conf.Hz
    ret.worker_default_scheduling = conf.worker_default_scheduling
    ret.workers = {}
+   ret.state_change_listeners = {}
    ret.rpc_callee = rpc.prepare_callee('snabb-config-leader-v1')
    ret.rpc_handler = rpc.dispatch_handler(ret, 'rpc_')
 
@@ -93,6 +94,31 @@ function Manager:debug(fmt, ...) self:log("DEBUG", fmt, ...) end
 function Manager:info(fmt, ...) self:log("INFO", fmt, ...) end
 function Manager:warn(fmt, ...) self:log("WARN", fmt, ...) end
 
+function Manager:add_state_change_listener(listener)
+   table.insert(self.state_change_listeners, listener)
+   for id, worker in pairs(self.workers) do
+      listener:worker_starting(id)
+      if worker.channel then listener:worker_started(id, worker.pid) end
+      if worker.shutting_down then listener:worker_stopping(id) end
+   end
+end
+
+function Manager:remove_state_change_listener(listener)
+   for i, x in ipairs(self.state_change_listeners) do
+      if x == listener then
+         table.remove(self.state_change_listeners, i)
+         return
+      end
+   end
+   error("listener not found")
+end
+
+function Manager:state_change_event(event, ...)
+   for _,listener in ipairs(self.state_change_listeners) do
+      listener[event](listener, ...)
+   end
+end
+
 function Manager:set_initial_configuration (configuration)
    self.current_configuration = configuration
    self.current_in_place_dependencies = {}
@@ -123,6 +149,7 @@ end
 function Manager:stop_worker(id)
    self:info('Asking worker %s to shut down.', id)
    local stop_actions = {{'shutdown', {}}, {'commit', {}}}
+   self:state_change_event('worker_stopping', id)
    self:enqueue_config_actions_for_worker(id, stop_actions)
    self:send_messages_to_workers()
    self.workers[id].shutting_down = true
@@ -138,6 +165,7 @@ function Manager:remove_stale_workers()
       end
    end
    for _, id in ipairs(stale) do
+      self:state_change_event('worker_stopped', id)
       if self.workers[id].scheduling.cpu then
 	 self.cpuset:release(self.workers[id].scheduling.cpu)
       end
@@ -172,6 +200,7 @@ function Manager:start_worker_for_graph(id, graph)
    self.workers[id] = { scheduling=scheduling,
                         pid=self:start_worker(scheduling),
                         queue={}, graph=graph }
+   self:state_change_event('worker_starting', id)
    self:debug('Worker %s has PID %s.', id, self.workers[id].pid)
    local actions = self.support.compute_config_actions(
       app_graph.new(), self.workers[id].graph, {}, 'load')
@@ -871,6 +900,7 @@ function Manager:send_messages_to_workers()
          local success, channel = pcall(channel.open, name)
          if success then
             worker.channel = channel
+            self:state_change_event('worker_started', id, worker.pid)
             self:info("Worker %s has started (PID %s).", id, worker.pid)
          end
       end
@@ -966,7 +996,7 @@ function Manager:main (duration)
    local stop = now + (duration or 1/0)
    while now < stop do
       next_time = now + self.period
-      timer.run_to_time(now * 1e9)
+      if timer.ticks then timer.run_to_time(now * 1e9) end
       self:remove_stale_workers()
       self:handle_calls_from_peers()
       self:send_messages_to_workers()
@@ -1001,7 +1031,14 @@ function selftest ()
                           schema_name='ietf-inet-types',
                           initial_configuration={},
                           log_level="DEBUG"})
+   local l = {log={}}
+   function l:worker_starting(...) table.insert(self.log,{'starting',...}) end
+   function l:worker_started(...)  table.insert(self.log,{'started',...})  end
+   function l:worker_stopping(...) table.insert(self.log,{'stopping',...}) end
+   function l:worker_stopped(...)  table.insert(self.log,{'stopped',...})  end
+   m:add_state_change_listener(l)
    assert(m.workers[1])
+   local pid = m.workers[1].pid
    assert(m.workers[1].graph.links)
    assert(m.workers[1].graph.links["source.foo -> sink.bar"])
    -- Worker will be started once main loop starts to run.
@@ -1010,5 +1047,8 @@ function selftest ()
    while not m.workers[1].channel do m:main(0.005) end
    m:stop()
    assert(m.workers[1] == nil)
+   assert(lib.equal(l.log,
+                    { {'starting', 1}, {'started', 1, pid}, {'stopping', 1},
+                      {'stopped', 1} }))
    print('selftest: ok')
 end

From 6080eca67088487afa20c6d23118f7e569e42670 Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@igalia.com>
Date: Thu, 7 Dec 2017 15:21:57 +0100
Subject: [PATCH 16/31] Timers can be cancelled

Allow timers to be cancelled.
---
 src/core/timer.lua | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/src/core/timer.lua b/src/core/timer.lua
index 996caef73c..3e93f94b71 100644
--- a/src/core/timer.lua
+++ b/src/core/timer.lua
@@ -33,6 +33,7 @@ local function call_timers (l)
       if debug then
          print(string.format("running timer %s at tick %s", timer.name, ticks))
       end
+      timer.next_tick = nil
       timer.fn(timer)
       if timer.repeating then activate(timer) end
    end
@@ -49,6 +50,7 @@ function run_to_time (ns)
 end
 
 function activate (t)
+   assert(t.next_tick == nil, "timer already activated")
    -- Initialize time
    if not ticks then
       ticks = math.floor(tonumber(C.get_time_ns() / ns_per_tick))
@@ -59,6 +61,19 @@ function activate (t)
    else
       timers[tick] = {t}
    end
+   t.next_tick = tick
+end
+
+function cancel (t)
+   if t.next_tick then
+      for idx, timer in ipairs(timers[t.next_tick]) do
+         if timer == t then
+            table.remove(timers[t.next_tick], idx)
+            t.next_tick = nil
+            return true
+         end
+      end
+   end
 end
 
 function new (name, fn, nanos, mode)

From 5c3600ab32e4b6e8872cddaea91a2a1a0422bfde Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@igalia.com>
Date: Thu, 7 Dec 2017 15:23:20 +0100
Subject: [PATCH 17/31] Port lwAFTR over to use lib.ptree

Having the multiprocess tree and configuration system be implemented as
apps was a neat hack that re-used the app abstraction as a way to add a
reconfiguration back-channel to a dataplane, but it goes against the
abstraction that the core.app / engine facility should really be
targeted towards the dataplane domain; it's not necessarily a great
generic process composition facility.
---
 src/apps/lwaftr/lwaftr.lua         |   6 +-
 src/lib/yang/alarms.lua            |   2 +-
 src/program/lwaftr/bench/bench.lua |  39 +++-----
 src/program/lwaftr/csv_stats.lua   | 107 +++++++++++++---------
 src/program/lwaftr/query/query.lua |  33 +++----
 src/program/lwaftr/run/run.lua     |  34 ++++---
 src/program/lwaftr/setup.lua       | 138 +++--------------------------
 src/program/ps/ps.lua              |   4 +-
 8 files changed, 131 insertions(+), 232 deletions(-)

diff --git a/src/apps/lwaftr/lwaftr.lua b/src/apps/lwaftr/lwaftr.lua
index e1dc1abfc0..c88a54afcf 100644
--- a/src/apps/lwaftr/lwaftr.lua
+++ b/src/apps/lwaftr/lwaftr.lua
@@ -473,9 +473,9 @@ function LwAftr:new(conf)
    return o
 end
 
--- The following two methods are called by apps.config.follower in
--- reaction to binding table changes, via
--- apps/config/support/snabb-softwire-v2.lua.
+-- The following two methods are called by lib.ptree.worker in reaction
+-- to binding table changes, via
+-- lib/ptree/support/snabb-softwire-v2.lua.
 function LwAftr:add_softwire_entry(entry_blob)
    self.binding_table:add_softwire_entry(entry_blob)
 end
diff --git a/src/lib/yang/alarms.lua b/src/lib/yang/alarms.lua
index d5456e7eb6..b5fb24836b 100644
--- a/src/lib/yang/alarms.lua
+++ b/src/lib/yang/alarms.lua
@@ -3,7 +3,7 @@ module(..., package.seeall)
 local data = require('lib.yang.data')
 local lib = require('core.lib')
 local util = require('lib.yang.util')
-local alarm_codec = require('apps.config.alarm_codec')
+local alarm_codec = require('lib.ptree.alarm_codec')
 local counter = require("core.counter")
 
 local format_date_as_iso_8601 = util.format_date_as_iso_8601
diff --git a/src/program/lwaftr/bench/bench.lua b/src/program/lwaftr/bench/bench.lua
index 4b9239f0c7..8802a3c046 100644
--- a/src/program/lwaftr/bench/bench.lua
+++ b/src/program/lwaftr/bench/bench.lua
@@ -38,23 +38,6 @@ function parse_args(args)
    return opts, scheduling, unpack(args)
 end
 
--- Finds current followers for leader (note it puts the pid as the key)
-local function find_followers()
-   local followers = {}
-   local mypid = S.getpid()
-   for _, name in ipairs(shm.children("/")) do
-      local pid = tonumber(name)
-      if pid ~= nil and shm.exists("/"..pid.."/group") then
-         local path = S.readlink(shm.root.."/"..pid.."/group")
-         local parent = tonumber(lib.basename(lib.dirname(path)))
-         if parent == mypid then
-            followers[pid] = true
-         end
-      end
-   end
-   return followers
-end
-
 function run(args)
    local opts, scheduling, conf_file, inv4_pcap, inv6_pcap = parse_args(args)
    local conf = setup.read_config(conf_file)
@@ -65,23 +48,29 @@ function run(args)
       conf.softwire_config.name = opts.name
    end
 
-   local graph = config.new()
    local function setup_fn(graph, lwconfig)
       return setup.load_bench(graph, lwconfig, inv4_pcap, inv6_pcap, 'sinkv4',
 			      'sinkv6')
    end
 
-   setup.reconfigurable(scheduling, setup_fn, graph, conf)
-   app.configure(graph)
+   local manager = setup.ptree_manager(scheduling, setup_fn, conf)
 
-   local function start_sampling_for_pid(pid, write_header)
+   local stats = {csv={}}
+   function stats:worker_starting(id) end
+   function stats:worker_started(id, pid)
       local csv = csv_stats.CSVStatsTimer:new(opts.bench_file, opts.hydra, pid)
       csv:add_app('sinkv4', { 'input' }, { input=opts.hydra and 'decap' or 'Decap.' })
       csv:add_app('sinkv6', { 'input' }, { input=opts.hydra and 'encap' or 'Encap.' })
-      csv:activate(write_header)
+      self.csv[id] = csv
+      self.csv[id]:start()
    end
+   function stats:worker_stopping(id)
+      self.csv[id]:stop()
+      self.csv[id] = nil
+   end
+   function stats:worker_stopped(id) end
+   manager:add_state_change_listener(stats)
 
-   setup.start_sampling(start_sampling_for_pid)
-
-   app.main({duration=opts.duration})
+   manager:main(opts.duration)
+   manager:stop()
 end
diff --git a/src/program/lwaftr/csv_stats.lua b/src/program/lwaftr/csv_stats.lua
index dae22b6e45..ef2e9acd60 100644
--- a/src/program/lwaftr/csv_stats.lua
+++ b/src/program/lwaftr/csv_stats.lua
@@ -47,31 +47,18 @@ function CSVStatsTimer:new(filename, hydra_mode, pid)
    local file = filename and io.open(filename, "w") or io.stdout
    local o = { hydra_mode=hydra_mode, link_data={}, file=file, period=1,
       header = hydra_mode and "benchmark,id,score,unit" or "Time (s)"}
+   o.ready = false
+   o.deferred_apps = {}
    o.pid = pid or S.getpid()
-   o.links_by_app = open_link_counters(o.pid)
    return setmetatable(o, {__index = CSVStatsTimer})
 end
 
--- Add links from an app whose identifier is ID to the CSV timer.  If
--- present, LINKS is an array of strings identifying a subset of links
--- to monitor.  The optional LINK_NAMES table maps link names to
--- human-readable names, for the column headers.
-function CSVStatsTimer:add_app(id, links, link_names)
-   local function add_link_data(name, link)
-      local link_name = link_names[name] or name
-      if not self.hydra_mode then
-         local h = (',%s MPPS,%s Gbps'):format(link_name, link_name)
-         self.header = self.header..h
-      end
-      local data = {
-         link_name = link_name,
-         txpackets = link.txpackets,
-         txbytes = link.txbytes,
-      }
-      table.insert(self.link_data, data)
-   end
-
-   local app = assert(self.links_by_app[id], "App named "..id.." not found")
+function CSVStatsTimer:resolve_app(deferred)
+   local id, links, link_names = unpack(assert(deferred))
+   self.links_by_app = open_link_counters(self.pid)
+   local app = self.links_by_app[id]
+   if not app then return false end
+   local resolved_links = {}
    for _,name in ipairs(links) do
       local link = app.input[name] or app.output[name]
       -- If we didn't find these links, allow a link name of "rx" to be
@@ -84,41 +71,81 @@ function CSVStatsTimer:add_app(id, links, link_names)
          if name == 'rx' then link = app.input.input end
          if name == 'tx' then link = app.output.output end
       end
-      assert(link, "Link named "..name.." not found in "..id)
-      add_link_data(name, link)
+      if not link then return false end
+      table.insert(resolved_links, {name, link})
    end
+   for _, resolved_link in ipairs(resolved_links) do
+      local name, link = unpack(resolved_link)
+      local link_name = link_names[name] or name
+      local data = {
+         link_name = link_name,
+         txpackets = link.txpackets,
+         txbytes = link.txbytes,
+      }
+      if not self.hydra_mode then
+         local h = (',%s MPPS,%s Gbps'):format(link_name, link_name)
+         self.header = self.header..h
+      end
+      table.insert(self.link_data, data)
+   end
+   return true
+end
+
+-- Add links from an app whose identifier is ID to the CSV timer.  If
+-- present, LINKS is an array of strings identifying a subset of links
+-- to monitor.  The optional LINK_NAMES table maps link names to
+-- human-readable names, for the column headers.
+function CSVStatsTimer:add_app(id, links, link_names)
+   -- Because we are usually measuring counters from another process and
+   -- that process is probably spinning up as we are installing the
+   -- counter, we defer the resolve operation and try to resolve it from
+   -- inside the timer.
+   table.insert(self.deferred_apps, {id, links, link_names})
 end
 
 function CSVStatsTimer:set_period(period) self.period = period end
 
 -- Activate the timer with a period of PERIOD seconds.
-function CSVStatsTimer:activate(write_header)
-   if write_header then
-      self.file:write(self.header..'\n')
-      self.file:flush()
+function CSVStatsTimer:start()
+   local function tick() return self:tick() end
+   self.tick_timer = timer.new('csv_stats', tick, self.period*1e9, 'repeating')
+   tick()
+   timer.activate(self.tick_timer)
+end
+
+function CSVStatsTimer:stop()
+   self:tick() -- ?
+   timer.cancel(self.tick_timer)
+end
+
+function CSVStatsTimer:is_ready()
+   if self.ready then return true end
+   for i,data in ipairs(self.deferred_apps) do
+      if not data then
+         -- pass
+      elseif self:resolve_app(data) then
+         self.deferred_apps[i] = false
+      else
+         return false
+      end
    end
+   -- print header
+   self.file:write(self.header..'\n')
+   self.file:flush()
    self.start = engine.now()
    self.prev_elapsed = 0
    for _,data in ipairs(self.link_data) do
       data.prev_txpackets = counter.read(data.txpackets)
       data.prev_txbytes = counter.read(data.txbytes)
    end
-   local function tick() return self:tick() end
-   self.tick_timer = timer.new('csv_stats', tick, self.period*1e9, 'repeating')
-   timer.activate(self.tick_timer)
-   return self.tick_timer
-end
-
-function CSVStatsTimer:check_alive()
-   -- Instances can be terminated periodically, this checks for that and if so
-   -- removes the timer so the3 stats don't get displayed indefinitely.
-   if S.waitpid(self.pid, S.c.W["NOHANG"]) ~= 0 then
-      self.tick_timer.repeating = false
-   end
+   self.ready = true
+   -- Return false for the last time, so that our first reading is
+   -- legit.
+   return false
 end
 
 function CSVStatsTimer:tick()
-   self:check_alive()
+   if not self:is_ready() then return end
    local elapsed = engine.now() - self.start
    local dt = elapsed - self.prev_elapsed
    self.prev_elapsed = elapsed
diff --git a/src/program/lwaftr/query/query.lua b/src/program/lwaftr/query/query.lua
index e0c57e68cd..8ba68c7057 100644
--- a/src/program/lwaftr/query/query.lua
+++ b/src/program/lwaftr/query/query.lua
@@ -82,8 +82,9 @@ local function print_counters (pid, filter)
    end
 end
 
--- Return the pid that was specified, unless it was a leader process,
--- in which case, return the follower pid that actually has useful counters.
+-- Return the pid that was specified, unless it was a manager process,
+-- in which case, return the worker pid that actually has useful
+-- counters.
 local function pid_to_parent(pid)
    -- It's meaningless to get the parent of a nil 'pid'.
    if not pid then return pid end
@@ -91,10 +92,11 @@ local function pid_to_parent(pid)
    for _, name in ipairs(shm.children("/")) do
       local p = tonumber(name)
       if p and ps.is_worker(p) then
-         local leader_pid = tonumber(ps.get_leader_pid(p))
-         -- If the precomputed by-name pid is the leader pid, set the pid
-         -- to be the follower's pid instead to get meaningful counters.
-         if leader_pid == pid then pid = p end
+         local manager_pid = tonumber(ps.get_manager_pid(p))
+         -- If the precomputed by-name pid is the manager pid, set the
+         -- pid to be the worker's pid instead to get meaningful
+         -- counters.
+         if manager_pid == pid then pid = p end
       end
    end
    return pid
@@ -115,18 +117,19 @@ function run (raw_args)
          fatal(("Couldn't find process with name '%s'"):format(opts.name))
       end
 
-      -- Check if it was run with --reconfigurable
-      -- If it was, find the children, then find the pid of their parent.
-      -- Note that this approach will break as soon as there can be multiple
-      -- followers which need to have their statistics aggregated, as it will
-      -- only print the statistics for one child, not for all of them.
+      -- Check if it was run with --reconfigurable If it was, find the
+      -- children, then find the pid of their parent.  Note that this
+      -- approach will break as soon as there can be multiple workers
+      -- which need to have their statistics aggregated, as it will only
+      -- print the statistics for one child, not for all of them.
       for _, name in ipairs(shm.children("/")) do
          local p = tonumber(name)
          if p and ps.is_worker(p) then
-            local leader_pid = tonumber(ps.get_leader_pid(p))
-            -- If the precomputed by-name pid is the leader pid, set the pid
-            -- to be the follower's pid instead to get meaningful counters.
-            if leader_pid == pid then pid = p end
+            local manager_pid = tonumber(ps.get_manager_pid(p))
+            -- If the precomputed by-name pid is the manager pid, set
+            -- the pid to be the worker's pid instead to get meaningful
+            -- counters.
+            if manager_pid == pid then pid = p end
          end
       end
    end
diff --git a/src/program/lwaftr/run/run.lua b/src/program/lwaftr/run/run.lua
index 881b383be2..51b1da7b47 100644
--- a/src/program/lwaftr/run/run.lua
+++ b/src/program/lwaftr/run/run.lua
@@ -176,20 +176,19 @@ function run(args)
       end
    end
 
-   local c = config.new()
+   local manager = setup.ptree_manager(scheduling, setup_fn, conf)
 
-   conf.alarm_notification = true
-   setup.reconfigurable(scheduling, setup_fn, c, conf)
-   engine.configure(c)
-
-   if opts.verbosity >= 2 then
+   -- FIXME: Doesn't work in multi-process environment.
+   if false and opts.verbosity >= 2 then
       local function lnicui_info() engine.report_apps() end
       local t = timer.new("report", lnicui_info, 1e9, 'repeating')
       timer.activate(t)
    end
 
    if opts.verbosity >= 1 then
-      function add_csv_stats_for_pid(pid, write_header)
+      local stats = {csv={}}
+      function stats:worker_starting(id) end
+      function stats:worker_started(id, pid)
          local csv = csv_stats.CSVStatsTimer:new(opts.bench_file, opts.hydra, pid)
          -- Link names like "tx" are from the app's perspective, but
          -- these labels are from the perspective of the lwAFTR as a
@@ -205,18 +204,17 @@ function run(args)
             csv:add_app('inetNic', { 'tx', 'rx' }, { tx=ipv4_tx, rx=ipv4_rx })
             csv:add_app('b4sideNic', { 'tx', 'rx' }, { tx=ipv6_tx, rx=ipv6_rx })
          end
-         csv:activate(write_header)
+         self.csv[id] = csv
+         self.csv[id]:start()
       end
-      setup.start_sampling(add_csv_stats_for_pid)
-   end
-
-   if opts.ingress_drop_monitor then
-      io.stderr:write("Warning: Ingress drop monitor not yet supported\n")
+      function stats:worker_stopping(id)
+         self.csv[id]:stop()
+         self.csv[id] = nil
+      end
+      function stats:worker_stopped(id) end
+      manager:add_state_change_listener(stats)
    end
 
-   if opts.duration then
-      engine.main({duration=opts.duration, report={showlinks=true}})
-   else
-      engine.main({report={showlinks=true}})
-   end
+   manager:main(opts.duration)
+   manager:stop()
 end
diff --git a/src/program/lwaftr/setup.lua b/src/program/lwaftr/setup.lua
index 55e9aafdb5..b60d919640 100644
--- a/src/program/lwaftr/setup.lua
+++ b/src/program/lwaftr/setup.lua
@@ -1,8 +1,7 @@
 module(..., package.seeall)
 
 local config     = require("core.config")
-local leader     = require("apps.config.leader")
-local follower   = require("apps.config.follower")
+local manager    = require("lib.ptree.manager")
 local PcapFilter = require("apps.packet_filter.pcap_filter").PcapFilter
 local V4V6       = require("apps.lwaftr.V4V6").V4V6
 local VirtioNet  = require("apps.virtio_net.virtio_net").VirtioNet
@@ -562,123 +561,6 @@ function load_soak_test_on_a_stick (c, conf, inv4_pcap, inv6_pcap)
    link_sink(c, unpack(sinks))
 end
 
-local apply_scheduling_opts = {
-   pci_addrs = { default={} },
-   real_time = { default=false },
-   ingress_drop_monitor = { default='flush' },
-   j = {}
-}
-function apply_scheduling(opts)
-   local lib = require("core.lib")
-   local ingress_drop_monitor = require("lib.timers.ingress_drop_monitor")
-   local fatal = lwutil.fatal
-
-   opts = lib.parse(opts, apply_scheduling_opts)
-   if opts.ingress_drop_monitor then
-      local mon = ingress_drop_monitor.new({action=opts.ingress_drop_monitor})
-      timer.activate(mon:timer())
-   end
-   if opts.real_time then
-      if not S.sched_setscheduler(0, "fifo", 1) then
-         fatal('Failed to enable real-time scheduling.  Try running as root.')
-      end
-   end
-   if opts.j then
-      local arg = opts.j
-      if arg:match("^v") then
-         local file = arg:match("^v=(.*)")
-         if file == '' then file = nil end
-         require("jit.v").start(file)
-      elseif arg:match("^p") then
-         local opts, file = arg:match("^p=([^,]*),?(.*)")
-         if file == '' then file = nil end
-         local prof = require('jit.p')
-         prof.start(opts, file)
-         local function report() prof.stop(); prof.start(opts, file) end
-         timer.activate(timer.new('p', report, 10e9, 'repeating'))
-      elseif arg:match("^dump") then
-         local opts, file = arg:match("^dump=([^,]*),?(.*)")
-         if file == '' then file = nil end
-         require("jit.dump").on(opts, file)
-      elseif arg:match("^tprof") then
-         local prof = require('lib.traceprof.traceprof')
-         prof.start()
-         local function report() prof.stop(); prof.start() end
-         timer.activate(timer.new('tprof', report, 10e9, 'repeating'))
-      end
-   end
-end
-
-function run_worker(scheduling)
-   local app = require("core.app")
-   apply_scheduling(scheduling)
-   local myconf = config.new()
-   config.app(myconf, "follower", follower.Follower, {})
-   app.configure(myconf)
-   app.busywait = true
-   app.main({})
-end
-
-local function stringify(x)
-   if type(x) == 'string' then return string.format('%q', x) end
-   if type(x) == 'number' then return tostring(x) end
-   if type(x) == 'boolean' then return x and 'true' or 'false' end
-   assert(type(x) == 'table')
-   local ret = {"{"}
-   local first = true
-   for k,v in pairs(x) do
-      if first then first = false else table.insert(ret, ",") end
-      table.insert(ret, string.format('[%s]=%s', stringify(k), stringify(v)))
-   end
-   table.insert(ret, "}")
-   return table.concat(ret)
-end
-
--- Takes a function (which takes a follower PID) and starts sampling
---
--- The function searches for followers of the leader and when a new one
--- appears it calls the sampling function (passed in) with the follower
--- PID to begin the sampling. The sampling function should look like:
---    function(pid, write_header)
--- If write_header is false it should not write a new header.
-function start_sampling(sample_fn)
-   local header_written = false
-   local followers = {}
-   local function find_followers()
-      local ret = {}
-      local mypid = S.getpid()
-      for _, name in ipairs(shm.children("/")) do
-         local pid = tonumber(name)
-         if pid ~= nil and shm.exists("/"..pid.."/group") then
-            local path = S.readlink(shm.root.."/"..pid.."/group")
-            local parent = tonumber(lib.basename(lib.dirname(path)))
-            if parent == mypid then
-               ret[pid] = true
-            end
-         end
-      end
-      return ret
-   end
-
-   local function sample_for_new_followers()
-      local new_followers = find_followers()
-      for pid, _ in pairs(new_followers) do
-         if followers[pid] == nil then
-            if not pcall(sample_fn, pid, (not header_written)) then
-               new_followers[pid] = nil
-               io.stderr:write("Waiting on follower "..pid..
-                  " to start ".."before recording statistics...\n")
-            else
-               header_written = true
-            end
-         end
-      end
-      followers = new_followers
-   end
-   timer.activate(timer.new('start_sampling', sample_for_new_followers,
-      1e9, 'repeating'))
-end
-
 -- Produces configuration for each worker.  Each queue on each device
 -- will get its own worker process.
 local function compute_worker_configs(conf)
@@ -707,7 +589,7 @@ local function compute_worker_configs(conf)
    return ret
 end
 
-function reconfigurable(scheduling, f, graph, conf)
+function ptree_manager(scheduling, f, conf)
    -- Always enabled in reconfigurable mode.
    alarm_notification = true
 
@@ -721,12 +603,12 @@ function reconfigurable(scheduling, f, graph, conf)
       return worker_app_graphs
    end
 
-   local worker_code = "require('program.lwaftr.setup').run_worker(%s)"
-   worker_code = worker_code:format(stringify(scheduling))
-
-   config.app(graph, 'leader', leader.Leader,
-              { setup_fn = setup_fn, initial_configuration = conf,
-                worker_start_code = worker_code,
-                schema_name = 'snabb-softwire-v2',
-                default_schema = 'ietf-softwire-br'})
+   return manager.new_manager {
+      setup_fn = setup_fn,
+      initial_configuration = conf,
+      schema_name = 'snabb-softwire-v2',
+      default_schema = 'ietf-softwire-br',
+      worker_default_scheduling = scheduling,
+      -- log_level="DEBUG"
+   }
 end
diff --git a/src/program/ps/ps.lua b/src/program/ps/ps.lua
index 5760bb6355..975b64f4a5 100644
--- a/src/program/ps/ps.lua
+++ b/src/program/ps/ps.lua
@@ -47,7 +47,7 @@ local function is_addressable (pid)
    return false
 end
 
-function get_leader_pid (pid)
+function get_manager_pid (pid)
    local fq = shm.root.."/"..pid.."/group"
    local path = S.readlink(fq)
    return basename(dirname(path))
@@ -65,7 +65,7 @@ local function compute_snabb_instances()
       if p and p ~= my_pid then
          local instance = {pid=p, name=name}
          if is_worker(p) then
-            instance.leader = get_leader_pid(p)
+            instance.leader = get_manager_pid(p)
          end
          if is_addressable(p) then
             instance.addressable = true

From cb72a6e311607ad61d0c8f8e6768001e80d05c01 Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@igalia.com>
Date: Thu, 7 Dec 2017 15:32:36 +0100
Subject: [PATCH 18/31] Remove apps.config

---
 src/apps/config/README.md                     | 145 ---
 src/apps/config/action_codec.lua              | 234 -----
 src/apps/config/alarm_codec.lua               | 314 ------
 src/apps/config/channel.lua                   | 231 -----
 src/apps/config/follower.lua                  |  96 --
 src/apps/config/leader.lua                    | 956 ------------------
 src/apps/config/support.lua                   | 231 -----
 src/apps/config/support/snabb-softwire-v2.lua | 706 -------------
 8 files changed, 2913 deletions(-)
 delete mode 100644 src/apps/config/README.md
 delete mode 100644 src/apps/config/action_codec.lua
 delete mode 100644 src/apps/config/alarm_codec.lua
 delete mode 100644 src/apps/config/channel.lua
 delete mode 100644 src/apps/config/follower.lua
 delete mode 100644 src/apps/config/leader.lua
 delete mode 100644 src/apps/config/support.lua
 delete mode 100644 src/apps/config/support/snabb-softwire-v2.lua

diff --git a/src/apps/config/README.md b/src/apps/config/README.md
deleted file mode 100644
index cb1d37dc8c..0000000000
--- a/src/apps/config/README.md
+++ /dev/null
@@ -1,145 +0,0 @@
-# Config leader and follower
-
-Sometimes you want to query the state or configuration of a running
-Snabb data plane, or reload its configuration, or incrementally update
-that configuration.  However, you want to minimize the impact of
-configuration query and update on data plane performance.  The
-`Leader` and `Follower` apps are here to fulfill this need, while
-minimizing performance overhead.
-
-The high-level design is that a `Leader` app is responsible for
-knowing the state and configuration of a data plane.  The leader
-offers an interface to allow the outside world to query the
-configuration and state, and to request configuration updates.  To
-avoid data-plane overhead, the `Leader` app should be deployed in a
-separate process.  Because it knows the data-plane state, it can
-respond to queries directly, without involving the data plane.  It
-processes update requests into a form that the data plane can handle,
-and feeds those requests to the data plane via a high-performance
-back-channel.
-
-The data plane runs a `Follower` app that reads and applies update
-messages sent to it from the leader.  Checking for update availability
-requires just a memory access, not a system call, so the overhead of
-including a follower in the data plane is very low.
-
-## Two protocols
-
-The leader communicates with its followers using a private protocol.
-Because the leader and the follower are from the same Snabb version,
-the details of this protocol are subject to change.  The private
-protocol's only design constraint is that it should cause the lowest
-overhead for the data plane.
-
-The leader communicates with the world via a public protocol.  The
-"snabb config" command-line tool speaks this protocol.  "snabb config
-get foo /bar" will find the local Snabb instance named "foo", open the
-UNIX socket that the "foo" instance is listening on, issue a request,
-then read the response, then close the socket.
-
-## Public protocol
-
-The design constraint on the public protocol is that it be expressive
-and future-proof.  We also want to enable the leader to talk to more
-than one "snabb config" at a time.  In particular someone should be
-able to have a long-lived "snabb config listen" session open, and that
-shouldn't impede someone else from doing a "snabb config get" to read
-state.
-
-To this end the public protocol container is very simple:
-
-```
-Message = Length "\n" RPC*
-```
-
-Length is a base-10 string of characters indicating the length of the
-message.  There may be a maximum length restriction.  This requires
-that "snabb config" build up the whole message as a string and measure
-its length, but that's OK.  Knowing the length ahead of time allows
-"snabb config" to use nonblocking operations to slurp up the whole
-message as a string.  A partial read can be resumed later.  The
-message can then be parsed without fear of blocking the main process.
-
-The RPC is an RPC request or response for the
-[`snabb-config-leader-v1` YANG
-schema](../../lib/yang/snabb-config-leader-v1.yang), expressed in the
-Snabb [textual data format for YANG data](../../lib/yang/README.md).
-For example the `snabb-config-leader-v1` schema supports a
-`get-config` RPC defined like this in the schema:
-
-```yang
-rpc get-config {
-  input {
-    leaf schema { type string; mandatory true; }
-    leaf revision { type string; }
-    leaf path { type string; default "/"; }
-  }
-  output {
-    leaf config { type string; }
-  }
-}
-```
-
-A request to this RPC might look like:
-
-```yang
-get-config {
-  schema snabb-softwire-v1;
-  path "/foo";
-}
-```
-
-As you can see, non-mandatory inputs can be left out.  A response
-might look like:
-
-```yang
-get-config {
-  config "blah blah blah";
-}
-```
-
-Responses are prefixed by the RPC name.  One message can include a
-number of RPCs; the RPCs will be made in order.  See the
-[`snabb-config-leader-v1` YANG
-schema](../../lib/yang/snabb-config-leader-v1.yang) for full details
-of available RPCs.
-
-## Private protocol
-
-The leader maintains a configuration for the program as a whole.  As
-it gets requests, it computes the set of changes to app graphs that
-would be needed to apply that configuration.  These changes are then
-passed through the private protocol to the follower.  No response from
-the follower is necessary.
-
-In some remote or perhaps not so remote future, all Snabb apps will
-have associated YANG schemas describing their individual
-configurations.  In this happy future, the generic way to ship
-configurations from the leader to a follower is by the binary
-serialization of YANG data, implemented already in the YANG modules.
-Until then however, there is also generic Lua data without a schema.
-The private protocol supports both kinds of information transfer.
-
-In the meantime, the way to indicate that an app's configuration data
-conforms to a YANG schema is to set the `schema_name` property on the
-app's class.
-
-The private protocol consists of binary messages passed over a ring
-buffer.  A follower's leader writes to the buffer, and the follower
-reads from it.  There are no other readers or writers.  Given that a
-message may in general be unbounded in size, whereas a ring buffer is
-naturally fixed, messages which may include arbtrary-sized data may be
-forced to put that data in the filesystem, and refer to it from the
-messages in the ring buffer.  Since this file system is backed by
-`tmpfs`, stalls will be minimal.
-
-## User interface
-
-The above sections document how the leader and follower apps are
-implemented so that a data-plane developer can understand the overhead
-of run-time (re)configuration.  End users won't be typing at a UNIX
-socket though; we include the `snabb config` program as a command-line
-interface to this functionality.
-
-See [the `snabb config` documentation](../../program/config/README.md)
-for full details.
diff --git a/src/apps/config/action_codec.lua b/src/apps/config/action_codec.lua
deleted file mode 100644
index 6f04559218..0000000000
--- a/src/apps/config/action_codec.lua
+++ /dev/null
@@ -1,234 +0,0 @@
--- Use of this source code is governed by the Apache 2.0 license; see COPYING.
-
-module(...,package.seeall)
-
-local S = require("syscall")
-local lib = require("core.lib")
-local ffi = require("ffi")
-local yang = require("lib.yang.yang")
-local binary = require("lib.yang.binary")
-local shm = require("core.shm")
-
-local action_names = { 'unlink_output', 'unlink_input', 'free_link',
-                       'new_link', 'link_output', 'link_input', 'stop_app',
-                       'start_app', 'reconfig_app',
-                       'call_app_method_with_blob', 'commit', 'shutdown' }
-local action_codes = {}
-for i, name in ipairs(action_names) do action_codes[name] = i end
-
-local actions = {}
-
-function actions.unlink_output (codec, appname, linkname)
-   local appname = codec:string(appname)
-   local linkname = codec:string(linkname)
-   return codec:finish(appname, linkname)
-end
-function actions.unlink_input (codec, appname, linkname)
-   local appname = codec:string(appname)
-   local linkname = codec:string(linkname)
-   return codec:finish(appname, linkname)
-end
-function actions.free_link (codec, linkspec)
-   local linkspec = codec:string(linkspec)
-   return codec:finish(linkspec)
-end
-function actions.new_link (codec, linkspec)
-   local linkspec = codec:string(linkspec)
-   return codec:finish(linkspec)
-end
-function actions.link_output (codec, appname, linkname, linkspec)
-   local appname = codec:string(appname)
-   local linkname = codec:string(linkname)
-   local linkspec = codec:string(linkspec)
-   return codec:finish(appname, linkname, linkspec)
-end
-function actions.link_input (codec, appname, linkname, linkspec)
-   local appname = codec:string(appname)
-   local linkname = codec:string(linkname)
-   local linkspec = codec:string(linkspec)
-   return codec:finish(appname, linkname, linkspec)
-end
-function actions.stop_app (codec, appname)
-   local appname = codec:string(appname)
-   return codec:finish(appname)
-end
-function actions.start_app (codec, appname, class, arg)
-   local appname = codec:string(appname)
-   local _class = codec:class(class)
-   local config = codec:config(class, arg)
-   return codec:finish(appname, _class, config)
-end
-function actions.reconfig_app (codec, appname, class, arg)
-   local appname = codec:string(appname)
-   local _class = codec:class(class)
-   local config = codec:config(class, arg)
-   return codec:finish(appname, _class, config)
-end
-function actions.call_app_method_with_blob (codec, appname, methodname, blob)
-   local appname = codec:string(appname)
-   local methodname = codec:string(methodname)
-   local blob = codec:blob(blob)
-   return codec:finish(appname, methodname, blob)
-end
-function actions.commit (codec)
-   return codec:finish()
-end
-function actions.shutdown (codec)
-   return codec:finish()
-end
-
-local public_names = {}
-local function find_public_name(obj)
-   if public_names[obj] then return unpack(public_names[obj]) end
-   for modname, mod in pairs(package.loaded) do
-      if type(mod) == 'table' then
-         for name, val in pairs(mod) do
-            if val == obj then
-               if type(val) == 'table' and type(val.new) == 'function' then
-                  public_names[obj] = { modname, name }
-                  return modname, name
-               end
-            end
-         end
-      end
-   end
-   error('could not determine public name for object: '..tostring(obj))
-end
-
-local function random_file_name()
-   local basename = 'app-conf-'..lib.random_printable_string(160)
-   return shm.root..'/'..shm.resolve(basename)
-end
-
-local function encoder()
-   local encoder = { out = {} }
-   function encoder:uint32(len)
-      table.insert(self.out, ffi.new('uint32_t[1]', len))
-   end
-   function encoder:string(str)
-      self:uint32(#str)
-      local buf = ffi.new('uint8_t[?]', #str)
-      ffi.copy(buf, str, #str)
-      table.insert(self.out, buf)
-   end
-   function encoder:blob(blob)
-      self:uint32(ffi.sizeof(blob))
-      table.insert(self.out, blob)
-   end
-   function encoder:class(class)
-      local require_path, name = find_public_name(class)
-      self:string(require_path)
-      self:string(name)
-   end
-   function encoder:config(class, arg)
-      local file_name = random_file_name()
-      if class.yang_schema then
-         yang.compile_config_for_schema_by_name(class.yang_schema, arg,
-                                                file_name)
-      else
-         if arg == nil then arg = {} end
-         binary.compile_ad_hoc_lua_data_to_file(file_name, arg)
-      end
-      self:string(file_name)
-   end
-   function encoder:finish()
-      local size = 0
-      for _,src in ipairs(self.out) do size = size + ffi.sizeof(src) end
-      local dst = ffi.new('uint8_t[?]', size)
-      local pos = 0
-      for _,src in ipairs(self.out) do
-         ffi.copy(dst + pos, src, ffi.sizeof(src))
-         pos = pos + ffi.sizeof(src)
-      end
-      return dst, size
-   end
-   return encoder
-end
-
-function encode(action)
-   local name, args = unpack(action)
-   local codec = encoder()
-   codec:uint32(assert(action_codes[name], name))
-   return assert(actions[name], name)(codec, unpack(args))
-end
-
-local uint32_ptr_t = ffi.typeof('uint32_t*')
-local function decoder(buf, len)
-   local decoder = { buf=buf, len=len, pos=0 }
-   function decoder:read(count)
-      local ret = self.buf + self.pos
-      self.pos = self.pos + count
-      assert(self.pos <= self.len)
-      return ret
-   end
-   function decoder:uint32()
-      return ffi.cast(uint32_ptr_t, self:read(4))[0]
-   end
-   function decoder:string()
-      local len = self:uint32()
-      return ffi.string(self:read(len), len)
-   end
-   function decoder:blob()
-      local len = self:uint32()
-      local blob = ffi.new('uint8_t[?]', len)
-      ffi.copy(blob, self:read(len), len)
-      return blob
-   end
-   function decoder:class()
-      local require_path, name = self:string(), self:string()
-      return assert(require(require_path)[name])
-   end
-   function decoder:config()
-      return binary.load_compiled_data_file(self:string()).data
-   end
-   function decoder:finish(...)
-      return { ... }
-   end
-   return decoder
-end
-
-function decode(buf, len)
-   local codec = decoder(buf, len)
-   local name = assert(action_names[codec:uint32()])
-   return { name, assert(actions[name], name)(codec) }
-end
-
-function selftest ()
-   print('selftest: apps.config.action_codec')
-   local function serialize(data)
-      local tmp = random_file_name()
-      print('serializing to:', tmp)
-      binary.compile_ad_hoc_lua_data_to_file(tmp, data)
-      local loaded = binary.load_compiled_data_file(tmp)
-      assert(loaded.schema_name == '')
-      assert(lib.equal(data, loaded.data))
-      os.remove(tmp)
-   end
-   serialize('foo')
-   serialize({foo='bar'})
-   serialize({foo={qux='baz'}})
-   serialize(1)
-   serialize(1LL)
-   local function test_action(action)
-      local encoded, len = encode(action)
-      local decoded = decode(encoded, len)
-      assert(lib.equal(action, decoded))
-   end
-   local appname, linkname, linkspec = 'foo', 'bar', 'foo.a -> bar.q'
-   local class, arg = require('apps.basic.basic_apps').Tee, {}
-   -- Because lib.equal only returns true when comparing cdata of
-   -- exactly the same type, here we have to use uint8_t[?].
-   local methodname, blob = 'zog', ffi.new('uint8_t[?]', 3, 1, 2, 3)
-   test_action({'unlink_output', {appname, linkname}})
-   test_action({'unlink_input', {appname, linkname}})
-   test_action({'free_link', {linkspec}})
-   test_action({'new_link', {linkspec}})
-   test_action({'link_output', {appname, linkname, linkspec}})
-   test_action({'link_input', {appname, linkname, linkspec}})
-   test_action({'stop_app', {appname}})
-   test_action({'start_app', {appname, class, arg}})
-   test_action({'reconfig_app', {appname, class, arg}})
-   test_action({'call_app_method_with_blob', {appname, methodname, blob}})
-   test_action({'commit', {}})
-   print('selftest: ok')
-end
diff --git a/src/apps/config/alarm_codec.lua b/src/apps/config/alarm_codec.lua
deleted file mode 100644
index 301092622a..0000000000
--- a/src/apps/config/alarm_codec.lua
+++ /dev/null
@@ -1,314 +0,0 @@
--- Use of this source code is governed by the Apache 2.0 license; see COPYING.
-
-module(...,package.seeall)
-
-local S = require("syscall")
-local channel = require("apps.config.channel")
-local ffi = require("ffi")
-
-local UINT32_MAX = 0xffffffff
-
-local alarm_names = { 'raise_alarm', 'clear_alarm', 'add_to_inventory', 'declare_alarm' }
-local alarm_codes = {}
-for i, name in ipairs(alarm_names) do alarm_codes[name] = i end
-
-local alarms = {}
-
-function alarms.raise_alarm (codec, resource, alarm_type_id, alarm_type_qualifier,
-   perceived_severity, alarm_text)
-
-   local resource = codec:string(resource)
-   local alarm_type_id = codec:string(alarm_type_id)
-   local alarm_type_qualifier = codec:string(alarm_type_qualifier)
-
-   local perceived_severity = codec:maybe_string(perceived_severity)
-   local alarm_text = codec:maybe_string(alarm_text)
-
-   return codec:finish(resource, alarm_type_id, alarm_type_qualifier,
-                       perceived_severity, alarm_text)
-end
-function alarms.clear_alarm (codec, resource, alarm_type_id, alarm_type_qualifier)
-   local resource = codec:string(resource)
-   local alarm_type_id = codec:string(alarm_type_id)
-   local alarm_type_qualifier = codec:string(alarm_type_qualifier)
-
-   return codec:finish(resource, alarm_type_id, alarm_type_qualifier)
-end
-function alarms.add_to_inventory (codec, alarm_type_id, alarm_type_qualifier,
-   resource, has_clear, description)
-
-   local alarm_type_id = codec:string(alarm_type_id)
-   local alarm_type_qualifier = codec:maybe_string(alarm_type_qualifier)
-
-   local resource = codec:string(resource)
-   local has_clear = codec:string((has_clear and "true" or "false"))
-   local description = codec:maybe_string(description)
-
-   return codec:finish(alarm_type_id, alarm_type_qualifier,
-                       resource, has_clear, description)
-end
-function alarms.declare_alarm (codec, resource, alarm_type_id, alarm_type_qualifier,
-   perceived_severity, alarm_text)
-
-   local resource = codec:string(resource)
-   local alarm_type_id = codec:string(alarm_type_id)
-   local alarm_type_qualifier = codec:maybe_string(alarm_type_qualifier)
-
-   local perceived_severity = codec:maybe_string(perceived_severity)
-   local alarm_text = codec:maybe_string(alarm_text)
-
-   return codec:finish(resource, alarm_type_id, alarm_type_qualifier,
-                       perceived_severity, alarm_text)
-end
-
-local function encoder()
-   local encoder = { out = {} }
-   function encoder:uint32(len)
-      table.insert(self.out, ffi.new('uint32_t[1]', len))
-   end
-   function encoder:string(str)
-      self:uint32(#str)
-      local buf = ffi.new('uint8_t[?]', #str)
-      ffi.copy(buf, str, #str)
-      table.insert(self.out, buf)
-   end
-   function encoder:maybe_string(str)
-      if str == nil then
-         self:uint32(UINT32_MAX)
-      else
-         self:string(str)
-      end
-   end
-   function encoder:finish()
-      local size = 0
-      for _,src in ipairs(self.out) do size = size + ffi.sizeof(src) end
-      local dst = ffi.new('uint8_t[?]', size)
-      local pos = 0
-      for _,src in ipairs(self.out) do
-         ffi.copy(dst + pos, src, ffi.sizeof(src))
-         pos = pos + ffi.sizeof(src)
-      end
-      return dst, size
-   end
-   return encoder
-end
-
-function encode_raise_alarm (...)
-   local codec = encoder()
-   codec:uint32(assert(alarm_codes['raise_alarm']))
-   return assert(alarms['raise_alarm'])(codec, ...)
-end
-
-function encode_clear_alarm (...)
-   local codec = encoder()
-   codec:uint32(assert(alarm_codes['clear_alarm']))
-   return assert(alarms['clear_alarm'])(codec, ...)
-end
-
-function encode_add_to_inventory (...)
-   local codec = encoder()
-   codec:uint32(assert(alarm_codes['add_to_inventory']))
-   return assert(alarms['add_to_inventory'])(codec, ...)
-end
-
-function encode_declare_alarm (...)
-   local codec = encoder()
-   codec:uint32(assert(alarm_codes['declare_alarm']))
-   return assert(alarms['declare_alarm'])(codec, ...)
-end
-
-local uint32_ptr_t = ffi.typeof('uint32_t*')
-local function decoder(buf, len)
-   local decoder = { buf=buf, len=len, pos=0 }
-   function decoder:read(count)
-      local ret = self.buf + self.pos
-      self.pos = self.pos + count
-      assert(self.pos <= self.len)
-      return ret
-   end
-   function decoder:uint32()
-      return ffi.cast(uint32_ptr_t, self:read(4))[0]
-   end
-   function decoder:string()
-      local len = self:uint32()
-      return ffi.string(self:read(len), len)
-   end
-   function decoder:maybe_string()
-      local len = self:uint32()
-      if len == UINT32_MAX then return nil end
-      return ffi.string(self:read(len), len)
-   end
-   function decoder:finish(...)
-      return { ... }
-   end
-   return decoder
-end
-
-function decode(buf, len)
-   local codec = decoder(buf, len)
-   local name = assert(alarm_names[codec:uint32()])
-   return { name, assert(alarms[name], name)(codec) }
-end
-
----
-
-local alarms_channel
-
-function get_channel()
-   if alarms_channel then return alarms_channel end
-   local name = '/'..S.getpid()..'/alarms-follower-channel'
-   local success, value = pcall(channel.open, name)
-   if success then
-      alarms_channel = value
-   else
-      alarms_channel = channel.create('alarms-follower-channel', 1e6)
-   end
-   return alarms_channel
-end
-
-local function normalize (t, attrs)
-   t = t or {}
-   local ret = {}
-   for i, k in ipairs(attrs) do ret[i] = t[k] end
-   return unpack(ret)
-end
-
-local alarm = {
-   key_attrs = {'resource', 'alarm_type_id', 'alarm_type_qualifier'},
-   args_attrs = {'perceived_severity', 'alarm_text'},
-}
-function alarm:normalize_key (t)
-   return normalize(t, self.key_attrs)
-end
-function alarm:normalize_args (t)
-   return normalize(t, self.args_attrs)
-end
-
--- To be used by the leader to group args into key and args.
-function to_alarm (args)
-   local key = {
-      resource = args[1],
-      alarm_type_id = args[2],
-      alarm_type_qualifier = args[3],
-   }
-   local args = {
-      perceived_severity = args[4],
-      alarm_text = args[5],
-   }
-   return key, args
-end
-
-local alarm_type = {
-   key_attrs = {'alarm_type_id', 'alarm_type_qualifier'},
-   args_attrs = {'resource', 'has_clear', 'description'},
-}
-function alarm_type:normalize_key (t)
-   return normalize(t, self.key_attrs)
-end
-function alarm_type:normalize_args (t)
-   return normalize(t, self.args_attrs)
-end
-
-function to_alarm_type (args)
-   local alarm_type_id, alarm_type_qualifier, resource, has_clear, description = unpack(args)
-   local key = {
-      alarm_type_id = args[1],
-      alarm_type_qualifier = args[2],
-   }
-   local args = {
-      resource = args[3],
-      has_clear = args[4],
-      description = args[5],
-   }
-   return key, args
-end
-
-function raise_alarm (key, args)
-   local channel = get_channel()
-   if channel then
-      local resource, alarm_type_id, alarm_type_qualifier = alarm:normalize_key(key)
-      local perceived_severity, alarm_text = alarm:normalize_args(args)
-      local buf, len = encode_raise_alarm(
-         resource, alarm_type_id, alarm_type_qualifier,
-         perceived_severity, alarm_text
-      )
-      channel:put_message(buf, len)
-   end
-end
-
-function clear_alarm (key)
-   local channel = get_channel()
-   if channel then
-      local resource, alarm_type_id, alarm_type_qualifier = alarm:normalize_key(key)
-      local buf, len = encode_clear_alarm(resource, alarm_type_id, alarm_type_qualifier)
-      channel:put_message(buf, len)
-   end
-end
-
-function add_to_inventory (key, args)
-   local channel = get_channel()
-   if channel then
-      local alarm_type_id, alarm_type_qualifier = alarm_type:normalize_key(key)
-      local resource, has_clear, description = alarm_type:normalize_args(args)
-      local buf, len = encode_add_to_inventory(
-         alarm_type_id, alarm_type_qualifier,
-         resource, has_clear, description
-      )
-      channel:put_message(buf, len)
-   end
-end
-
-function declare_alarm (key, args)
-   local channel = get_channel()
-   if channel then
-      local resource, alarm_type_id, alarm_type_qualifier = alarm:normalize_key(key)
-      local perceived_severity, alarm_text = alarm:normalize_args(args)
-      local buf, len = encode_declare_alarm(
-         resource, alarm_type_id, alarm_type_qualifier,
-         perceived_severity, alarm_text
-      )
-      channel:put_message(buf, len)
-   end
-end
-
-function selftest ()
-   print('selftest: apps.config.alarm_codec')
-   local lib = require("core.lib")
-   local function test_alarm (name, args)
-      local encoded, len
-      if name == 'raise_alarm' then
-         encoded, len = encode_raise_alarm(unpack(args))
-      elseif name == 'clear_alarm' then
-         encoded, len = encode_clear_alarm(unpack(args))
-      else
-         error('not valid alarm name: '..alarm)
-      end
-      local decoded = decode(encoded, len)
-      assert(lib.equal({name, args}, decoded))
-   end
-   local function test_raise_alarm ()
-      local key = {resource='res1', alarm_type_id='type1', alarm_type_qualifier=''}
-      local args = {perceived_severity='critical'}
-
-      local resource, alarm_type_id, alarm_type_qualifier = alarm:normalize_key(key)
-      local perceived_severity, alarm_text = alarm:normalize_args(args)
-      local alarm = {resource, alarm_type_id, alarm_type_qualifier,
-                     perceived_severity, alarm_text}
-
-      test_alarm('raise_alarm', alarm)
-   end
-   local function test_clear_alarm ()
-      local key = {resource='res1', alarm_type_id='type1', alarm_type_qualifier=''}
-      local resource, alarm_type_id, alarm_type_qualifier = alarm:normalize_key(key)
-      local alarm = {resource, alarm_type_id, alarm_type_qualifier}
-      test_alarm('clear_alarm', alarm)
-   end
-
-   test_raise_alarm()
-   test_clear_alarm()
-
-   local a, b = normalize({b='foo'}, {'a', 'b'})
-   assert(a == nil and b == 'foo')
-
-   print('selftest: ok')
-end
diff --git a/src/apps/config/channel.lua b/src/apps/config/channel.lua
deleted file mode 100644
index 49c94c2186..0000000000
--- a/src/apps/config/channel.lua
+++ /dev/null
@@ -1,231 +0,0 @@
--- Use of this source code is governed by the Apache 2.0 license; see COPYING.
-
-module(...,package.seeall)
-
--- A channel is a ring buffer used by the config leader app to send
--- updates to a follower.  Each follower has its own ring buffer and is
--- the only reader to the buffer.  The config leader is the only writer
--- to these buffers also.  The ring buffer is just bytes; putting a
--- message onto the buffer will write a header indicating the message
--- size, then the bytes of the message.  The channel ring buffer is
--- mapped into shared memory.  Access to a channel will never block or
--- cause a system call.
-
-local ffi = require('ffi')
-local S = require("syscall")
-local lib = require('core.lib')
-local shm = require('core.shm')
-
-local ring_buffer_t = ffi.typeof([[struct {
-   uint32_t read;
-   uint32_t write;
-   uint32_t size;
-   uint8_t buf[?];
-}]])
-
--- Q: Why not just use shm.map?
--- A: We need a variable-sized mapping.
-local function create_ring_buffer (name, size)
-   local path = shm.resolve(name)
-   shm.mkdir(lib.dirname(path))
-   path = shm.root..'/'..path
-   local fd, err = S.open(path, "creat, rdwr, excl", '0664')
-   if not fd then
-      err = tostring(err or "unknown error")
-      error('error creating file "'..path..'": '..err)
-   end
-   local len = ffi.sizeof(ring_buffer_t, size)
-   assert(fd:ftruncate(len), "ring buffer: ftruncate failed")
-   local mem, err = S.mmap(nil, len, "read, write", "shared", fd, 0)
-   fd:close()
-   if mem == nil then error("mmap failed: " .. tostring(err)) end
-   mem = ffi.cast(ffi.typeof("$*", ring_buffer_t), mem)
-   ffi.gc(mem, function (ptr) S.munmap(ptr, len) end)
-   mem.size = size
-   return mem
-end
-
-local function open_ring_buffer (name)
-   local path = shm.resolve(name)
-   path = shm.root..'/'..path
-   local fd, err = S.open(path, "rdwr")
-   if not fd then
-      err = tostring(err or "unknown error")
-      error('error opening file "'..path..'": '..err)
-   end
-   local stat = S.fstat(fd)
-   local len = stat and stat.size
-   if len < ffi.sizeof(ring_buffer_t, 0) then
-      error("unexpected size for ring buffer")
-   end
-   local mem, err = S.mmap(nil, len, "read, write", "shared", fd, 0)
-   fd:close()
-   if mem == nil then error("mmap failed: " .. tostring(err)) end
-   mem = ffi.cast(ffi.typeof("$*", ring_buffer_t), mem)
-   ffi.gc(mem, function (ptr) S.munmap(ptr, len) end)
-   if len ~= ffi.sizeof(ring_buffer_t, mem.size) then
-      error("unexpected ring buffer size: "..tostring(len))
-   end
-   return mem
-end
-
-local function to_uint32 (num)
-   local buf = ffi.new('uint32_t[1]')
-   buf[0] = num
-   return buf[0]
-end
-
-local function read_avail (ring)
-   lib.compiler_barrier()
-   return to_uint32(ring.write - ring.read)
-end
-
-local function write_avail (ring)
-   return ring.size - read_avail(ring)
-end
-
-Channel = {}
-
--- Messages typically encode up to 3 or 4 strings like app names, link
--- names, module names, or the like.  All of that and the length headers
--- probably fits within 256 bytes per message certainly.  So make room
--- for around 4K messages, why not.
-local default_buffer_size = 1024*1024
-function create(name, size)
-   local ret = {}
-   size = size or default_buffer_size
-   ret.ring_buffer = create_ring_buffer(name, size)
-   return setmetatable(ret, {__index=Channel})
-end
-
-function open(name)
-   local ret = {}
-   ret.ring_buffer = open_ring_buffer(name)
-   return setmetatable(ret, {__index=Channel})
-end
-
--- The coordination needed between the reader and the writer is that:
---
---  1. If the reader sees a a bumped write pointer, that the data written
---     to the ring buffer will be available to the reader, i.e. the writer
---     has done whatever is needed to synchronize the data.
---
---  2. It should be possible for the reader to update the read pointer
---     without stompling other memory, notably the write pointer.
---
---  3. It should be possible for the writer to update the write pointer
---     without stompling other memory, notably the read pointer.
---
---  4. Updating a write pointer or a read pointer should eventually be
---     visible to the reader or writer, respectively.
---
--- The full memory barrier after updates to the read or write pointer
--- ensures (1).  The x86 memory model, and the memory model of C11,
--- guarantee (2) and (3).  For (4), the memory barrier on the writer
--- side ensures that updates to the read or write pointers are
--- eventually visible to other CPUs, but we also have to insert a
--- compiler barrier before reading them to prevent LuaJIT from caching
--- their value somewhere else, like in a register.  See
--- https://www.kernel.org/doc/Documentation/memory-barriers.txt for more
--- discussion on memory models, and
--- http://www.freelists.org/post/luajit/Compiler-loadstore-barrier-volatile-pointer-barriers-in-general,3
--- for more on compiler barriers in LuaJIT.
---
--- If there are multiple readers or writers, they should serialize their
--- accesses through some other mechanism.
---
-
--- Put some bytes onto the channel, but without updating the write
--- pointer.  Precondition: the caller has checked that COUNT bytes are
--- indeed available for writing.
-function Channel:put_bytes(bytes, count, offset)
-   offset = offset or 0
-   local ring = self.ring_buffer
-   local start = (ring.write + offset) % ring.size
-   if start + count > ring.size then
-      local head = ring.size - start
-      ffi.copy(ring.buf + start, bytes, head)
-      ffi.copy(ring.buf, bytes + head, count - head)
-   else
-      ffi.copy(ring.buf + start, bytes, count)
-   end
-end
-
--- Peek some bytes into the channel.  If the COUNT bytes are contiguous,
--- return a pointer into the channel.  Otherwise allocate a buffer for
--- those bytes and return that.  Precondition: the caller has checked
--- that COUNT bytes are indeed available for reading.
-function Channel:peek_bytes(count, offset)
-   offset = offset or 0
-   local ring = self.ring_buffer
-   local start = (ring.read + offset) % ring.size
-   local len
-   if start + count > ring.size then
-      local buf = ffi.new('uint8_t[?]', count)
-      local head_count = ring.size - start
-      local tail_count = count - head_count
-      ffi.copy(buf, ring.buf + start, head_count)
-      ffi.copy(buf + head_count, ring.buf, tail_count)
-      return buf
-   else
-      return ring.buf + start
-   end
-end
-
-function Channel:put_message(bytes, count)
-   local ring = self.ring_buffer
-   if write_avail(ring) < count + 4 then return false end
-   self:put_bytes(ffi.cast('uint8_t*', ffi.new('uint32_t[1]', count)), 4)
-   self:put_bytes(bytes, count, 4)
-   ring.write = ring.write + count + 4
-   ffi.C.full_memory_barrier()
-   return true;
-end
-
-function Channel:peek_payload_len()
-   local ring = self.ring_buffer
-   local avail = read_avail(ring)
-   local count = 4
-   if avail < count then return nil end
-   local len = ffi.cast('uint32_t*', self:peek_bytes(4))[0]
-   if avail < count + len then return nil end
-   return len
-end
-
-function Channel:peek_message()
-   local payload_len = self:peek_payload_len()
-   if not payload_len then return nil, nil end
-   return self:peek_bytes(payload_len, 4), payload_len
-end
-
-function Channel:discard_message(payload_len)
-   local ring = self.ring_buffer
-   ring.read = ring.read + payload_len + 4
-   ffi.C.full_memory_barrier()
-end
-
-function selftest()
-   print('selftest: apps.config.channel')
-   local msg_t = ffi.typeof('struct { uint8_t a; uint8_t b; }')
-   local ch = create('test/config-channel', (4+2)*16 + 1)
-   local function put(i)
-      return ch:put_message(ffi.new('uint8_t[2]', {i, i+16}), 2)
-   end
-   for _=1,4 do
-      for i=1,16 do assert(put(i)) end
-      assert(not put(17))
-      local function assert_pop(i)
-         local msg, len = ch:peek_message()
-         assert(msg)
-         assert(len == 2)
-         assert(msg[0] == i)
-         assert(msg[1] == i + 16)
-         ch:discard_message(len)
-      end
-      assert_pop(1)
-      assert(put(17))
-      for i=2,17 do assert_pop(i) end
-      assert(not ch:peek_message())
-   end
-   print('selftest: channel ok')
-end
diff --git a/src/apps/config/follower.lua b/src/apps/config/follower.lua
deleted file mode 100644
index 3a1f8c6cc7..0000000000
--- a/src/apps/config/follower.lua
+++ /dev/null
@@ -1,96 +0,0 @@
--- Use of this source code is governed by the Apache 2.0 license; see COPYING.
-
-module(...,package.seeall)
-
-local S = require("syscall")
-local ffi = require("ffi")
-local yang = require("lib.yang.yang")
-local rpc = require("lib.yang.rpc")
-local app = require("core.app")
-local shm = require("core.shm")
-local app_graph = require("core.config")
-local channel = require("apps.config.channel")
-local action_codec = require("apps.config.action_codec")
-local alarm_codec = require("apps.config.alarm_codec")
-
-Follower = {
-   config = {
-      Hz = {default=1000},
-   }
-}
-
-function Follower:new (conf)
-   local ret = setmetatable({}, {__index=Follower})
-   ret.period = 1/conf.Hz
-   ret.next_time = app.now()
-   ret.channel = channel.create('config-follower-channel', 1e6)
-   ret.alarms_channel = alarm_codec.get_channel()
-   ret.pending_actions = {}
-   return ret
-end
-
-function Follower:shutdown()
-   -- This will shutdown everything.
-   engine.configure(app_graph.new())
-
-   -- Now we can exit.
-   S.exit(0)
-end
-
-function Follower:commit_pending_actions()
-   local to_apply = {}
-   local should_flush = false
-   for _,action in ipairs(self.pending_actions) do
-      local name, args = unpack(action)
-      if name == 'call_app_method_with_blob' then
-         if #to_apply > 0 then
-            app.apply_config_actions(to_apply)
-            to_apply = {}
-         end
-         local callee, method, blob = unpack(args)
-         local obj = assert(app.app_table[callee])
-         assert(obj[method])(obj, blob)
-      elseif name == "shutdown" then
-         self:shutdown()
-      else
-         if name == 'start_app' or name == 'reconfig_app' then
-            should_flush = true
-         end
-         table.insert(to_apply, action)
-      end
-   end
-   if #to_apply > 0 then app.apply_config_actions(to_apply) end
-   self.pending_actions = {}
-   if should_flush then require('jit').flush() end
-end
-
-function Follower:handle_actions_from_leader()
-   local channel = self.channel
-   for i=1,4 do
-      local buf, len = channel:peek_message()
-      if not buf then break end
-      local action = action_codec.decode(buf, len)
-      if action[1] == 'commit' then
-         self:commit_pending_actions()
-      else
-         table.insert(self.pending_actions, action)
-      end
-      channel:discard_message(len)
-   end
-end
-
-function Follower:pull ()
-   if app.now() < self.next_time then return end
-   self.next_time = app.now() + self.period
-   self:handle_actions_from_leader()
-end
-
-function selftest ()
-   print('selftest: apps.config.follower')
-   local c = config.new()
-   config.app(c, "follower", Follower, {})
-   engine.configure(c)
-   engine.main({ duration = 0.0001, report = {showapps=true,showlinks=true}})
-   engine.configure(config.new())
-   print('selftest: ok')
-end
diff --git a/src/apps/config/leader.lua b/src/apps/config/leader.lua
deleted file mode 100644
index 9bf5f4cdd7..0000000000
--- a/src/apps/config/leader.lua
+++ /dev/null
@@ -1,956 +0,0 @@
--- Use of this source code is governed by the Apache 2.0 license; see COPYING.
-
-module(...,package.seeall)
-
-local S = require("syscall")
-local ffi = require("ffi")
-local lib = require("core.lib")
-local cltable = require("lib.cltable")
-local cpuset = require("lib.cpuset")
-local yang = require("lib.yang.yang")
-local data = require("lib.yang.data")
-local util = require("lib.yang.util")
-local schema = require("lib.yang.schema")
-local rpc = require("lib.yang.rpc")
-local state = require("lib.yang.state")
-local path_mod = require("lib.yang.path")
-local app = require("core.app")
-local shm = require("core.shm")
-local worker = require("core.worker")
-local app_graph = require("core.config")
-local action_codec = require("apps.config.action_codec")
-local alarm_codec = require("apps.config.alarm_codec")
-local support = require("apps.config.support")
-local channel = require("apps.config.channel")
-local alarms = require("lib.yang.alarms")
-
-Leader = {
-   config = {
-      socket_file_name = {default='config-leader-socket'},
-      setup_fn = {required=true},
-      -- Could relax this requirement.
-      initial_configuration = {required=true},
-      schema_name = {required=true},
-      worker_start_code = {required=true},
-      default_schema = {},
-      cpuset = {default=cpuset.global_cpuset()},
-      Hz = {default=100},
-   }
-}
-
-local function open_socket (file)
-   S.signal('pipe', 'ign')
-   local socket = assert(S.socket("unix", "stream, nonblock"))
-   S.unlink(file) --unlink to avoid EINVAL on bind()
-   local sa = S.t.sockaddr_un(file)
-   assert(socket:bind(sa))
-   assert(socket:listen())
-   return socket
-end
-
-function Leader:new (conf)
-   local ret = setmetatable({}, {__index=Leader})
-   ret.cpuset = conf.cpuset
-   ret.socket_file_name = conf.socket_file_name
-   if not ret.socket_file_name:match('^/') then
-      local instance_dir = shm.root..'/'..tostring(S.getpid())
-      ret.socket_file_name = instance_dir..'/'..ret.socket_file_name
-   end
-   ret.schema_name = conf.schema_name
-   ret.default_schema = conf.default_schema or conf.schema_name
-   ret.support = support.load_schema_config_support(conf.schema_name)
-   ret.socket = open_socket(ret.socket_file_name)
-   ret.peers = {}
-   ret.setup_fn = conf.setup_fn
-   ret.period = 1/conf.Hz
-   ret.next_time = app.now()
-   ret.worker_start_code = conf.worker_start_code
-   ret.followers = {}
-   ret.rpc_callee = rpc.prepare_callee('snabb-config-leader-v1')
-   ret.rpc_handler = rpc.dispatch_handler(ret, 'rpc_')
-
-   ret:set_initial_configuration(conf.initial_configuration)
-
-   return ret
-end
-
-function Leader:set_initial_configuration (configuration)
-   self.current_configuration = configuration
-   self.current_in_place_dependencies = {}
-
-   -- Start the followers and configure them.
-   local worker_app_graphs = self.setup_fn(configuration)
-
-   -- Calculate the dependences
-   self.current_in_place_dependencies =
-      self.support.update_mutable_objects_embedded_in_app_initargs (
-	    {}, worker_app_graphs, self.schema_name, 'load',
-            '/', self.current_configuration)
-
-   -- Iterate over followers starting the workers and queuing up actions.
-   for id, worker_app_graph in pairs(worker_app_graphs) do
-      self:start_follower_for_graph(id, worker_app_graph)
-   end
-end
-
-function Leader:start_worker(cpu)
-   local start_code = { self.worker_start_code }
-   if cpu then
-      table.insert(start_code, 1, "print('Bound data plane to CPU:',"..cpu..")")
-      table.insert(start_code, 1, "require('lib.numa').bind_to_cpu("..cpu..")")
-   end
-   return worker.start("follower", table.concat(start_code, "\n"))
-end
-
-function Leader:stop_worker(id)
-   -- Tell the worker to terminate
-   local stop_actions = {{'shutdown', {}}, {'commit', {}}}
-   self:enqueue_config_actions_for_follower(id, stop_actions)
-   self:send_messages_to_followers()
-   self.followers[id].shutting_down = true
-end
-
-function Leader:remove_stale_followers()
-   local stale = {}
-   for id, follower in pairs(self.followers) do
-      if follower.shutting_down then
-	 if S.waitpid(follower.pid, S.c.W["NOHANG"]) ~= 0 then
-	    stale[#stale + 1] = id
-	 end
-      end
-   end
-   for _, id in ipairs(stale) do
-      if self.followers[id].cpu then
-	 self.cpuset:release(self.followers[id].cpu)
-      end
-      self.followers[id] = nil
-
-   end
-end
-
-function Leader:acquire_cpu_for_follower(id, app_graph)
-   local pci_addresses = {}
-   -- Grovel through app initargs for keys named "pciaddr".  Hacky!
-   for name, init in pairs(app_graph.apps) do
-      if type(init.arg) == 'table' then
-         for k, v in pairs(init.arg) do
-            if k == 'pciaddr' then table.insert(pci_addresses, v) end
-         end
-      end
-   end
-   return self.cpuset:acquire_for_pci_addresses(pci_addresses)
-end
-
-function Leader:start_follower_for_graph(id, graph)
-   local cpu = self:acquire_cpu_for_follower(id, graph)
-   self.followers[id] = { cpu=cpu, pid=self:start_worker(cpu), queue={},
-                          graph=graph }
-   local actions = self.support.compute_config_actions(
-      app_graph.new(), self.followers[id].graph, {}, 'load')
-   self:enqueue_config_actions_for_follower(id, actions)
-   return self.followers[id]
-end
-
-function Leader:take_follower_message_queue ()
-   local actions = self.config_action_queue
-   self.config_action_queue = nil
-   return actions
-end
-
-local verbose = os.getenv('SNABB_LEADER_VERBOSE') and true
-
-function Leader:enqueue_config_actions_for_follower(follower, actions)
-   for _,action in ipairs(actions) do
-      if verbose then print('encode', action[1], unpack(action[2])) end
-      local buf, len = action_codec.encode(action)
-      table.insert(self.followers[follower].queue, { buf=buf, len=len })
-   end
-end
-
-function Leader:enqueue_config_actions (actions)
-   for id,_ in pairs(self.followers) do
-      self.enqueue_config_actions_for_follower(id, actions)
-   end
-end
-
-function Leader:rpc_describe (args)
-   local alternate_schemas = {}
-   for schema_name, translator in pairs(self.support.translators) do
-      table.insert(alternate_schemas, schema_name)
-   end
-   return { native_schema = self.schema_name,
-	    default_schema = self.default_schema,
-            alternate_schema = alternate_schemas,
-            capability = schema.get_default_capabilities() }
-end
-
-local function path_printer_for_grammar(grammar, path, format, print_default)
-   local getter, subgrammar = path_mod.resolver(grammar, path)
-   local printer
-   if format == "xpath" then
-      printer = data.xpath_printer_from_grammar(subgrammar, print_default, path)
-   else
-      printer = data.data_printer_from_grammar(subgrammar, print_default)
-   end
-   return function(data, file)
-      return printer(getter(data), file)
-   end
-end
-
-local function path_printer_for_schema(schema, path, is_config,
-                                       format, print_default)
-   local grammar = data.data_grammar_from_schema(schema, is_config)
-   return path_printer_for_grammar(grammar, path, format, print_default)
-end
-
-local function path_printer_for_schema_by_name(schema_name, path, is_config,
-                                               format, print_default)
-   local schema = yang.load_schema_by_name(schema_name)
-   return path_printer_for_schema(schema, path, is_config, format,
-                                  print_default)
-end
-
-function Leader:rpc_get_config (args)
-   local function getter()
-      if args.schema ~= self.schema_name then
-         return self:foreign_rpc_get_config(
-            args.schema, args.path, args.format, args.print_default)
-      end
-      local printer = path_printer_for_schema_by_name(
-         args.schema, args.path, true, args.format, args.print_default)
-      local config = printer(self.current_configuration, yang.string_output_file())
-      return { config = config }
-   end
-   local success, response = pcall(getter)
-   if success then return response else return {status=1, error=response} end
-end
-
-function Leader:rpc_set_alarm_operator_state (args)
-   local function getter()
-      if args.schema ~= self.schema_name then
-         return false, ("Set-operator-state operation not supported in"..
-                        "'%s' schema"):format(args.schema)
-      end
-      local key = {resource=args.resource, alarm_type_id=args.alarm_type_id,
-                   alarm_type_qualifier=args.alarm_type_qualifier}
-      local params = {state=args.state, text=args.text}
-      return { success = alarms.set_operator_state(key, params) }
-   end
-   local success, response = pcall(getter)
-   if success then return response else return {status=1, error=response} end
-end
-
-function Leader:rpc_purge_alarms (args)
-   local function purge()
-      if args.schema ~= self.schema_name then
-         return false, ("Purge-alarms operation not supported in"..
-                        "'%s' schema"):format(args.schema)
-      end
-      return { purged_alarms = alarms.purge_alarms(args) }
-   end
-   local success, response = pcall(purge)
-   if success then return response else return {status=1, error=response} end
-end
-
-function Leader:rpc_compress_alarms (args)
-   local function compress()
-      if args.schema ~= self.schema_name then
-         return false, ("Compress-alarms operation not supported in"..
-                        "'%s' schema"):format(args.schema)
-      end
-      return { compressed_alarms = alarms.compress_alarms(args) }
-   end
-   local success, response = pcall(compress)
-   if success then return response else return {status=1, error=response} end
-end
-
-
-local function path_parser_for_grammar(grammar, path)
-   local getter, subgrammar = path_mod.resolver(grammar, path)
-   return data.data_parser_from_grammar(subgrammar)
-end
-
-local function path_parser_for_schema(schema, path)
-   local grammar = data.config_grammar_from_schema(schema)
-   return path_parser_for_grammar(grammar, path)
-end
-
-local function path_parser_for_schema_by_name(schema_name, path)
-   return path_parser_for_schema(yang.load_schema_by_name(schema_name), path)
-end
-
-local function path_setter_for_grammar(grammar, path)
-   if path == "/" then
-      return function(config, subconfig) return subconfig end
-   end
-   local head, tail = lib.dirname(path), lib.basename(path)
-   local tail_path = path_mod.parse_path(tail)
-   local tail_name, query = tail_path[1].name, tail_path[1].query
-   if lib.equal(query, {}) then
-      -- No query; the simple case.
-      local getter, grammar = path_mod.resolver(grammar, head)
-      assert(grammar.type == 'struct')
-      local tail_id = data.normalize_id(tail_name)
-      return function(config, subconfig)
-         getter(config)[tail_id] = subconfig
-         return config
-      end
-   end
-
-   -- Otherwise the path ends in a query; it must denote an array or
-   -- table item.
-   local getter, grammar = path_mod.resolver(grammar, head..'/'..tail_name)
-   if grammar.type == 'array' then
-      local idx = path_mod.prepare_array_lookup(query)
-      return function(config, subconfig)
-         local array = getter(config)
-         assert(idx <= #array)
-         array[idx] = subconfig
-         return config
-      end
-   elseif grammar.type == 'table' then
-      local key = path_mod.prepare_table_lookup(grammar.keys,
-                                                grammar.key_ctype, query)
-      if grammar.string_key then
-         key = key[data.normalize_id(grammar.string_key)]
-         return function(config, subconfig)
-            local tab = getter(config)
-            assert(tab[key] ~= nil)
-            tab[key] = subconfig
-            return config
-         end
-      elseif grammar.key_ctype and grammar.value_ctype then
-         return function(config, subconfig)
-            getter(config):update(key, subconfig)
-            return config
-         end
-      elseif grammar.key_ctype then
-         return function(config, subconfig)
-            local tab = getter(config)
-            assert(tab[key] ~= nil)
-            tab[key] = subconfig
-            return config
-         end
-      else
-         return function(config, subconfig)
-            local tab = getter(config)
-            for k,v in pairs(tab) do
-               if lib.equal(k, key) then
-                  tab[k] = subconfig
-                  return config
-               end
-            end
-            error("Not found")
-         end
-      end
-   else
-      error('Query parameters only allowed on arrays and tables')
-   end
-end
-
-local function path_setter_for_schema(schema, path)
-   local grammar = data.config_grammar_from_schema(schema)
-   return path_setter_for_grammar(grammar, path)
-end
-
-function compute_set_config_fn (schema_name, path)
-   return path_setter_for_schema(yang.load_schema_by_name(schema_name), path)
-end
-
-local function path_adder_for_grammar(grammar, path)
-   local top_grammar = grammar
-   local getter, grammar = path_mod.resolver(grammar, path)
-   if grammar.type == 'array' then
-      if grammar.ctype then
-         -- It's an FFI array; have to create a fresh one, sadly.
-         local setter = path_setter_for_grammar(top_grammar, path)
-         local elt_t = data.typeof(grammar.ctype)
-         local array_t = ffi.typeof('$[?]', elt_t)
-         return function(config, subconfig)
-            local cur = getter(config)
-            local new = array_t(#cur + #subconfig)
-            local i = 1
-            for _,elt in ipairs(cur) do new[i-1] = elt; i = i + 1 end
-            for _,elt in ipairs(subconfig) do new[i-1] = elt; i = i + 1 end
-            return setter(config, util.ffi_array(new, elt_t))
-         end
-      end
-      -- Otherwise we can add entries in place.
-      return function(config, subconfig)
-         local cur = getter(config)
-         for _,elt in ipairs(subconfig) do table.insert(cur, elt) end
-         return config
-      end
-   elseif grammar.type == 'table' then
-      -- Invariant: either all entries in the new subconfig are added,
-      -- or none are.
-      if grammar.key_ctype and grammar.value_ctype then
-         -- ctable.
-         return function(config, subconfig)
-            local ctab = getter(config)
-            for entry in subconfig:iterate() do
-               if ctab:lookup_ptr(entry.key) ~= nil then
-                  error('already-existing entry')
-               end
-            end
-            for entry in subconfig:iterate() do
-               ctab:add(entry.key, entry.value)
-            end
-            return config
-         end
-      elseif grammar.string_key or grammar.key_ctype then
-         -- cltable or string-keyed table.
-         local pairs = grammar.key_ctype and cltable.pairs or pairs
-         return function(config, subconfig)
-            local tab = getter(config)
-            for k,_ in pairs(subconfig) do
-               if tab[k] ~= nil then error('already-existing entry') end
-            end
-            for k,v in pairs(subconfig) do tab[k] = v end
-            return config
-         end
-      else
-         -- Sad quadratic loop.
-         return function(config, subconfig)
-            local tab = getter(config)
-            for key,val in pairs(tab) do
-               for k,_ in pairs(subconfig) do
-                  if lib.equal(key, k) then
-                     error('already-existing entry', key)
-                  end
-               end
-            end
-            for k,v in pairs(subconfig) do tab[k] = v end
-            return config
-         end
-      end
-   else
-      error('Add only allowed on arrays and tables')
-   end
-end
-
-local function path_adder_for_schema(schema, path)
-   local grammar = data.config_grammar_from_schema(schema)
-   return path_adder_for_grammar(grammar, path)
-end
-
-function compute_add_config_fn (schema_name, path)
-   return path_adder_for_schema(yang.load_schema_by_name(schema_name), path)
-end
-compute_add_config_fn = util.memoize(compute_add_config_fn)
-
-local function path_remover_for_grammar(grammar, path)
-   local top_grammar = grammar
-   local head, tail = lib.dirname(path), lib.basename(path)
-   local tail_path = path_mod.parse_path(tail)
-   local tail_name, query = tail_path[1].name, tail_path[1].query
-   local head_and_tail_name = head..'/'..tail_name
-   local getter, grammar = path_mod.resolver(grammar, head_and_tail_name)
-   if grammar.type == 'array' then
-      if grammar.ctype then
-         -- It's an FFI array; have to create a fresh one, sadly.
-         local idx = path_mod.prepare_array_lookup(query)
-         local setter = path_setter_for_grammar(top_grammar, head_and_tail_name)
-         local elt_t = data.typeof(grammar.ctype)
-         local array_t = ffi.typeof('$[?]', elt_t)
-         return function(config)
-            local cur = getter(config)
-            assert(idx <= #cur)
-            local new = array_t(#cur - 1)
-            for i,elt in ipairs(cur) do
-               if i < idx then new[i-1] = elt end
-               if i > idx then new[i-2] = elt end
-            end
-            return setter(config, util.ffi_array(new, elt_t))
-         end
-      end
-      -- Otherwise we can remove the entry in place.
-      return function(config)
-         local cur = getter(config)
-         assert(i <= #cur)
-         table.remove(cur, i)
-         return config
-      end
-   elseif grammar.type == 'table' then
-      local key = path_mod.prepare_table_lookup(grammar.keys,
-                                                grammar.key_ctype, query)
-      if grammar.string_key then
-         key = key[data.normalize_id(grammar.string_key)]
-         return function(config)
-            local tab = getter(config)
-            assert(tab[key] ~= nil)
-            tab[key] = nil
-            return config
-         end
-      elseif grammar.key_ctype and grammar.value_ctype then
-         return function(config)
-            getter(config):remove(key)
-            return config
-         end
-      elseif grammar.key_ctype then
-         return function(config)
-            local tab = getter(config)
-            assert(tab[key] ~= nil)
-            tab[key] = nil
-            return config
-         end
-      else
-         return function(config)
-            local tab = getter(config)
-            for k,v in pairs(tab) do
-               if lib.equal(k, key) then
-                  tab[k] = nil
-                  return config
-               end
-            end
-            error("Not found")
-         end
-      end
-   else
-      error('Remove only allowed on arrays and tables')
-   end
-end
-
-local function path_remover_for_schema(schema, path)
-   local grammar = data.config_grammar_from_schema(schema)
-   return path_remover_for_grammar(grammar, path)
-end
-
-function compute_remove_config_fn (schema_name, path)
-   return path_remover_for_schema(yang.load_schema_by_name(schema_name), path)
-end
-
-function Leader:notify_pre_update (config, verb, path, ...)
-   for _,translator in pairs(self.support.translators) do
-      translator.pre_update(config, verb, path, ...)
-   end
-end
-
-function Leader:update_configuration (update_fn, verb, path, ...)
-   self:notify_pre_update(self.current_configuration, verb, path, ...)
-   local to_restart =
-      self.support.compute_apps_to_restart_after_configuration_update (
-         self.schema_name, self.current_configuration, verb, path,
-         self.current_in_place_dependencies, ...)
-   local new_config = update_fn(self.current_configuration, ...)
-   local new_graphs = self.setup_fn(new_config, ...)
-   for id, graph in pairs(new_graphs) do
-      if self.followers[id] == nil then
-	 self:start_follower_for_graph(id, graph)
-      end
-   end
-
-   for id, follower in pairs(self.followers) do
-      if new_graphs[id] == nil then
-         self:stop_worker(id)
-      else
-	 local actions = self.support.compute_config_actions(
-	    follower.graph, new_graphs[id], to_restart, verb, path, ...)
-	 self:enqueue_config_actions_for_follower(id, actions)
-	 follower.graph = new_graphs[id]
-      end
-   end
-   self.current_configuration = new_config
-   self.current_in_place_dependencies =
-      self.support.update_mutable_objects_embedded_in_app_initargs (
-         self.current_in_place_dependencies, new_graphs, verb, path, ...)
-end
-
-function Leader:handle_rpc_update_config (args, verb, compute_update_fn)
-   local path = path_mod.normalize_path(args.path)
-   local parser = path_parser_for_schema_by_name(args.schema, path)
-   self:update_configuration(compute_update_fn(args.schema, path),
-                             verb, path, parser(args.config))
-   return {}
-end
-
-function Leader:get_native_state ()
-   local states = {}
-   local state_reader = self.support.compute_state_reader(self.schema_name)
-   for _, follower in pairs(self.followers) do
-      local follower_config = self.support.configuration_for_follower(
-         follower, self.current_configuration)
-      table.insert(states, state_reader(follower.pid, follower_config))
-   end
-   return self.support.process_states(states)
-end
-
-function Leader:get_translator (schema_name)
-   local translator = self.support.translators[schema_name]
-   if translator then return translator end
-   error('unsupported schema: '..schema_name)
-end
-function Leader:apply_translated_rpc_updates (updates)
-   for _,update in ipairs(updates) do
-      local verb, args = unpack(update)
-      local method = assert(self['rpc_'..verb..'_config'])
-      method(self, args)
-   end
-   return {}
-end
-function Leader:foreign_rpc_get_config (schema_name, path, format,
-                                        print_default)
-   path = path_mod.normalize_path(path)
-   local translate = self:get_translator(schema_name)
-   local foreign_config = translate.get_config(self.current_configuration)
-   local printer = path_printer_for_schema_by_name(
-      schema_name, path, true, format, print_default)
-   local config = printer(foreign_config, yang.string_output_file())
-   return { config = config }
-end
-function Leader:foreign_rpc_get_state (schema_name, path, format,
-                                       print_default)
-   path = path_mod.normalize_path(path)
-   local translate = self:get_translator(schema_name)
-   local foreign_state = translate.get_state(self:get_native_state())
-   local printer = path_printer_for_schema_by_name(
-      schema_name, path, false, format, print_default)
-   local state = printer(foreign_state, yang.string_output_file())
-   return { state = state }
-end
-function Leader:foreign_rpc_set_config (schema_name, path, config_str)
-   path = path_mod.normalize_path(path)
-   local translate = self:get_translator(schema_name)
-   local parser = path_parser_for_schema_by_name(schema_name, path)
-   local updates = translate.set_config(self.current_configuration, path,
-                                        parser(config_str))
-   return self:apply_translated_rpc_updates(updates)
-end
-function Leader:foreign_rpc_add_config (schema_name, path, config_str)
-   path = path_mod.normalize_path(path)
-   local translate = self:get_translator(schema_name)
-   local parser = path_parser_for_schema_by_name(schema_name, path)
-   local updates = translate.add_config(self.current_configuration, path,
-                                        parser(config_str))
-   return self:apply_translated_rpc_updates(updates)
-end
-function Leader:foreign_rpc_remove_config (schema_name, path)
-   path = path_mod.normalize_path(path)
-   local translate = self:get_translator(schema_name)
-   local updates = translate.remove_config(self.current_configuration, path)
-   return self:apply_translated_rpc_updates(updates)
-end
-
-function Leader:rpc_set_config (args)
-   local function setter()
-      if self.listen_peer ~= nil and self.listen_peer ~= self.rpc_peer then
-         error('Attempt to modify configuration while listener attached')
-      end
-      if args.schema ~= self.schema_name then
-         return self:foreign_rpc_set_config(args.schema, args.path, args.config)
-      end
-      return self:handle_rpc_update_config(args, 'set', compute_set_config_fn)
-   end
-   local success, response = pcall(setter)
-   if success then return response else return {status=1, error=response} end
-end
-
-function Leader:rpc_add_config (args)
-   local function adder()
-      if self.listen_peer ~= nil and self.listen_peer ~= self.rpc_peer then
-         error('Attempt to modify configuration while listener attached')
-      end
-      if args.schema ~= self.schema_name then
-         return self:foreign_rpc_add_config(args.schema, args.path, args.config)
-      end
-      return self:handle_rpc_update_config(args, 'add', compute_add_config_fn)
-   end
-   local success, response = pcall(adder)
-   if success then return response else return {status=1, error=response} end
-end
-
-function Leader:rpc_remove_config (args)
-   local function remover()
-      if self.listen_peer ~= nil and self.listen_peer ~= self.rpc_peer then
-         error('Attempt to modify configuration while listener attached')
-      end
-      if args.schema ~= self.schema_name then
-         return self:foreign_rpc_remove_config(args.schema, args.path)
-      end
-      local path = path_mod.normalize_path(args.path)
-      self:update_configuration(compute_remove_config_fn(args.schema, path),
-                              'remove', path)
-      return {}
-   end
-   local success, response = pcall(remover)
-   if success then return response else return {status=1, error=response} end
-end
-
-function Leader:rpc_attach_listener (args)
-   local function attacher()
-      if self.listen_peer ~= nil then error('Listener already attached') end
-      self.listen_peer = self.rpc_peer
-      return {}
-   end
-   local success, response = pcall(attacher)
-   if success then return response else return {status=1, error=response} end
-end
-
-function Leader:rpc_get_state (args)
-   local function getter()
-      if args.schema ~= self.schema_name then
-         return self:foreign_rpc_get_state(args.schema, args.path,
-                                           args.format, args.print_default)
-      end
-      local state = self:get_native_state()
-      local printer = path_printer_for_schema_by_name(
-         self.schema_name, args.path, false, args.format, args.print_default)
-      return { state = printer(state, yang.string_output_file()) }
-   end
-   local success, response = pcall(getter)
-   if success then return response else return {status=1, error=response} end
-end
-
-function Leader:rpc_get_alarms_state (args)
-   local function getter()
-      assert(args.schema == "ietf-alarms")
-      local printer = path_printer_for_schema_by_name(
-         args.schema, args.path, false, args.format, args.print_default)
-      local state = {
-         alarms = alarms.get_state()
-      }
-      state = printer(state, yang.string_output_file())
-      return { state = state }
-   end
-   local success, response = pcall(getter)
-   if success then return response else return {status=1, error=response} end
-end
-
-function Leader:handle (payload)
-   return rpc.handle_calls(self.rpc_callee, payload, self.rpc_handler)
-end
-
-local dummy_unix_sockaddr = S.t.sockaddr_un()
-
-function Leader:handle_calls_from_peers()
-   local peers = self.peers
-   while true do
-      local fd, err = self.socket:accept(dummy_unix_sockaddr)
-      if not fd then
-         if err.AGAIN then break end
-         assert(nil, err)
-      end
-      fd:nonblock()
-      table.insert(peers, { state='length', len=0, fd=fd })
-   end
-   local i = 1
-   while i <= #peers do
-      local peer = peers[i]
-      local visit_peer_again = false
-      while peer.state == 'length' do
-         local ch, err = peer.fd:read(nil, 1)
-         if not ch then
-            if err.AGAIN then break end
-            peer.state = 'error'
-            peer.msg = tostring(err)
-         elseif ch == '\n' then
-            peer.pos = 0
-            peer.buf = ffi.new('uint8_t[?]', peer.len)
-            peer.state = 'payload'
-         elseif tonumber(ch) then
-            peer.len = peer.len * 10 + tonumber(ch)
-            if peer.len > 1e8 then
-               peer.state = 'error'
-               peer.msg = 'length too long: '..peer.len
-            end
-         elseif ch == '' then
-            if peer.len == 0 then
-               peer.state = 'done'
-            else
-               peer.state = 'error'
-               peer.msg = 'unexpected EOF'
-            end
-         else
-            peer.state = 'error'
-            peer.msg = 'unexpected character: '..ch
-         end
-      end
-      while peer.state == 'payload' do
-         if peer.pos == peer.len then
-            peer.state = 'ready'
-            peer.payload = ffi.string(peer.buf, peer.len)
-            peer.buf, peer.len = nil, nil
-         else
-            local count, err = peer.fd:read(peer.buf + peer.pos,
-                                            peer.len - peer.pos)
-            if not count then
-               if err.AGAIN then break end
-               peer.state = 'error'
-               peer.msg = tostring(err)
-            elseif count == 0 then
-               peer.state = 'error'
-               peer.msg = 'short read'
-            else
-               peer.pos = peer.pos + count
-               assert(peer.pos <= peer.len)
-            end
-         end
-      end
-      while peer.state == 'ready' do
-         -- Uncomment to get backtraces.
-         self.rpc_peer = peer
-         -- local success, reply = true, self:handle(peer.payload)
-         local success, reply = pcall(self.handle, self, peer.payload)
-         self.rpc_peer = nil
-         peer.payload = nil
-         if success then
-            assert(type(reply) == 'string')
-            reply = #reply..'\n'..reply
-            peer.state = 'reply'
-            peer.buf = ffi.new('uint8_t[?]', #reply+1, reply)
-            peer.pos = 0
-            peer.len = #reply
-         else
-            peer.state = 'error'
-            peer.msg = reply
-         end
-      end
-      while peer.state == 'reply' do
-         if peer.pos == peer.len then
-            visit_peer_again = true
-            peer.state = 'length'
-            peer.buf, peer.pos = nil, nil
-            peer.len = 0
-         else
-            local count, err = peer.fd:write(peer.buf + peer.pos,
-                                             peer.len - peer.pos)
-            if not count then
-               if err.AGAIN then break end
-               peer.state = 'error'
-               peer.msg = tostring(err)
-            elseif count == 0 then
-               peer.state = 'error'
-               peer.msg = 'short write'
-            else
-               peer.pos = peer.pos + count
-               assert(peer.pos <= peer.len)
-            end
-         end
-      end
-      if peer.state == 'done' or peer.state == 'error' then
-         if peer.state == 'error' then print('error: '..peer.msg) end
-         peer.fd:close()
-         table.remove(peers, i)
-         if self.listen_peer == peer then self.listen_peer = nil end
-      elseif not visit_peer_again then
-         i = i + 1
-      end
-   end
-end
-
-function Leader:send_messages_to_followers()
-   for _,follower in pairs(self.followers) do
-      if not follower.channel then
-         local name = '/'..tostring(follower.pid)..'/config-follower-channel'
-         local success, channel = pcall(channel.open, name)
-         if success then follower.channel = channel end
-      end
-      local channel = follower.channel
-      if channel then
-         local queue = follower.queue
-         follower.queue = {}
-         local requeue = false
-         for _,msg in ipairs(queue) do
-            if not requeue then
-               requeue = not channel:put_message(msg.buf, msg.len)
-            end
-            if requeue then table.insert(follower.queue, msg) end
-         end
-      end
-   end
-end
-
-function Leader:pull ()
-   if app.now() < self.next_time then return end
-   self.next_time = app.now() + self.period
-   self:remove_stale_followers()
-   self:handle_calls_from_peers()
-   self:send_messages_to_followers()
-   self:receive_alarms_from_followers()
-end
-
-function Leader:receive_alarms_from_followers ()
-   for _,follower in pairs(self.followers) do
-      self:receive_alarms_from_follower(follower)
-   end
-end
-
-function Leader:receive_alarms_from_follower (follower)
-   if not follower.alarms_channel then
-      local name = '/'..tostring(follower.pid)..'/alarms-follower-channel'
-      local success, channel = pcall(channel.open, name)
-      if not success then return end
-      follower.alarms_channel = channel
-   end
-   local channel = follower.alarms_channel
-   while true do
-      local buf, len = channel:peek_message()
-      if not buf then break end
-      local alarm = alarm_codec.decode(buf, len)
-      self:handle_alarm(follower, alarm)
-      channel:discard_message(len)
-   end
-end
-
-function Leader:handle_alarm (follower, alarm)
-   local fn, args = unpack(alarm)
-   if fn == 'raise_alarm' then
-      local key, args = alarm_codec.to_alarm(args)
-      alarms.raise_alarm(key, args)
-   end
-   if fn == 'clear_alarm' then
-      local key = alarm_codec.to_alarm(args)
-      alarms.clear_alarm(key)
-   end
-   if fn == 'add_to_inventory' then
-      local key, args = alarm_codec.to_alarm_type(args)
-      alarms.do_add_to_inventory(key, args)
-   end
-   if fn == 'declare_alarm' then
-      local key, args = alarm_codec.to_alarm(args)
-      alarms.do_declare_alarm(key, args)
-   end
-end
-
-function Leader:stop ()
-   for _,peer in ipairs(self.peers) do peer.fd:close() end
-   self.peers = {}
-   self.socket:close()
-   S.unlink(self.socket_file_name)
-end
-
-function test_worker()
-   local follower = require("apps.config.follower")
-   local myconf = config.new()
-   config.app(myconf, "follower", follower.Follower, {})
-   app.configure(myconf)
-   app.busywait = true
-   app.main({})
-end
-
-function selftest ()
-   print('selftest: apps.config.leader')
-   local graph = app_graph.new()
-   local function setup_fn(cfg)
-      local graph = app_graph.new()
-      local basic_apps = require('apps.basic.basic_apps')
-      app_graph.app(graph, "source", basic_apps.Source, {})
-      app_graph.app(graph, "sink", basic_apps.Sink, {})
-      app_graph.link(graph, "source.foo -> sink.bar")
-      return {graph}
-   end
-   local worker_start_code = "require('apps.config.leader').test_worker()"
-   app_graph.app(graph, "leader", Leader,
-                 {setup_fn=setup_fn, worker_start_code=worker_start_code,
-                  -- Use a schema with no data nodes, just for
-                  -- testing.
-                  schema_name='ietf-inet-types', initial_configuration={}})
-   engine.configure(graph)
-   engine.main({ duration = 0.05, report = {showapps=true,showlinks=true}})
-   assert(app.app_table.leader.followers[1])
-   assert(app.app_table.leader.followers[1].graph.links)
-   assert(app.app_table.leader.followers[1].graph.links["source.foo -> sink.bar"])
-   local link = app.link_table["source.foo -> sink.bar"]
-   engine.configure(app_graph.new())
-   print('selftest: ok')
-end
diff --git a/src/apps/config/support.lua b/src/apps/config/support.lua
deleted file mode 100644
index 5633adbba3..0000000000
--- a/src/apps/config/support.lua
+++ /dev/null
@@ -1,231 +0,0 @@
--- Use of this source code is governed by the Apache 2.0 license; see COPYING.
-
-module(...,package.seeall)
-
-local app = require("core.app")
-local app_graph_mod = require("core.config")
-local path_mod = require("lib.yang.path")
-local yang = require("lib.yang.yang")
-local data = require("lib.yang.data")
-local cltable = require("lib.cltable")
-
-function compute_parent_paths(path)
-   local function sorted_keys(t)
-      local ret = {}
-      for k, v in pairs(t) do table.insert(ret, k) end
-      table.sort(ret)
-      return ret
-   end
-   local ret = { '/' }
-   local head = ''
-   for _,part in ipairs(path_mod.parse_path(path)) do
-      head = head..'/'..part.name
-      table.insert(ret, head)
-      local keys = sorted_keys(part.query)
-      if #keys > 0 then
-         for _,k in ipairs(keys) do
-            head = head..'['..k..'='..part.query[k]..']'
-         end
-         table.insert(ret, head)
-      end
-   end
-   return ret
-end
-
-local function add_child_objects(accum, grammar, config)
-   local visitor = {}
-   local function visit(grammar, config)
-      assert(visitor[grammar.type])(grammar, config)
-   end
-   local function visit_child(grammar, config)
-      if grammar.type == 'scalar' then return end
-      table.insert(accum, config)
-      return visit(grammar, config)
-   end
-   function visitor.table(grammar, config)
-      -- Ctables are raw data, and raw data doesn't contain children
-      -- with distinct identity.
-      if grammar.key_ctype and grammar.value_ctype then return end
-      local child_grammar = {type="struct", members=grammar.values,
-                             ctype=grammar.value_ctype}
-      if grammar.key_ctype then
-         for k, v in cltable.pairs(config) do visit_child(child_grammar, v) end
-      else
-         for k, v in pairs(config) do visit_child(child_grammar, v) end
-      end
-   end
-   function visitor.array(grammar, config)
-      -- Children are leaves; nothing to do.
-   end
-   function visitor.struct(grammar, config)
-      -- Raw data doesn't contain children with distinct identity.
-      if grammar.ctype then return end
-      for k,grammar in pairs(grammar.members) do
-         local id = data.normalize_id(k)
-         local child = config[id]
-         if child ~= nil then visit_child(grammar, child) end
-      end
-   end
-   return visit(grammar, config)
-end
-
-local function compute_objects_maybe_updated_in_place (schema_name, config,
-                                                       changed_path)
-   local schema = yang.load_schema_by_name(schema_name)
-   local grammar = data.config_grammar_from_schema(schema)
-   local objs = {}
-   local getter, subgrammar
-   for _,path in ipairs(compute_parent_paths(changed_path)) do
-      -- Calling the getter is avg O(N) in depth, so that makes the
-      -- loop O(N^2), though it is generally bounded at a shallow
-      -- level so perhaps it's OK.  path_mod.resolver is O(N) too but
-      -- memoization makes it O(1).
-      getter, subgrammar = path_mod.resolver(grammar, path)
-      -- Scalars can't be updated in place.
-      if subgrammar.type == 'scalar' then return objs end
-      table.insert(objs, getter(config))
-      -- Members of raw data can't be updated in place either.
-      if subgrammar.type == 'table' then
-         if subgrammar.key_ctype and subgrammar.value_ctype then return objs end
-      elseif subgrammar.type == 'struct' then
-         if subgrammar.ctype then return objs end
-      end
-   end
-   -- If the loop above finished normally, then it means that the
-   -- object at changed_path might contain in-place-updatable objects
-   -- inside of it, so visit its children.
-   add_child_objects(objs, subgrammar, objs[#objs])
-   return objs
-end
-
-local function record_mutable_objects_embedded_in_app_initarg (follower_id, app_name, obj, accum)
-   local function record(obj)
-      local tab = accum[obj]
-      if not tab then
-         tab = {}
-         accum[obj] = tab
-      end
-      if tab[follower_id] == nil then
-         tab[follower_id] = {app_name}
-      else
-         table.insert(tab[follower_id], app_name)
-      end
-   end
-   local function visit(obj)
-      if type(obj) == 'table' then
-         record(obj)
-         for _,v in pairs(obj) do visit(v) end
-      elseif type(obj) == 'cdata' then
-         record(obj)
-         -- Cdata contains sub-objects but they don't have identity;
-         -- it's only the cdata object itself that has identity.
-      else
-         -- Other object kinds can't be updated in place.
-      end
-   end
-   visit(obj)
-end
-
--- Takes a table of follower ids (app_graph_map) and returns a tabl≈e which has
--- the follower id as the key and a table listing all app names
---   i.e. {follower_id => {app name, ...}, ...}
-local function compute_mutable_objects_embedded_in_app_initargs (app_graph_map)
-   local deps = {}
-   for id, app_graph in pairs(app_graph_map) do
-      for name, info in pairs(app_graph.apps) do
-         record_mutable_objects_embedded_in_app_initarg(id, name, info.arg, deps)
-      end
-   end
-   return deps
-end
-
-local function compute_apps_to_restart_after_configuration_update (
-      schema_name, configuration, verb, changed_path, in_place_dependencies, arg)
-   local maybe_updated = compute_objects_maybe_updated_in_place(
-      schema_name, configuration, changed_path)
-   local needs_restart = {}
-   for _,place in ipairs(maybe_updated) do
-      for _, id in ipairs(in_place_dependencies[place] or {}) do
-         if needs_restart[id] == nil then needs_restart[id] = {} end
-         for _, appname in ipairs(in_place_dependencies[place][id] or {}) do
-            needs_restart[id][appname] = true
-         end
-      end
-   end
-   return needs_restart
-end
-
-local function add_restarts(actions, app_graph, to_restart)
-   for _,action in ipairs(actions) do
-      local name, args = unpack(action)
-      if name == 'stop_app' or name == 'reconfig_app' then
-         local appname = args[1]
-         to_restart[appname] = nil
-      end
-   end
-   local to_relink = {}
-   for id, apps in pairs(to_restart) do
-      for appname, _ in pairs(apps) do
-         local info = assert(app_graph.apps[appname])
-         local class, arg = info.class, info.arg
-         if class.reconfig then
-            table.insert(actions, {'reconfig_app', {appname, class, arg}})
-         else
-            table.insert(actions, {'stop_app', {appname}})
-            table.insert(actions, {'start_app', {appname, class, arg}})
-            to_relink[appname] = true
-         end
-      end
-   end
-   for linkspec,_ in pairs(app_graph.links) do
-      local fa, fl, ta, tl = app_graph_mod.parse_link(linkspec)
-      if to_relink[fa] then
-         table.insert(actions, {'link_output', {fa, fl, linkspec}})
-      end
-      if to_relink[ta] then
-         table.insert(actions, {'link_input', {ta, tl, linkspec}})
-      end
-   end
-   table.insert(actions, {'commit', {}})
-   return actions
-end
-
-local function configuration_for_follower(follower, configuration)
-   return configuration
-end
-
-local function compute_state_reader(schema_name)
-   return function(pid)
-      local reader = state.state_reader_from_schema_by_name(schema_name)
-      return reader(state.counters_for_pid(pid))
-   end
-end
-
-local function process_states(states)
-   return states[1]
-end
-
-generic_schema_config_support = {
-   compute_config_actions = function(
-         old_graph, new_graph, to_restart, verb, path, ...)
-      return add_restarts(app.compute_config_actions(old_graph, new_graph),
-                          new_graph, to_restart)
-   end,
-   update_mutable_objects_embedded_in_app_initargs = function(
-         in_place_dependencies, app_graph, schema_name, verb, path, arg)
-      return compute_mutable_objects_embedded_in_app_initargs(app_graph)
-   end,
-   compute_state_reader = compute_state_reader,
-   configuration_for_follower = configuration_for_follower,
-   process_states = process_states,
-   compute_apps_to_restart_after_configuration_update =
-      compute_apps_to_restart_after_configuration_update,
-   translators = {}
-}
-
-function load_schema_config_support(schema_name)
-   local mod_name = 'apps.config.support.'..schema_name:gsub('-', '_')
-   local success, support_mod = pcall(require, mod_name)
-   if success then return support_mod.get_config_support() end
-   return generic_schema_config_support
-end
diff --git a/src/apps/config/support/snabb-softwire-v2.lua b/src/apps/config/support/snabb-softwire-v2.lua
deleted file mode 100644
index 651dcc53e7..0000000000
--- a/src/apps/config/support/snabb-softwire-v2.lua
+++ /dev/null
@@ -1,706 +0,0 @@
--- Use of this source code is governed by the Apache 2.0 license; see COPYING.
-module(..., package.seeall)
-local ffi = require('ffi')
-local app = require('core.app')
-local corelib = require('core.lib')
-local equal = require('core.lib').equal
-local dirname = require('core.lib').dirname
-local data = require('lib.yang.data')
-local state = require('lib.yang.state')
-local ipv4_ntop = require('lib.yang.util').ipv4_ntop
-local ipv6 = require('lib.protocol.ipv6')
-local yang = require('lib.yang.yang')
-local ctable = require('lib.ctable')
-local cltable = require('lib.cltable')
-local path_mod = require('lib.yang.path')
-local generic = require('apps.config.support').generic_schema_config_support
-local binding_table = require("apps.lwaftr.binding_table")
-
-local binding_table_instance
-local function get_binding_table_instance(conf)
-   if binding_table_instance == nil then
-      binding_table_instance = binding_table.load(conf)
-   end
-   return binding_table_instance
-end
-
--- Packs snabb-softwire-v2 softwire entry into softwire and PSID blob
---
--- The data plane stores a separate table of psid maps and softwires. It
--- requires that we give it a blob it can quickly add. These look rather
--- similar to snabb-softwire-v1 structures however it maintains the br-address
--- on the softwire so are subtly different.
-local function pack_softwire(app_graph, entry)
-   assert(app_graph.apps['lwaftr'])
-   assert(entry.value.port_set, "Softwire lacks port-set definition")
-   local key, value = entry.key, entry.value
-   
-   -- Get the binding table
-   local bt_conf = app_graph.apps.lwaftr.arg.softwire_config.binding_table
-   bt = get_binding_table_instance(bt_conf)
-
-   local softwire_t = bt.softwires.entry_type()
-   psid_map_t = bt.psid_map.entry_type()
-
-   -- Now lets pack the stuff!
-   local packed_softwire = ffi.new(softwire_t)
-   packed_softwire.key.ipv4 = key.ipv4
-   packed_softwire.key.psid = key.psid
-   packed_softwire.value.b4_ipv6 = value.b4_ipv6
-   packed_softwire.value.br_address = value.br_address
-
-   local packed_psid_map = ffi.new(psid_map_t)
-   packed_psid_map.key.addr = key.ipv4
-   if value.port_set.psid_length then
-      packed_psid_map.value.psid_length = value.port_set.psid_length
-   end
-
-   return packed_softwire, packed_psid_map
-end
-
-local function add_softwire_entry_actions(app_graph, entries)
-   assert(app_graph.apps['lwaftr'])
-   local bt_conf = app_graph.apps.lwaftr.arg.softwire_config.binding_table
-   local bt = get_binding_table_instance(bt_conf)
-   local ret = {}
-   for entry in entries:iterate() do
-      local psoftwire, ppsid = pack_softwire(app_graph, entry)
-      assert(bt:is_managed_ipv4_address(psoftwire.key.ipv4))
-
-      local softwire_args = {'lwaftr', 'add_softwire_entry', psoftwire}
-      table.insert(ret, {'call_app_method_with_blob', softwire_args})
-   end
-   table.insert(ret, {'commit', {}})
-   return ret
-end
-
-local softwire_grammar
-local function get_softwire_grammar()
-   if not softwire_grammar then
-      local schema = yang.load_schema_by_name('snabb-softwire-v2')
-      local grammar = data.config_grammar_from_schema(schema)
-      softwire_grammar =
-         assert(grammar.members['softwire-config'].
-                   members['binding-table'].members['softwire'])
-   end
-   return softwire_grammar
-end
-
-local function remove_softwire_entry_actions(app_graph, path)
-   assert(app_graph.apps['lwaftr'])
-   path = path_mod.parse_path(path)
-   local grammar = get_softwire_grammar()
-   local key = path_mod.prepare_table_lookup(
-      grammar.keys, grammar.key_ctype, path[#path].query)
-   local args = {'lwaftr', 'remove_softwire_entry', key}
-   -- If it's the last softwire for the corresponding psid entry, remove it.
-   -- TODO: check if last psid entry and then remove.
-   return {{'call_app_method_with_blob', args}, {'commit', {}}}
-end
-
-local function compute_config_actions(old_graph, new_graph, to_restart,
-                                      verb, path, arg)
-   -- If the binding cable changes, remove our cached version.
-   if path ~= nil and path:match("^/softwire%-config/binding%-table") then
-      binding_table_instance = nil
-   end
-
-   if verb == 'add' and path == '/softwire-config/binding-table/softwire' then
-      if to_restart == false then
-	 return add_softwire_entry_actions(new_graph, arg)
-      end
-   elseif (verb == 'remove' and
-           path:match('^/softwire%-config/binding%-table/softwire')) then
-      return remove_softwire_entry_actions(new_graph, path)
-   elseif (verb == 'set' and path == '/softwire-config/name') then
-      return {}
-   end
-   return generic.compute_config_actions(
-      old_graph, new_graph, to_restart, verb, path, arg)
-end
-
-local function update_mutable_objects_embedded_in_app_initargs(
-      in_place_dependencies, app_graph, schema_name, verb, path, arg)
-   if verb == 'add' and path == '/softwire-config/binding-table/softwire' then
-      return in_place_dependencies
-   elseif (verb == 'remove' and
-           path:match('^/softwire%-config/binding%-table/softwire')) then
-      return in_place_dependencies
-   else
-      return generic.update_mutable_objects_embedded_in_app_initargs(
-         in_place_dependencies, app_graph, schema_name, verb, path, arg)
-   end
-end
-
-local function compute_apps_to_restart_after_configuration_update(
-      schema_name, configuration, verb, path, in_place_dependencies, arg)
-   if verb == 'add' and path == '/softwire-config/binding-table/softwire' then
-      -- We need to check if the softwire defines a new port-set, if so we need to
-      -- restart unfortunately. If not we can just add the softwire.
-      local bt = get_binding_table_instance(configuration.softwire_config.binding_table)
-      local to_restart = false
-      for entry in arg:iterate() do
-	 to_restart = (bt:is_managed_ipv4_address(entry.key.ipv4) == false) or false
-      end
-      if to_restart == false then return {} end
-   elseif (verb == 'remove' and
-           path:match('^/softwire%-config/binding%-table/softwire')) then
-      return {}
-   elseif (verb == 'set' and path == '/softwire-config/name') then
-      return {}
-   end
-   return generic.compute_apps_to_restart_after_configuration_update(
-      schema_name, configuration, verb, path, in_place_dependencies, arg)
-end
-
-local function memoize1(f)
-   local memoized_arg, memoized_result
-   return function(arg)
-      if arg == memoized_arg then return memoized_result end
-      memoized_result = f(arg)
-      memoized_arg = arg
-      return memoized_result
-   end
-end
-
-local function cltable_for_grammar(grammar)
-   assert(grammar.key_ctype)
-   assert(not grammar.value_ctype)
-   local key_t = data.typeof(grammar.key_ctype)
-   return cltable.new({key_type=key_t}), key_t
-end
-
-local ietf_br_instance_grammar
-local function get_ietf_br_instance_grammar()
-   if not ietf_br_instance_grammar then
-      local schema = yang.load_schema_by_name('ietf-softwire-br')
-      local grammar = data.config_grammar_from_schema(schema)
-      grammar = assert(grammar.members['br-instances'])
-      grammar = assert(grammar.members['br-type'])
-      grammar = assert(grammar.choices['binding'].binding)
-      grammar = assert(grammar.members['br-instance'])
-      ietf_br_instance_grammar = grammar
-   end
-   return ietf_br_instance_grammar
-end
-
-local ietf_softwire_grammar
-local function get_ietf_softwire_grammar()
-   if not ietf_softwire_grammar then
-      local grammar = get_ietf_br_instance_grammar()
-      grammar = assert(grammar.values['binding-table'])
-      grammar = assert(grammar.members['binding-entry'])
-      ietf_softwire_grammar = grammar
-   end
-   return ietf_softwire_grammar
-end
-
-local function ietf_binding_table_from_native(bt)
-   local ret, key_t = cltable_for_grammar(get_ietf_softwire_grammar())
-   for softwire in bt.softwire:iterate() do
-      local k = key_t({ binding_ipv6info = softwire.value.b4_ipv6 })
-      local v = {
-         binding_ipv4_addr = softwire.key.ipv4,
-         port_set = {
-            psid_offset = softwire.value.port_set.reserved_ports_bit_count,
-            psid_len = softwire.value.port_set.psid_length,
-            psid = softwire.key.psid
-         },
-         br_ipv6_addr = softwire.value.br_address,
-      }
-      ret[k] = v
-   end
-   return ret
-end
-
-local function schema_getter(schema_name, path)
-   local schema = yang.load_schema_by_name(schema_name)
-   local grammar = data.config_grammar_from_schema(schema)
-   return path_mod.resolver(grammar, path)
-end
-
-local function snabb_softwire_getter(path)
-   return schema_getter('snabb-softwire-v2', path)
-end
-
-local function ietf_softwire_br_getter(path)
-   return schema_getter('ietf-softwire-br', path)
-end
-
-local function native_binding_table_from_ietf(ietf)
-   local _, softwire_grammar =
-      snabb_softwire_getter('/softwire-config/binding-table/softwire')
-   local softwire_key_t = data.typeof(softwire_grammar.key_ctype)
-   local softwire = cltable.new({key_type=softwire_key_t})
-   for k,v in cltable.pairs(ietf) do
-      local softwire_key =
-         softwire_key_t({ipv4=v.binding_ipv4_addr, psid=v.port_set.psid})
-      local softwire_value = {
-         br_address=v.br_ipv6_addr,
-         b4_ipv6=k.binding_ipv6info,
-         port_set={
-            psid_length=v.port_set.psid_len,
-            reserved_ports_bit_count=v.port_set.psid_offset
-         }
-      }
-      cltable.set(softwire, softwire_key, softwire_value)
-   end
-   return {softwire=softwire}
-end
-
-local function serialize_binding_table(bt)
-   local _, grammar = snabb_softwire_getter('/softwire-config/binding-table')
-   local printer = data.data_printer_from_grammar(grammar)
-   return printer(bt, yang.string_output_file())
-end
-
-local uint64_ptr_t = ffi.typeof('uint64_t*')
-function ipv6_equals(a, b)
-   local x, y = ffi.cast(uint64_ptr_t, a), ffi.cast(uint64_ptr_t, b)
-   return x[0] == y[0] and x[1] == y[1]
-end
-
-local function ietf_softwire_br_translator ()
-   local ret = {}
-   local instance_id_map = {}
-   local cached_config
-   local function instance_id_by_device(device)
-      local last
-      for id, pciaddr in ipairs(instance_id_map) do
-	 if pciaddr == device then return id end
-	 last = id
-      end
-      if last == nil then
-	 last = 1
-      else
-	 last = last + 1
-      end
-      instance_id_map[last] = device
-      return last
-   end
-   function ret.get_config(native_config)
-      if cached_config ~= nil then return cached_config end
-      local int = native_config.softwire_config.internal_interface
-      local int_err = int.error_rate_limiting
-      local ext = native_config.softwire_config.external_interface
-      local br_instance, br_instance_key_t =
-         cltable_for_grammar(get_ietf_br_instance_grammar())
-      for device, instance in pairs(native_config.softwire_config.instance) do
-	 br_instance[br_instance_key_t({id=instance_id_by_device(device)})] = {
-	    name = native_config.softwire_config.name,
-	    tunnel_payload_mtu = int.mtu,
-	    tunnel_path_mru = ext.mtu,
-	    -- FIXME: There's no equivalent of softwire-num-threshold in
-            -- snabb-softwire-v1.
-	    softwire_num_threshold = 0xffffffff,
-            enable_hairpinning = int.hairpinning,
-	    binding_table = {
-	       binding_entry = ietf_binding_table_from_native(
-		  native_config.softwire_config.binding_table)
-	    },
-            icmp_policy = {
-               icmpv4_errors = {
-                  allow_incoming_icmpv4 = ext.allow_incoming_icmp,
-                  generate_icmpv4_errors = ext.generate_icmp_errors
-               },
-               icmpv6_errors = {
-                  generate_icmpv6_errors = int.generate_icmp_errors,
-                  icmpv6_errors_rate =
-                     math.floor(int_err.packets / int_err.period)
-               }
-            }
-	 }
-      end
-      cached_config = {
-         br_instances = {
-            binding = { br_instance = br_instance }
-         }
-      }
-      return cached_config
-   end
-   function ret.get_state(native_state)
-      -- Even though this is a different br-instance node, it is a
-      -- cltable with the same key type, so just re-use the key here.
-      local br_instance, br_instance_key_t =
-         cltable_for_grammar(get_ietf_br_instance_grammar())
-      for device, instance in pairs(native_state.softwire_config.instance) do
-         local c = instance.softwire_state
-	 br_instance[br_instance_key_t({id=instance_id_by_device(device)})] = {
-            traffic_stat = {
-               sent_ipv4_packet = c.out_ipv4_packets,
-               sent_ipv4_byte = c.out_ipv4_bytes,
-               sent_ipv6_packet = c.out_ipv6_packets,
-               sent_ipv6_byte = c.out_ipv6_bytes,
-               rcvd_ipv4_packet = c.in_ipv4_packets,
-               rcvd_ipv4_byte = c.in_ipv4_bytes,
-               rcvd_ipv6_packet = c.in_ipv6_packets,
-               rcvd_ipv6_byte = c.in_ipv6_bytes,
-               dropped_ipv4_packet = c.drop_all_ipv4_iface_packets,
-               dropped_ipv4_byte = c.drop_all_ipv4_iface_bytes,
-               dropped_ipv6_packet = c.drop_all_ipv6_iface_packets,
-               dropped_ipv6_byte = c.drop_all_ipv6_iface_bytes,
-               dropped_ipv4_fragments = 0, -- FIXME
-               dropped_ipv4_bytes = 0, -- FIXME
-               ipv6_fragments_reassembled = c.in_ipv6_frag_reassembled,
-               ipv6_fragments_bytes_reassembled = 0, -- FIXME
-               out_icmpv4_error_packets = c.out_icmpv4_error_packets,
-               out_icmpv6_error_packets = c.out_icmpv6_error_packets,
-               hairpin_ipv4_bytes = c.hairpin_ipv4_bytes,
-               hairpin_ipv4_packets = c.hairpin_ipv4_packets,
-               active_softwire_num = 0, -- FIXME
-            }
-         }
-      end
-      return {
-         br_instances = {
-            binding = { br_instance = br_instance }
-         }
-      }
-   end
-   local function sets_whole_table(path, count)
-      if #path > count then return false end
-      if #path == count then
-         for k,v in pairs(path[#path].query) do return false end
-      end
-      return true
-   end
-   function ret.set_config(native_config, path_str, arg)
-      path = path_mod.parse_path(path_str)
-      local br_instance_paths = {'br-instances', 'binding', 'br-instance'}
-      local bt_paths = {'binding-table', 'binding-entry'}
-
-      -- Can't actually set the instance itself.
-      if #path <= #br_instance_paths then
-         error("Unspported path: "..path_str)
-      end
-
-      -- Handle special br attributes (tunnel-payload-mtu, tunnel-path-mru, softwire-num-threshold).
-      if #path > #br_instance_paths then
-         local maybe_leaf = path[#path].name
-         local path_tails = {
-            ['tunnel-payload-mtu'] = 'internal-interface/mtu',
-            ['tunnel-path-mtu'] = 'external-interface/mtu',
-            ['name'] = 'name',
-            ['enable-hairpinning'] = 'internal-interface/hairpinning',
-            ['allow-incoming-icmpv4'] = 'external-interface/allow-incoming-icmp',
-            ['generate-icmpv4-errors'] = 'external-interface/generate-icmp-errors',
-            ['generate-icmpv6-errors'] = 'internal-interface/generate-icmp-errors'
-         }
-         local path_tail = path_tails[maybe_leaf]
-         if path_tail then
-            return {{'set', {schema='snabb-softwire-v2',
-                             path='/softwire-config/'..path_tail,
-                             config=tostring(arg)}}}
-         elseif maybe_leaf == 'icmpv6-errors-rate' then
-            local head = '/softwire-config/internal-interface/error-rate-limiting'
-            return {
-               {'set', {schema='snabb-softwire-v2', path=head..'/packets',
-                        config=tostring(arg * 2)}},
-               {'set', {schema='snabb-softwire-v2', path=head..'/period',
-                        config='2'}}}
-         else
-            error('unrecognized leaf: '..maybe_leaf)
-         end
-      end
-
-      -- Two kinds of updates: setting the whole binding table, or
-      -- updating one entry.
-      if sets_whole_table(path, #br_instance_paths + #bt_paths) then
-         -- Setting the whole binding table.
-         if sets_whole_table(path, #br_instance_paths) then
-            for i=#path+1,#br_instance_paths do
-               arg = arg[data.normalize_id(br_instance_paths[i])]
-            end
-            local instance
-            for k,v in cltable.pairs(arg) do
-               if instance then error('multiple instances in config') end
-               if k.id ~= 1 then error('instance id not 1: '..tostring(k.id)) end
-               instance = v
-            end
-            if not instance then error('no instances in config') end
-            arg = instance
-         end
-         for i=math.max(#path-#br_instance_paths,0)+1,#bt_paths do
-            arg = arg[data.normalize_id(bt_paths[i])]
-         end
-         local bt = native_binding_table_from_ietf(arg)
-         return {{'set', {schema='snabb-softwire-v2',
-                          path='/softwire-config/binding-table',
-                          config=serialize_binding_table(bt)}}}
-      else
-         -- An update to an existing entry.  First, get the existing entry.
-         local config = ret.get_config(native_config)
-         local entry_path = path_str
-         local entry_path_len = #br_instance_paths + #bt_paths
-         for i=entry_path_len+1, #path do
-            entry_path = dirname(entry_path)
-         end
-         local old = ietf_softwire_br_getter(entry_path)(config)
-         -- Now figure out what the new entry should look like.
-         local new
-         if #path == entry_path_len then
-            new = arg
-         else
-            new = {
-               port_set = {
-                  psid_offset = old.port_set.psid_offset,
-                  psid_len = old.port_set.psid_len,
-                  psid = old.port_set.psid
-               },
-               binding_ipv4_addr = old.binding_ipv4_addr,
-               br_ipv6_addr = old.br_ipv6_addr
-            }
-            if path[entry_path_len + 1].name == 'port-set' then
-               if #path == entry_path_len + 1 then
-                  new.port_set = arg
-               else
-                  local k = data.normalize_id(path[#path].name)
-                  new.port_set[k] = arg
-               end
-            elseif path[#path].name == 'binding-ipv4-addr' then
-               new.binding_ipv4_addr = arg
-            elseif path[#path].name == 'br-ipv6-addr' then
-               new.br_ipv6_addr = arg
-            else
-               error('bad path element: '..path[#path].name)
-            end
-         end
-         -- Apply changes.  Ensure  that the port-set
-         -- changes are compatible with the existing configuration.
-         local updates = {}
-         local softwire_path = '/softwire-config/binding-table/softwire'
-
-         -- Lets remove this softwire entry and add a new one.
-         local function q(ipv4, psid)
-            return string.format('[ipv4=%s][psid=%s]', ipv4_ntop(ipv4), psid)
-         end
-         local old_query = q(old.binding_ipv4_addr, old.port_set.psid)
-         -- FIXME: This remove will succeed but the add could fail if
-         -- there's already a softwire with this IPv4 and PSID.  We need
-         -- to add a check here that the IPv4/PSID is not present in the
-         -- binding table.
-         table.insert(updates,
-                      {'remove', {schema='snabb-softwire-v2',
-                                  path=softwire_path..old_query}})
-
-         local config_str = string.format([[{
-            ipv4 %s;
-            psid %s;
-            br-address %s;
-            b4-ipv6 %s;
-            port-set {
-               psid-length %s;
-               reserved-ports-bit-count %s;
-            }
-         }]], ipv4_ntop(new.binding_ipv4_addr), new.port_set.psid,
-              ipv6:ntop(new.br_ipv6_addr),
-              path[entry_path_len].query['binding-ipv6info'],
-              new.port_set.psid_len, new.port_set.psid_offset)
-         table.insert(updates,
-                      {'add', {schema='snabb-softwire-v2',
-                               path=softwire_path,
-                               config=config_str}})
-         return updates
-      end
-   end
-   function ret.add_config(native_config, path_str, data)
-      local binding_entry_path = {'br-instances', 'binding', 'br-instance',
-                                  'binding-table', 'binding-entry'}
-      local path = path_mod.parse_path(path_str)
-
-      if #path ~= #binding_entry_path then
-         error('unsupported path: '..path)
-      end
-      local config = ret.get_config(native_config)
-      local ietf_bt = ietf_softwire_br_getter(path_str)(config)
-      local old_bt = native_config.softwire_config.binding_table
-      local new_bt = native_binding_table_from_ietf(data)
-      local updates = {}
-      local softwire_path = '/softwire-config/binding-table/softwire'
-      local psid_map_path = '/softwire-config/binding-table/psid-map'
-      -- Add softwires.
-      local additions = {}
-      for entry in new_bt.softwire:iterate() do
-         local key, value = entry.key, entry.value
-         if old_bt.softwire:lookup_ptr(key) ~= nil then
-            error('softwire already present in table: '..
-                     inet_ntop(key.ipv4)..'/'..key.psid)
-         end
-         local config_str = string.format([[{
-            ipv4 %s;
-            psid %s;
-            br-address %s;
-            b4-ipv6 %s;
-            port-set {
-               psid-length %s;
-               reserved-ports-bit-count %s;
-            }
-         }]], ipv4_ntop(key.ipv4), key.psid,
-              ipv6:ntop(value.br_address),
-              ipv6:ntop(value.b4_ipv6),
-              value.port_set.psid_length,
-              value.port_set.reserved_ports_bit_count
-         )
-         table.insert(additions, config_str)
-      end
-      table.insert(updates,
-                   {'add', {schema='snabb-softwire-v2',
-                            path=softwire_path,
-                            config=table.concat(additions, '\n')}})
-      return updates
-   end
-   function ret.remove_config(native_config, path_str)
-      local path = path_mod.parse_path(path_str)
-      local ietf_binding_table_path = {'softwire-config', 'binding', 'br',
-         'br-instances', 'br-instance', 'binding-table'}
-      local ietf_instance_path = {'softwire-config', 'binding', 'br',
-         'br-instances', 'br-instance'}
-
-      if #path == #ietf_instance_path then
-         -- Remove appropriate instance
-         local ietf_instance_id = tonumber(assert(path[5].query).id)
-         local instance_path = "/softwire-config/instance"
-
-         -- If it's not been populated in instance_id_map this is meaningless
-         -- and dangerous as they have no mapping from snabb's "device".
-         local function q(device) return
-            string.format("[device=%s]", device)
-         end
-         local device = instance_id_map[ietf_instance_id]
-         if device then
-            return {{'remove', {schema='snabb-softwire-v2',
-                                path=instance_path..q(device)}}}
-         else
-            error(string.format(
-               "Could not find '%s' in ietf instance mapping", ietf_instance_id
-            ))
-         end
-      elseif #path == #ietf_binding_table_path then
-         local softwire_path = '/softwire-config/binding-table/softwire'
-         if path:sub(-1) ~= ']' then error('unsupported path: '..path_str) end
-         local config = ret.get_config(native_config)
-         local entry = ietf_softwire_getter(path_str)(config)
-         local function q(ipv4, psid)
-            return string.format('[ipv4=%s][psid=%s]', ipv4_ntop(ipv4), psid)
-         end
-         local query = q(entry.binding_ipv4_addr, entry.port_set.psid)
-         return {{'remove', {schema='snabb-softwire-v2',
-                             path=softwire_path..query}}}
-      else
-         return error('unsupported path: '..path_str)
-      end
-   end
-   function ret.pre_update(native_config, verb, path, data)
-      -- Given the notification that the native config is about to be
-      -- updated, make our cached config follow along if possible (and
-      -- if we have one).  Otherwise throw away our cached config; we'll
-      -- re-make it next time.
-      if cached_config == nil then return end
-      local br_instance = cached_config.br_instances.binding.br_instance
-      if (verb == 'remove' and
-          path:match('^/softwire%-config/binding%-table/softwire')) then
-         -- Remove a softwire.
-         local value = snabb_softwire_getter(path)(native_config)
-         for _,instance in cltable.pairs(br_instance) do
-            local grammar = get_ietf_softwire_grammar()
-            local key = path_mod.prepare_table_lookup(
-               grammar.keys, grammar.key_ctype, {['binding-ipv6info']='::'})
-            key.binding_ipv6info = value.b4_ipv6
-            assert(instance.binding_table.binding_entry[key] ~= nil)
-            instance.binding_table.binding_entry[key] = nil
-         end
-      elseif (verb == 'add' and
-              path == '/softwire-config/binding-table/softwire') then
-         local bt = native_config.softwire_config.binding_table
-         for k,v in cltable.pairs(
-            ietf_binding_table_from_native({softwire = data})) do
-            for _,instance in cltable.pairs(br_instance) do
-               instance.binding_table.binding_entry[k] = v
-            end
-         end
-      elseif (verb == 'set' and path == "/softwire-config/name") then
-	 local br = cached_config.softwire_config.binding.br
-	 for _, instance in cltable.pairs(br_instance) do
-	    instance.name = data
-	 end
-      else
-         cached_config = nil
-      end
-   end
-   return ret
-end
-
-local function configuration_for_follower(follower, configuration)
-   return follower.graph.apps.lwaftr.arg
-end
-
-local function compute_state_reader(schema_name)
-   -- The schema has two lists which we want to look in.
-   local schema = yang.load_schema_by_name(schema_name)
-   local grammar = data.data_grammar_from_schema(schema, false)
-
-   local instance_list_gmr = grammar.members["softwire-config"].members.instance
-   local instance_state_gmr = instance_list_gmr.values["softwire-state"]
-
-   local base_reader = state.state_reader_from_grammar(grammar)
-   local instance_state_reader = state.state_reader_from_grammar(instance_state_gmr)
-
-   return function(pid, data)
-      local counters = state.counters_for_pid(pid)
-      local ret = base_reader(counters)
-      ret.softwire_config.instance = {}
-
-      for device, instance in pairs(data.softwire_config.instance) do
-         local instance_state = instance_state_reader(counters)
-         ret.softwire_config.instance[device] = {}
-         ret.softwire_config.instance[device].softwire_state = instance_state
-      end
-
-      return ret
-   end
-end
-
-local function process_states(states)
-   -- We need to create a summation of all the states as well as adding all the
-   -- instance specific state data to create a total in software-state.
-
-   local unified = {
-      softwire_config = {instance = {}},
-      softwire_state = {}
-   }
-
-   local function total_counter(name, softwire_stats, value)
-      if softwire_stats[name] == nil then
-         return value
-      else
-         return softwire_stats[name] + value
-      end
-   end
-
-   for _, inst_config in ipairs(states) do
-      local name, instance = next(inst_config.softwire_config.instance)
-      unified.softwire_config.instance[name] = instance
-
-      for name, value in pairs(instance.softwire_state) do
-         unified.softwire_state[name] = total_counter(
-            name, unified.softwire_state, value)
-      end
-   end
-
-   return unified
-end
-
-
-function get_config_support()
-   return {
-      compute_config_actions = compute_config_actions,
-      update_mutable_objects_embedded_in_app_initargs =
-         update_mutable_objects_embedded_in_app_initargs,
-      compute_apps_to_restart_after_configuration_update =
-         compute_apps_to_restart_after_configuration_update,
-      compute_state_reader = compute_state_reader,
-      process_states = process_states,
-      configuration_for_follower = configuration_for_follower,
-      translators = { ['ietf-softwire-br'] = ietf_softwire_br_translator () }
-   }
-end

From 27280f4273e3276d8844692a67fd8a3d945ba5c9 Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@igalia.com>
Date: Fri, 8 Dec 2017 11:42:55 +0100
Subject: [PATCH 19/31] Add facility to add yang schemas to Snabb's database.

---
 src/lib/yang/README.md  | 12 ++++++++++++
 src/lib/yang/schema.lua | 19 +++++++++++++++++++
 src/lib/yang/yang.lua   |  3 +++
 3 files changed, 34 insertions(+)

diff --git a/src/lib/yang/README.md b/src/lib/yang/README.md
index f42a76b165..0df8788d47 100644
--- a/src/lib/yang/README.md
+++ b/src/lib/yang/README.md
@@ -230,6 +230,18 @@ schema itself, or as `import *name* { ... }` in other YANG modules that
 import this module.  *revision* optionally indicates that a certain
 revision data should be required.
 
+— Function **add_schema** *src* *filename*
+
+Add the YANG schema from the string *src* to Snabb's database of YANG
+schemas, making it available to `load_schema_by_name` and related
+functionality.  *filename* is used when signalling any parse errors.
+Returns the name of the newly added schema.
+
+— Function **add_schema_file** *filename*
+
+Like `add_schema`, but reads the YANG schema in from a file.  Returns
+the name of the newly added schema.
+
 — Function **load_config_for_schema** *schema* *src* *filename*
 
 Given the schema object *schema*, load the configuration from the string
diff --git a/src/lib/yang/schema.lua b/src/lib/yang/schema.lua
index a576cbeb0f..143565b4c9 100644
--- a/src/lib/yang/schema.lua
+++ b/src/lib/yang/schema.lua
@@ -941,6 +941,25 @@ function load_schema_by_name(name, revision)
 end
 load_schema_by_name = util.memoize(load_schema_by_name)
 
+function add_schema(src, filename)
+   -- Assert that the source actually parses, and get the ID.
+   local s, e = load_schema(src, filename)
+   -- Assert that this schema isn't known.
+   assert(not pcall(load_schema_by_name, s.id))
+   assert(s.id)
+   -- Intern.
+   print('lib.yang.'..s.id:gsub('-', '_')..'_yang')
+   package.loaded['lib.yang.'..s.id:gsub('-', '_')..'_yang'] = src
+   return s.id
+end
+
+function add_schema_file(filename)
+   local file_in = assert(io.open(filename))
+   local contents = file_in:read("*a")
+   file_in:close()
+   return add_schema(contents, filename)
+end
+
 function lookup_identity (fqid)
    local schema_name, id = fqid:match("^([^:]*):(.*)$")
    local schema, env = load_schema_by_name(schema_name)
diff --git a/src/lib/yang/yang.lua b/src/lib/yang/yang.lua
index 3d6a40414e..fe92e6fea4 100644
--- a/src/lib/yang/yang.lua
+++ b/src/lib/yang/yang.lua
@@ -12,6 +12,9 @@ load_schema = schema.load_schema
 load_schema_file = schema.load_schema_file
 load_schema_by_name = schema.load_schema_by_name
 
+add_schema = schema.add_schema
+add_schema_file = schema.add_schema_file
+
 load_config_for_schema = data.load_config_for_schema
 load_config_for_schema_by_name = data.load_config_for_schema_by_name
 

From 94200753eba98fd9fe238e27baa60af4aa0664bf Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@igalia.com>
Date: Mon, 11 Dec 2017 09:43:33 +0100
Subject: [PATCH 20/31] Rename lib.ptree.manager to lib.ptree.ptree

This name better matches how users will think of it.  Also add
functionality to set program name to the manager, and bind the manager
to a NUMA node as appropriate.  Unclaim the name as the manager stops.
---
 src/lib/ptree/{manager.lua => ptree.lua} | 15 +++++++++++++--
 src/program/lwaftr/bench/bench.lua       |  1 -
 src/program/lwaftr/run/run.lua           |  1 -
 src/program/lwaftr/setup.lua             |  2 +-
 4 files changed, 14 insertions(+), 5 deletions(-)
 rename src/lib/ptree/{manager.lua => ptree.lua} (99%)

diff --git a/src/lib/ptree/manager.lua b/src/lib/ptree/ptree.lua
similarity index 99%
rename from src/lib/ptree/manager.lua
rename to src/lib/ptree/ptree.lua
index e0cf2b78d8..d28e5ed8db 100644
--- a/src/lib/ptree/manager.lua
+++ b/src/lib/ptree/ptree.lua
@@ -33,6 +33,7 @@ local default_log_level = "WARN"
 if os.getenv('SNABB_MANAGER_VERBOSE') then default_log_level = "DEBUG" end
 
 local manager_config_spec = {
+   name = {},
    socket_file_name = {default='config-manager-socket'},
    setup_fn = {required=true},
    -- Could relax this requirement.
@@ -57,7 +58,9 @@ end
 
 function new_manager (conf)
    local conf = lib.parse(conf, manager_config_spec)
+
    local ret = setmetatable({}, {__index=Manager})
+   ret.name = conf.name
    ret.log_level = assert(log_levels[conf.log_level])
    ret.cpuset = conf.cpuset
    ret.socket_file_name = conf.socket_file_name
@@ -68,7 +71,6 @@ function new_manager (conf)
    ret.schema_name = conf.schema_name
    ret.default_schema = conf.default_schema or conf.schema_name
    ret.support = support.load_schema_config_support(conf.schema_name)
-   ret.socket = open_socket(ret.socket_file_name)
    ret.peers = {}
    ret.setup_fn = conf.setup_fn
    ret.period = 1/conf.Hz
@@ -80,6 +82,8 @@ function new_manager (conf)
 
    ret:set_initial_configuration(conf.initial_configuration)
 
+   ret:start()
+
    return ret
 end
 
@@ -138,6 +142,12 @@ function Manager:set_initial_configuration (configuration)
    end
 end
 
+function Manager:start ()
+   if self.name then engine.claim_name(self.name) end
+   self.cpuset:bind_to_numa_node()
+   self.socket = open_socket(self.socket_file_name)
+end
+
 function Manager:start_worker(sched_opts)
    local code = {
       scheduling.stage(sched_opts),
@@ -988,6 +998,7 @@ function Manager:stop ()
       self:remove_stale_workers()
       C.usleep(5000)
    end
+   if self.name then engine.unclaim_name(self.name) end
    self:info('Shutdown complete.')
 end
 
@@ -1016,7 +1027,7 @@ function main (opts, duration)
 end
 
 function selftest ()
-   print('selftest: lib.ptree.manager')
+   print('selftest: lib.ptree.ptree')
    local function setup_fn(cfg)
       local graph = app_graph.new()
       local basic_apps = require('apps.basic.basic_apps')
diff --git a/src/program/lwaftr/bench/bench.lua b/src/program/lwaftr/bench/bench.lua
index 8802a3c046..f9cea5dc62 100644
--- a/src/program/lwaftr/bench/bench.lua
+++ b/src/program/lwaftr/bench/bench.lua
@@ -34,7 +34,6 @@ function parse_args(args)
    args = lib.dogetopt(args, handlers, "j:n:hyb:D:", {
       help="h", hydra="y", ["bench-file"]="b", duration="D", name="n", cpu=1})
    if #args ~= 3 then show_usage(1) end
-   cpuset.global_cpuset():bind_to_numa_node()
    return opts, scheduling, unpack(args)
 end
 
diff --git a/src/program/lwaftr/run/run.lua b/src/program/lwaftr/run/run.lua
index 51b1da7b47..5b53e78aa4 100644
--- a/src/program/lwaftr/run/run.lua
+++ b/src/program/lwaftr/run/run.lua
@@ -119,7 +119,6 @@ function parse_args(args)
    if opts.mirror then
       assert(opts["on-a-stick"], "Mirror option is only valid in on-a-stick mode")
    end
-   cpuset.global_cpuset():bind_to_numa_node()
    if opts["on-a-stick"] then
       scheduling.pci_addrs = { v4 }
       return opts, scheduling, conf_file, v4
diff --git a/src/program/lwaftr/setup.lua b/src/program/lwaftr/setup.lua
index b60d919640..001ed6a6b5 100644
--- a/src/program/lwaftr/setup.lua
+++ b/src/program/lwaftr/setup.lua
@@ -1,7 +1,7 @@
 module(..., package.seeall)
 
 local config     = require("core.config")
-local manager    = require("lib.ptree.manager")
+local manager    = require("lib.ptree.ptree")
 local PcapFilter = require("apps.packet_filter.pcap_filter").PcapFilter
 local V4V6       = require("apps.lwaftr.V4V6").V4V6
 local VirtioNet  = require("apps.virtio_net.virtio_net").VirtioNet

From d6e022977522376dcd468549d8c15b9bb2e46302 Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@igalia.com>
Date: Mon, 11 Dec 2017 10:54:53 +0100
Subject: [PATCH 21/31] Add "snabb ptree" program

---
 src/doc/genbook.sh           |   8 ++
 src/program/ptree/README     |  42 ++++++++++
 src/program/ptree/README.inc |   1 +
 src/program/ptree/README.md  | 154 +++++++++++++++++++++++++++++++++++
 src/program/ptree/ptree.lua  |  86 +++++++++++++++++++
 5 files changed, 291 insertions(+)
 create mode 100644 src/program/ptree/README
 create mode 120000 src/program/ptree/README.inc
 create mode 100644 src/program/ptree/README.md
 create mode 100644 src/program/ptree/ptree.lua

diff --git a/src/doc/genbook.sh b/src/doc/genbook.sh
index be23d89ad2..ca23aadd1a 100755
--- a/src/doc/genbook.sh
+++ b/src/doc/genbook.sh
@@ -100,6 +100,14 @@ $(cat $mdroot/lib/ipsec/README.md)
 
 $(cat $mdroot/program/snabbnfv/README.md)
 
+## LISPER
+
+$(cat $mdroot/program/lisper/README.md)
+
+## Ptree
+
+$(cat $mdroot/program/ptree/README.md)
+
 ## Watchdog (lib.watchdog.watchdog)
 
 $(cat $mdroot/lib/watchdog/README.md)
diff --git a/src/program/ptree/README b/src/program/ptree/README
new file mode 100644
index 0000000000..38e165374d
--- /dev/null
+++ b/src/program/ptree/README
@@ -0,0 +1,42 @@
+Usage: ptree --help
+       ptree [OPTION...] SCHEMA.YANG SETUP.LUA CONF
+
+Run a multi-process network function.  The dataplane is constructed by
+the the "setup function" defined in SETUP.LUA.  This function takes
+configuration data conforming to a specified YANG schema as an argument,
+and returns table mapping worker ID to the corresponding graph of apps
+and links (a "core.config" object).
+
+This "ptree" program is so named because it uses the "lib.ptree" process
+tree facility from Snabb.  It's useful for prototyping network functions
+and can be a good starting point for a new network function -- just copy
+its code and modify to suit.
+
+The management process in a "ptree" network function listens on a socket
+for configuration queries and updates.  Users can change the network
+function's configuration while it is running using the "snabb config"
+program.  The management process will use the setup function to compute
+new app graphs for the workers and then apply any needed changes to the
+workers.
+
+Optional arguments:
+       -n NAME, --name NAME     Sets the name as the identifier of this program.
+                                This must be unique amongst other snab programs.
+       --cpu <cpuset>           Run data-plane processes on the given CPUs.
+       --real-time              Enable real-time SCHED_FIFO scheduler on
+                                data-plane processes.
+       --on-ingress-drop=ACTION Specify an action to take in a data-plane if
+                                too many ingress drops are detected.  Available
+                                actions: "warn" to print a warning, "flush"
+                                to flush the JIT, or "off" to do nothing.
+                                Default is "flush".
+       -D       <seconds>       Duration in seconds.
+       -v                       Verbose (repeat for more verbosity).
+
+Optional arguments for debugging and profiling:
+       -jv, -jv=FILE            Print out when traces are recorded
+       -jp, -jp=MODE,FILE       Profile the system by method
+       -jtprof                  Profile the system by trace
+
+CPUSET is a list of CPU ranges.  For example "3-5,7-9", or "3,4,5,7,8,9"
+both allow the data planes to run on the given CPUs.
diff --git a/src/program/ptree/README.inc b/src/program/ptree/README.inc
new file mode 120000
index 0000000000..100b93820a
--- /dev/null
+++ b/src/program/ptree/README.inc
@@ -0,0 +1 @@
+README
\ No newline at end of file
diff --git a/src/program/ptree/README.md b/src/program/ptree/README.md
new file mode 100644
index 0000000000..28a8f54d98
--- /dev/null
+++ b/src/program/ptree/README.md
@@ -0,0 +1,154 @@
+### Ptree (program.ptree)
+
+Example Snabb program for prototyping multi-process YANG-based network
+functions.
+
+#### Overview
+
+The `lib.ptree` facility in Snabb allows network engineers to build a
+network function out of a tree of processes described by a YANG schema.
+The root process runs the management plane, and the leaf processes (the
+"workers") run the data plane.  The apps and links in the workers are
+declaratively created as a function of a YANG configuration.
+
+This `snabb ptree` program is a tool to allow quick prototyping of
+network functions using the ptree facilities.  The invocation syntax of
+`snabb ptree` is as follows:
+
+```
+snabb ptree [OPTION...] SCHEMA.YANG SETUP.LUA CONF
+```
+
+The *schema.yang* file contains a YANG schema describing the network
+function's configuration.  *setup.lua* defines a Lua function mapping a
+configuration to apps and links for a set of worker processes.  *conf*
+is the initial configuration of the network function.
+
+#### Example: Simple packet filter
+
+Let's say we're going to make a packet filter application.  We can use
+Snabb's built-in support for filters expressed in pflang, the language
+used by `tcpdump`, and just hook that filter up to a full-duplex NIC.
+
+To begin with, we have to think about how to represent the configuration
+of the network function.  If we simply want to be able to specify the
+PCI device of a NIC, an RSS queue, and a filter string, we could
+describe it with a YANG schema like this:
+
+```yang
+module snabb-pf-v1 {
+  namespace snabb:pf-v1;
+  prefix pf-v1;
+
+  leaf device { type string; mandatory true; }
+  leaf rss-queue { type uint8; default 0; }
+  leaf filter { type string; default ""; }
+}
+```
+
+We throw this into a file `pf-v1.yang`.  In YANG, a `module`'s
+body contains configuration declarations, most importantly `leaf`,
+`container`, and `list`.  In our `snabb-pf-v1` schema, there is a
+`module` containing three `leaf`s: `device`, `rss-queue`, and `filter`.
+Snabb effectively generates a validating parser for configurations
+following this YANG schema; a configuration file must contain exactly
+one `device FOO;` declaration and may contain one `rss-queue` statement
+and one `filter` statement.  Thus a concrete configuration following
+this YANG schema might look like this:
+
+```
+device 83:00.0;
+rss-queue 0;
+filter "tcp port 80";
+```
+
+So let's just drop that into a file `pf-v1.cfg` and use that as our
+initial configuration.
+
+Now we just need to map from this configuration to app graphs in some
+set of workers.  The *setup.lua* file should define this function.
+
+```
+-- Function taking a snabb-pf-v1 configuration and
+-- returning a table mapping worker ID to app graph.
+return function (conf)
+   -- Write me :)
+end
+```
+
+The `conf` parameter to the setup function is a Lua representation of
+config data for this network function.  In our case it will be a table
+containing the keys `device`, `rss_queue`, and `filter`.  (Note that
+Snabb's YANG support maps dashes to underscores for the Lua data, so it
+really is `rss_queue` and not `rss-queue`.)
+
+The return value of the setup function is a table whose keys are "worker
+IDs", and whose values are the corresponding app graphs.  A worker ID
+can be any Lua value, for example a number or a string or whatever.  If
+the user later reconfigures the network function (perhaps setting a
+different filter string), the manager will re-run the setup function to
+produce a new set of worker IDs and app graphs.  The manager will then
+stop workers whose ID is no longer present, start new workers, and
+reconfigure workers whose ID is still present.
+
+In our case we're just going to have one worker, so we can use any
+worker ID.  If the user reconfigures the filter but keeps the same
+device and RSS queue, we don't want to interrupt packet flow, so we want
+to use a worker ID that won't change.  But if the user changes the
+device, probably we do want to restart the worker, so maybe we make the
+worker ID a function of the device name.
+
+With all of these considerations, we are ready to actually write the
+setup function.
+
+```lua
+local app_graph = require('core.config')
+local pci = require('lib.hardware.pci')
+local pcap_filter = require('apps.packet_filter.pcap_filter')
+
+-- Function taking a snabb-pf-v1 configuration and
+-- returning a table mapping worker ID to app graph.
+return function (conf)
+   -- Load NIC driver for PCI address.
+   local device_info = pci.device_info(conf.device)
+   local driver = require(device_info.driver).driver
+
+   -- Make a new app graph for this configuration.
+   local graph = app_graph.new()
+   app_graph.app(graph, "nic", driver,
+                 {pciaddr=conf.device, rxq=conf.rss_queue,
+                  txq=conf.rss_queue})
+   app_graph.app(graph, "filter", pcap_filter.PcapFilter,
+                 {filter=conf.filter})
+   app_graph.link(graph, "nic."..device.tx.." -> filter.input")
+   app_graph.link(graph, "filter.output -> nic."..device.rx)
+
+   -- Use DEVICE/QUEUE as the worker ID.
+   local id = conf.device..'/'..conf.rss_queue
+
+   -- One worker with the given ID and the given app graph.
+   return {[id]=graph}
+end
+```
+
+Put this in, say, `pf-v1.lua`, and we're good to go.  The network
+function can be run like this:
+
+```
+snabb ptree --name my-filter pf-v1.yang pf-v1.lua pf-v1.cfg
+```
+
+See [`snabb ptree --help`](./README) for full details on arguments like
+`--name`.
+
+#### Tuning
+
+(Document scheduling parameters here.)
+
+#### Reconfiguration
+
+(Document "snabb config" and NCS integration here.)
+
+#### Multi-process
+
+(Make a multi-process yang model here.)
diff --git a/src/program/ptree/ptree.lua b/src/program/ptree/ptree.lua
new file mode 100644
index 0000000000..b1da7bd735
--- /dev/null
+++ b/src/program/ptree/ptree.lua
@@ -0,0 +1,86 @@
+-- Use of this source code is governed by the Apache 2.0 license; see COPYING.
+
+module(..., package.seeall)
+
+local engine    = require("core.app")
+local app_graph = require("core.config")
+local lib       = require("core.lib")
+local cpuset    = require("lib.cpuset")
+local yang      = require("lib.yang.yang")
+local ptree     = require("lib.ptree.ptree")
+
+local function fatal (msg, ...)
+   print(string.format(msg, ...))
+   main.exit(1)
+end
+
+local function show_usage (exit_code)
+   print(require("program.ptree.README_inc"))
+   if exit_code then main.exit(exit_code) end
+end
+
+function parse_args (args)
+   local opts = { verbosity = 1, cpuset = cpuset.new() }
+   local scheduling = { ingress_drop_monitor = 'flush' }
+   local handlers = {}
+   function handlers.n (arg) opts.name = assert(arg) end
+   function handlers.v () opts.verbosity = opts.verbosity + 1 end
+   function handlers.D (arg)
+      opts.duration = assert(tonumber(arg), "duration must be a number")
+      assert(opts.duration >= 0, "duration can't be negative")
+   end
+   function handlers.cpu (arg)
+      opts.cpuset:add_from_string(arg)
+   end
+   handlers['real-time'] = function (arg)
+      scheduling.real_time = true
+   end
+   handlers["on-ingress-drop"] = function (arg)
+      if arg == 'flush' or arg == 'warn' then
+         scheduling.ingress_drop_monitor = arg
+      elseif arg == 'off' then
+         scheduling.ingress_drop_monitor = false
+      else
+         fatal("invalid --on-ingress-drop argument: %s (valid values: %s)",
+               arg, "flush, warn, off")
+      end
+   end
+   function handlers.j (arg) scheduling.j = arg end
+   function handlers.h () show_usage(0) end
+
+   args = lib.dogetopt(args, handlers, "vD:hn:j:",
+     { verbose = "v", duration = "D", help = "h", cpu = 1,
+       ["real-time"] = 0, ["on-ingress-drop"] = 1,
+       name="n" })
+
+   if #args ~= 3 then show_usage(1) end
+
+   return opts, scheduling, unpack(args)
+end
+
+function run (args)
+   local opts, scheduling, schema_file, setup_file, conf_file = parse_args(args)
+   local schema_name = yang.add_schema_file(schema_file)
+   local setup_thunk = loadfile(setup_file)
+   local conf = yang.load_configuration(conf_file, {schema_name=schema_name})
+
+   local setup_fn = setup_thunk()
+   if not type(setup_fn) then
+      fatal("Expected %s to evaluate to a function, instead got %s",
+            setup_file, tostring(setup_fn))
+   end
+
+   local manager = ptree.new_manager {
+      name = opts.name,
+      setup_fn = setup_fn,
+      cpuset = opts.cpuset,
+      initial_configuration = conf,
+      schema_name = schema_name,
+      worker_default_scheduling = scheduling,
+      log_level = ({"WARN","INFO","DEBUG"})[opts.verbosity or 1] or "DEBUG",
+   }
+
+   manager:main(opts.duration)
+
+   manager:stop()
+end

From 7ee298361e9ea93be7808a704d6e7bd378955f96 Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@igalia.com>
Date: Mon, 11 Dec 2017 11:22:02 +0100
Subject: [PATCH 22/31] Update ptree documentation

---
 src/lib/ptree/README.md      | 229 +++++++++++++++++++++++++----------
 src/program/alarms/README.md |  19 +--
 src/program/config/README.md |  22 ++--
 3 files changed, 188 insertions(+), 82 deletions(-)

diff --git a/src/lib/ptree/README.md b/src/lib/ptree/README.md
index cb1d37dc8c..7baa75e146 100644
--- a/src/lib/ptree/README.md
+++ b/src/lib/ptree/README.md
@@ -1,48 +1,149 @@
-# Config leader and follower
-
-Sometimes you want to query the state or configuration of a running
-Snabb data plane, or reload its configuration, or incrementally update
-that configuration.  However, you want to minimize the impact of
-configuration query and update on data plane performance.  The
-`Leader` and `Follower` apps are here to fulfill this need, while
-minimizing performance overhead.
-
-The high-level design is that a `Leader` app is responsible for
-knowing the state and configuration of a data plane.  The leader
-offers an interface to allow the outside world to query the
-configuration and state, and to request configuration updates.  To
-avoid data-plane overhead, the `Leader` app should be deployed in a
-separate process.  Because it knows the data-plane state, it can
-respond to queries directly, without involving the data plane.  It
-processes update requests into a form that the data plane can handle,
-and feeds those requests to the data plane via a high-performance
-back-channel.
-
-The data plane runs a `Follower` app that reads and applies update
-messages sent to it from the leader.  Checking for update availability
-requires just a memory access, not a system call, so the overhead of
-including a follower in the data plane is very low.
-
-## Two protocols
-
-The leader communicates with its followers using a private protocol.
-Because the leader and the follower are from the same Snabb version,
-the details of this protocol are subject to change.  The private
-protocol's only design constraint is that it should cause the lowest
-overhead for the data plane.
-
-The leader communicates with the world via a public protocol.  The
+# Process tree (`lib.ptree`)
+
+When prototyping a network function, it's useful to start with a single
+process that does packet forwarding.  A first draft of a prototype
+network function will take its configuration from command line
+arguments; once it's started, you can read some information from it via
+its counters but you can't affect its operation to make it do something
+else without restarting it.
+
+As you grow a prototype network function into a production system, new
+needs arise.  You might want to query the state or configuration of a
+running Snabb data plane.  You might want to reload its configuration,
+or incrementally update that configuration.  However, as you add these
+new capabilities, you want to minimize their impact on data plane
+performance.  The process tree facility is here to help with these tasks
+by allowing a network function to be divided into separate management
+and data-plane processes.
+
+Additionally, as a network function grows, you might want to dedicate
+multiple CPU cores to dataplane tasks.  Here too `lib.ptree` helps out,
+as a management process can be responsible for multiple workers.  All
+you need to do is to write a function that maps your network function's
+configuration to a set of app graphs\* (as a table from worker ID to app
+graph).  Each app graph in the result will be instantiated on a separate
+worker process.  If the configuration changes at run-time resulting in a
+different set of worker IDs, the `ptree` manager will start new
+workers and stop any old workers that are no longer necessary.
+
+\*: An "app graph" is an instance of `core.config`.  The `ptree`
+facility reserves the word "configuration" to refer to the user-facing
+configuration of a network function as a whole, and uses "app graph" to
+refer to the network of Snabb apps that runs in a single worker
+data-plane process.
+
+The high-level design is that a manager from `lib.ptree.manager` is
+responsible for knowing the state and configuration of a data plane.
+The manager also offers an interface to allow the outside world to query
+the configuration and state, and to request configuration updates.
+Because it knows the data-plane state, the manager can respond to
+queries directly, without involving the data plane.  It processes update
+requests into a form that the data plane(s) can handle, and feeds those
+requests to the data plane(s) via a high-performance back-channel.
+
+The data planes are started and stopped by the manager as needed.
+Internally they run a special main loop from `lib.ptree.worker` which,
+as part of its engine breathe loop, also reads and applies update
+messages sent to it from the manager.  Checking for update availability
+requires just a memory access, not a system call, so the overhead of the
+message channel on the data plane is very low.
+
+## Example
+
+See [the example `snabb ptree` program](../../program/ptree/README.md)
+for a full example.
+
+## API reference
+
+The public interface to `ptree` is the `lib.ptree.ptree` module.
+
+— Function **ptree.new_manager** *parameters*
+
+Create and start a new manager for a `ptree` process tree.  *parameters*
+is a table of key/value pairs.  The following keys are required:
+
+ * `schema_name`: The name of a YANG schema describing this network function.
+ * `setup_fn`: A function mapping a configuration to a worker set.  A
+   worker set is a table mapping worker IDs to app graphs (`core.config`
+   instances).  See [the setup function described in the `snabb ptree`
+   documentation](../../program/ptree/README.md) for a full example.
+ * `initial_configuration`: The initial network configuration for the
+   network function, for example as returned by
+   `lib.yang.yang.load_configuration`.  Must be an instance of
+   `schema_name`.
+
+Optional entries that may be present in the *parameters* table include:
+
+ * `socket_file_name`: The name of the socket on which to listen for
+   incoming connections from `snabb config` clients.  See [the `snabb
+   config` documentation](../../program/config/README.md) for more
+   information.  Default is `$SNABB_SHM_ROOT/PID/config-manager-socket`,
+   where the `$SNABB_SHM_ROOT` environment variable defaults to
+   `/var/run/snabb`.
+ * `name`: A name to claim for this process tree.  `snabb config` can
+   address network functions by name in addition to PID.  If the name is
+   already claimed on the local machine, an error will be signalled.
+   The name will be released when the manager stops.  Default is not to
+   claim a name.
+ * `worker_default_scheduling`: A table of scheduling parameters to
+   apply to worker processes, suitable for passing to
+   `lib.scheduling.apply()`.
+ * `default_schema`: Some network functions can respond to `snabb
+   config` queries against multiple schemas.  This parameter indicates
+   the default schema to expose, and defaults to *schema_name*.  Using
+   an alternate default schema requires a bit of behind-the-scenes
+   plumbing to work though from `lib.ptree.support`; see the code for
+   details.
+ * `log_level`: One of `"DEBUG"`, `"INFO"`, or `"WARN"`.  Default is
+   `"WARN"`.
+ * `cpuset`: A set of CPUs to devote to data-plane processes; an
+   instance of `lib.cpuset.new()`.  Default is
+   `lib.cpuset.global_cpuset()`.  The manager will try to bind
+   data-plane worker processes to CPUs local to the NUMA node of any PCI
+   address being used by the worker.
+ * `Hz`: Frequency at which to poll the config socket.  Default is
+   1000.
+
+The return value is a ptree manager object, whose public methods are as
+follows:
+
+— Manager method **:run** *duration*
+
+Run a process tree, servicing configuration and state queries and
+updates from remote `snabb config` clients, managing a tree of workers,
+feeding configuration updates to workers, and receiving state and alarm
+updates from those workers.  If *duration* is passed, stop after that
+many seconds; otherwise continue indefinitely.
+
+— Manager method **:stop**
+
+Stop a process tree by sending a shutdown message to all workers,
+waiting for them to shut down for short time, then forcibly terminating
+any remaining worker processes.  The manager's socket will be closed and
+the Snabb network function name will be released.
+
+## Internals
+
+### Two protocols
+
+The manager communicates with its worker using a private protocol.
+Because the manager and the worker are from the same Snabb version, the
+details of this protocol are subject to change.  The private protocol's
+only design constraint is that it should cause the lowest overhead for
+the data plane.
+
+The manager communicates with the world via a public protocol.  The
 "snabb config" command-line tool speaks this protocol.  "snabb config
 get foo /bar" will find the local Snabb instance named "foo", open the
 UNIX socket that the "foo" instance is listening on, issue a request,
 then read the response, then close the socket.
 
-## Public protocol
+### Public protocol
 
 The design constraint on the public protocol is that it be expressive
-and future-proof.  We also want to enable the leader to talk to more
-than one "snabb config" at a time.  In particular someone should be
-able to have a long-lived "snabb config listen" session open, and that
+and future-proof.  We also want to enable the manager to talk to more
+than one "snabb config" at a time.  In particular someone should be able
+to have a long-lived "snabb config listen" session open, and that
 shouldn't impede someone else from doing a "snabb config get" to read
 state.
 
@@ -73,8 +174,12 @@ rpc get-config {
     leaf schema { type string; mandatory true; }
     leaf revision { type string; }
     leaf path { type string; default "/"; }
+    leaf print-default { type boolean; }
+    leaf format { type string; }
   }
   output {
+    leaf status { type uint8; default 0; }
+    leaf error { type string; }
     leaf config { type string; }
   }
 }
@@ -104,42 +209,42 @@ number of RPCs; the RPCs will be made in order.  See the
 schema](../../lib/yang/snabb-config-leader-v1.yang) for full details
 of available RPCs.
 
-## Private protocol
+### Private protocol
 
-The leader maintains a configuration for the program as a whole.  As
-it gets requests, it computes the set of changes to app graphs that
-would be needed to apply that configuration.  These changes are then
-passed through the private protocol to the follower.  No response from
-the follower is necessary.
+The manager maintains a configuration for the network function as a
+whole.  As it gets requests, it computes the set of changes to worker
+app graphs that would be needed to apply that configuration.  These
+changes are then passed through the private protocol to the specific
+workers.  No response from the workers is necessary.
 
-In some remote or perhaps not so remote future, all Snabb apps will
-have associated YANG schemas describing their individual
-configurations.  In this happy future, the generic way to ship
-configurations from the leader to a follower is by the binary
-serialization of YANG data, implemented already in the YANG modules.
-Until then however, there is also generic Lua data without a schema.
-The private protocol supports both kinds of information transfer.
+In some remote or perhaps not so remote future, all Snabb apps will have
+associated YANG schemas describing how they may be configured.  In this
+happy future, the generic way to ship app configurations from the
+manager to a worker is by the binary serialization of YANG data,
+implemented already in the YANG modules.  Until then however, there is
+also generic Lua data without a schema.  The private protocol supports
+both kinds of information transfer.
 
 In the meantime, the way to indicate that an app's configuration data
 conforms to a YANG schema is to set the `schema_name` property on the
 app's class.
 
 The private protocol consists of binary messages passed over a ring
-buffer.  A follower's leader writes to the buffer, and the follower
-reads from it.  There are no other readers or writers.  Given that a
-message may in general be unbounded in size, whereas a ring buffer is
-naturally fixed, messages which may include arbtrary-sized data may be
-forced to put that data in the filesystem, and refer to it from the
-messages in the ring buffer.  Since this file system is backed by
-`tmpfs`, stalls will be minimal.
+buffer.  A worker's manager writes to the buffer, and the worker reads
+from it.  There are no other readers or writers.  Given that a message
+may in general be unbounded in size, whereas a ring buffer is naturally
+fixed, messages which may include arbitrary-sized data may be forced to
+put that data in the filesystem, and refer to it from the messages in
+the ring buffer.  Since this file system is backed by `tmpfs`, stalls
+will be minimal.
 
 ## User interface
 
-The above sections document how the leader and follower apps are
+The above sections document how the manager and worker libraries are
 implemented so that a data-plane developer can understand the overhead
-of run-time (re)configuration.  End users won't be typing at a UNIX
-socket though; we include the `snabb config` program as a command-line
-interface to this functionality.
+of using `lib.ptree` in their network function.  End users won't be
+typing at a UNIX socket though; we include the `snabb config` program as
+a command-line interface to this functionality.
 
 See [the `snabb config` documentation](../../program/config/README.md)
 for full details.
diff --git a/src/program/alarms/README.md b/src/program/alarms/README.md
index 958d7e4436..d63cf0a705 100644
--- a/src/program/alarms/README.md
+++ b/src/program/alarms/README.md
@@ -34,8 +34,8 @@ config`](../config/README.md) uses.  Only some Snabb data-planes have
 enabled `snabb config`; currently in fact it's only the
 [lwAFTR](../lwaftr/doc/README.md).  If you are implementing a data-plane
 and want to add support for alarms, first you add support for `snabb
-config`.  See the [`apps.config`
-documentation](../../apps/config/README.md) for more.
+config` by using the `lib.ptree` process tree facility.  See the
+[`lib.ptree` documentation](../../lib/ptree/README.md) for more.
 
 ## Resource state
 
@@ -184,12 +184,13 @@ See [`snabb alarms compress --help`](./compress/README) for more information.
 ## How does it work?
 
 The Snabb instance itself should be running in *multi-process mode*,
-whereby there is one main process that shepherds a number of worker
+whereby there is one manager process that shepherds a number of worker
 processes.  The workers perform the actual data-plane functionality, are
 typically bound to reserved CPU and NUMA nodes, and have soft-real-time
-constraints.  The main process however doesn't have much to do; it just
-coordinates the workers.  Workers tell the main process about the alarms
-that they support, and then also signal the main process when an alarm
-changes state.  The main process collects all of the alarms and makes
-them available to `snabb alarms`, over a socket.  See the [`apps.config`
-documentation](../../apps/config/README.md) for full details.
+constraints.  The manager process however doesn't have much to do; it
+just coordinates the workers.  Workers tell the manager process about
+the alarms that they support, and then also signal the manager process
+when an alarm changes state.  The manager process collects all of the
+alarms and makes them available to `snabb alarms`, over a socket.  See
+the [`lib.ptree` documentation](../../lib/ptree/README.md) for full
+details.
diff --git a/src/program/config/README.md b/src/program/config/README.md
index b2f34f7c46..11660df997 100644
--- a/src/program/config/README.md
+++ b/src/program/config/README.md
@@ -371,17 +371,17 @@ relevant standardized schemas.  Work here is ongoing.
 ## How does it work?
 
 The Snabb instance itself should be running in *multi-process mode*,
-whereby there is one main process that shepherds a number of worker
-processes.  The workers perform the actual data-plane functionality,
-are typically bound to reserved CPU and NUMA nodes, and have
-soft-real-time constraints.  The main process however doesn't have
-much to do; it just coordinates the workers.
-
-The main process runs a special app in its engine that listens on a
-UNIX socket for special remote procedure calls, translates those calls
-to updates that the data plane should apply, and dispatches those
-updates to the data plane in an efficient way.  See the [`apps.config`
-documentation](../../apps/config/README.md) for full details.
+whereby there is one manager process that shepherds a number of worker
+processes.  The workers perform the actual data-plane functionality, are
+typically bound to reserved CPU and NUMA nodes, and have soft-real-time
+constraints.  The manager process however doesn't have much to do; it
+just coordinates the workers.
+
+The manager process runs a special event loop that listens on a UNIX
+socket for remote procedure calls from `snabb config` programs,
+translates those calls to updates that the data plane should apply, and
+dispatches those updates to the data plane in an efficient way.  See the
+[`lib.ptree` documentation](../../lib/ptree/README.md) for full details.
 
 Some data planes, like the lwAFTR, add hooks to the `set`, `add`, and
 `remove` subcommands of `snabb config` to allow even more efficient

From a77867a314ea5a8505d2373981ffb084e1ace147 Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@igalia.com>
Date: Mon, 11 Dec 2017 11:25:46 +0100
Subject: [PATCH 23/31] Add programs.inc to gitignore

---
 .gitignore | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.gitignore b/.gitignore
index 7df97359ba..752b10df64 100644
--- a/.gitignore
+++ b/.gitignore
@@ -20,3 +20,4 @@ __pycache__
 /src/programs.inc
 .images
 /lib/luajit/usr
+/src/program/programs.inc

From 68d70ba9d2d56cbe379edee2f0ab228ebdbb74f7 Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@igalia.com>
Date: Mon, 11 Dec 2017 11:33:49 +0100
Subject: [PATCH 24/31] Fix example ptree setup.lua

---
 src/program/ptree/README.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/program/ptree/README.md b/src/program/ptree/README.md
index 28a8f54d98..fe2081dae8 100644
--- a/src/program/ptree/README.md
+++ b/src/program/ptree/README.md
@@ -120,8 +120,8 @@ return function (conf)
                   txq=conf.rss_queue})
    app_graph.app(graph, "filter", pcap_filter.PcapFilter,
                  {filter=conf.filter})
-   app_graph.link(graph, "nic."..device.tx.." -> filter.input")
-   app_graph.link(graph, "filter.output -> nic."..device.rx)
+   app_graph.link(graph, "nic."..device_info.tx.." -> filter.input")
+   app_graph.link(graph, "filter.output -> nic."..device_info.rx)
 
    -- Use DEVICE/QUEUE as the worker ID.
    local id = conf.device..'/'..conf.rss_queue

From 378701c641d5c9b6935f3cec4e966a0665b905c7 Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@igalia.com>
Date: Mon, 11 Dec 2017 11:45:30 +0100
Subject: [PATCH 25/31] More useful cpuset

Do assign CPU affinity even if it has to be on a remote node, but warn.
Warn if no CPUs available.
---
 src/lib/cpuset.lua | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/src/lib/cpuset.lua b/src/lib/cpuset.lua
index afb9c1e8a6..4d63cb9a0f 100644
--- a/src/lib/cpuset.lua
+++ b/src/lib/cpuset.lua
@@ -73,6 +73,25 @@ function CPUSet:acquire(on_node)
          end
       end
    end
+   if on_node ~= nil then
+      for node, cpus in pairs(self.by_node) do
+         for cpu, avail in pairs(cpus) do
+            if avail then
+               print("Warning: No CPU available on local NUMA node "..node)
+               print("Warning: Assigning CPU "..cpu.." from remote node "..node)
+               cpus[cpu] = false
+               return cpu
+            end
+         end
+      end
+   end
+   for node, cpus in pairs(self.by_node) do
+      print("Warning: All assignable CPUs in use; "
+               .."leaving data-plane process without assigned CPU.")
+      return
+   end
+   print("Warning: No assignable CPUs declared; "
+            .."leaving data-plane process without assigned CPU.")
 end
 
 function CPUSet:release(cpu)

From 4e965946e640c86e8c8438d02600ef09f6ccff38 Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@igalia.com>
Date: Mon, 11 Dec 2017 11:47:24 +0100
Subject: [PATCH 26/31] Fix NUMA node in cpuset warning

---
 src/lib/cpuset.lua | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/lib/cpuset.lua b/src/lib/cpuset.lua
index 4d63cb9a0f..251ebba19a 100644
--- a/src/lib/cpuset.lua
+++ b/src/lib/cpuset.lua
@@ -77,7 +77,7 @@ function CPUSet:acquire(on_node)
       for node, cpus in pairs(self.by_node) do
          for cpu, avail in pairs(cpus) do
             if avail then
-               print("Warning: No CPU available on local NUMA node "..node)
+               print("Warning: No CPU available on local NUMA node "..on_node)
                print("Warning: Assigning CPU "..cpu.." from remote node "..node)
                cpus[cpu] = false
                return cpu

From 42191acc82d36c5715faf3287769e4c5cca0e282 Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@igalia.com>
Date: Mon, 11 Dec 2017 11:50:24 +0100
Subject: [PATCH 27/31] Return ptree manager socket name to old name

Use config-leader-socket to avoid breaking existing clients.
---
 src/lib/ptree/README.md | 2 +-
 src/lib/ptree/ptree.lua | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/lib/ptree/README.md b/src/lib/ptree/README.md
index 7baa75e146..a64a54a2e1 100644
--- a/src/lib/ptree/README.md
+++ b/src/lib/ptree/README.md
@@ -77,7 +77,7 @@ Optional entries that may be present in the *parameters* table include:
  * `socket_file_name`: The name of the socket on which to listen for
    incoming connections from `snabb config` clients.  See [the `snabb
    config` documentation](../../program/config/README.md) for more
-   information.  Default is `$SNABB_SHM_ROOT/PID/config-manager-socket`,
+   information.  Default is `$SNABB_SHM_ROOT/PID/config-leader-socket`,
    where the `$SNABB_SHM_ROOT` environment variable defaults to
    `/var/run/snabb`.
  * `name`: A name to claim for this process tree.  `snabb config` can
diff --git a/src/lib/ptree/ptree.lua b/src/lib/ptree/ptree.lua
index d28e5ed8db..54418618bd 100644
--- a/src/lib/ptree/ptree.lua
+++ b/src/lib/ptree/ptree.lua
@@ -34,7 +34,7 @@ if os.getenv('SNABB_MANAGER_VERBOSE') then default_log_level = "DEBUG" end
 
 local manager_config_spec = {
    name = {},
-   socket_file_name = {default='config-manager-socket'},
+   socket_file_name = {default='config-leader-socket'},
    setup_fn = {required=true},
    -- Could relax this requirement.
    initial_configuration = {required=true},

From afe2fbd93ddbf6beaf1938e6b00a973f26a0f4db Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@igalia.com>
Date: Mon, 11 Dec 2017 12:12:16 +0100
Subject: [PATCH 28/31] If "snabb config" doesn't have a schema, fetch remotely

---
 src/lib/ptree/ptree.lua                  |  9 +++++++++
 src/lib/yang/schema.lua                  | 11 ++++++++---
 src/lib/yang/snabb-config-leader-v1.yang | 11 +++++++++++
 src/program/config/common.lua            |  7 +++++++
 4 files changed, 35 insertions(+), 3 deletions(-)

diff --git a/src/lib/ptree/ptree.lua b/src/lib/ptree/ptree.lua
index 54418618bd..dfa6a4515b 100644
--- a/src/lib/ptree/ptree.lua
+++ b/src/lib/ptree/ptree.lua
@@ -249,6 +249,15 @@ function Manager:rpc_describe (args)
             capability = schema.get_default_capabilities() }
 end
 
+function Manager:rpc_get_schema (args)
+   local function getter()
+      return { source = schema.load_schema_source_by_name(
+                  args.schema, args.revision) }
+   end
+   local success, response = pcall(getter)
+   if success then return response else return {status=1, error=response} end
+end
+
 local function path_printer_for_grammar(grammar, path, format, print_default)
    local getter, subgrammar = path_mod.resolver(grammar, path)
    local printer
diff --git a/src/lib/yang/schema.lua b/src/lib/yang/schema.lua
index 143565b4c9..3a5c974e53 100644
--- a/src/lib/yang/schema.lua
+++ b/src/lib/yang/schema.lua
@@ -933,11 +933,16 @@ function load_schema_file(filename)
    return inherit_config(s), e
 end
 load_schema_file = util.memoize(load_schema_file)
-function load_schema_by_name(name, revision)
+
+function load_schema_source_by_name(name, revision)
    -- FIXME: @ is not valid in a Lua module name.
    -- if revision then name = name .. '@' .. revision end
    name = name:gsub('-', '_')
-   return load_schema(require('lib.yang.'..name..'_yang'), name)
+   return require('lib.yang.'..name..'_yang')
+end
+
+function load_schema_by_name(name, revision)
+   return load_schema(load_schema_source_by_name(name, revision))
 end
 load_schema_by_name = util.memoize(load_schema_by_name)
 
@@ -945,7 +950,7 @@ function add_schema(src, filename)
    -- Assert that the source actually parses, and get the ID.
    local s, e = load_schema(src, filename)
    -- Assert that this schema isn't known.
-   assert(not pcall(load_schema_by_name, s.id))
+   assert(not pcall(load_schema_source_by_name, s.id))
    assert(s.id)
    -- Intern.
    print('lib.yang.'..s.id:gsub('-', '_')..'_yang')
diff --git a/src/lib/yang/snabb-config-leader-v1.yang b/src/lib/yang/snabb-config-leader-v1.yang
index 233cf2d120..7daf8354dc 100644
--- a/src/lib/yang/snabb-config-leader-v1.yang
+++ b/src/lib/yang/snabb-config-leader-v1.yang
@@ -40,6 +40,17 @@ module snabb-config-leader-v1 {
     }
   }
 
+  rpc get-schema {
+    input {
+      leaf schema { type string; mandatory true; }
+      leaf revision { type string; }
+    }
+    output {
+      uses error-reporting;
+      leaf source { type string; }
+    }
+  }
+
   rpc get-config {
     input {
       leaf schema { type string; mandatory true; }
diff --git a/src/program/config/common.lua b/src/program/config/common.lua
index f367166346..bd4838b4d7 100644
--- a/src/program/config/common.lua
+++ b/src/program/config/common.lua
@@ -91,6 +91,13 @@ function parse_command_line(args, opts)
       ret.schema_name = descr.default_schema
    end
    require('lib.yang.schema').set_default_capabilities(descr.capability)
+   if not pcall(yang.load_schema_by_name, ret.schema_name) then
+      local response = call_leader(
+         ret.instance_id, 'get-schema',
+         {schema=ret.schema_name, revision=ret.revision_date})
+      assert(not response.error, response.error)
+      yang.add_schema(response.source, ret.schema_name)
+   end
    if opts.with_config_file then
       if #args == 0 then err("missing config file argument") end
       local file = table.remove(args, 1)

From f034af88545f2f3cf90f7ca8015e6fcaec7752f3 Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@igalia.com>
Date: Mon, 11 Dec 2017 12:13:13 +0100
Subject: [PATCH 29/31] Remove debugging printout

---
 src/lib/yang/schema.lua | 1 -
 1 file changed, 1 deletion(-)

diff --git a/src/lib/yang/schema.lua b/src/lib/yang/schema.lua
index 3a5c974e53..cccaa007f7 100644
--- a/src/lib/yang/schema.lua
+++ b/src/lib/yang/schema.lua
@@ -953,7 +953,6 @@ function add_schema(src, filename)
    assert(not pcall(load_schema_source_by_name, s.id))
    assert(s.id)
    -- Intern.
-   print('lib.yang.'..s.id:gsub('-', '_')..'_yang')
    package.loaded['lib.yang.'..s.id:gsub('-', '_')..'_yang'] = src
    return s.id
 end

From ee6537f8f3b1e3c004e5f25b87929a9f06271818 Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@igalia.com>
Date: Mon, 11 Dec 2017 15:50:18 +0100
Subject: [PATCH 30/31] Enhance ptree readme with multi-process example

---
 src/program/ptree/README.md | 154 +++++++++++++++++++++++++++++++++++-
 1 file changed, 152 insertions(+), 2 deletions(-)

diff --git a/src/program/ptree/README.md b/src/program/ptree/README.md
index fe2081dae8..19e9804532 100644
--- a/src/program/ptree/README.md
+++ b/src/program/ptree/README.md
@@ -135,7 +135,7 @@ Put this in, say, `pf-v1.lua`, and we're good to go.  The network
 function can be run like this:
 
 ```
-snabb ptree --name my-filter pf-v1.yang pf-v1.lua pf-v1.cfg
+$ snabb ptree --name my-filter pf-v1.yang pf-v1.lua pf-v1.cfg
 ```
 
 See [`snabb ptree --help`](./README) for full details on arguments like
@@ -151,4 +151,154 @@ See [`snabb ptree --help`](./README) for full details on arguments like
 
 #### Multi-process
 
-(Make a multi-process yang model here.)
+Let's say your clients are really loving this network function, so much
+so that they are running an instance on each network card on your
+server.  Whenever the filter string updates though they are getting
+tired of having to `snabb config set` all of the different processes.
+Well you can make them even happier by refactoring the network function
+to be multi-process.
+
+```yang
+module snabb-pf-v2 {
+  namespace snabb:pf-v2;
+  prefix pf-v2;
+
+  /* Default filter string.  */
+  leaf filter { type string; default ""; }
+
+  list worker {
+    key "device rss-queue";
+    leaf device { type string; }
+    leaf rss-queue { type uint8; }
+    /* Optional worker-specific filter string.  */
+    leaf filter { type string; }
+  }
+}
+```
+
+Here we declare a new YANG model that instead of having one device and
+RSS queue, it has a whole list of them.  The `key "device rss-queue"`
+declaration says that the combination of device and RSS queue should be
+unique -- you can't have two different workers on the same device+queue
+pair, logically.  We declare a default `filter` at the top level, and
+also allow each worker to override with their own filter declaration.
+
+A configuration might look like this:
+
+```
+filter "tcp port 80";
+worker {
+  device 83:00.0;
+  rss-queue 0;
+}
+worker {
+  device 83:00.0;
+  rss-queue 1;
+}
+worker {
+  device 83:00.1;
+  rss-queue 0;
+  filter "tcp port 443";
+}
+worker {
+  device 83:00.1;
+  rss-queue 1;
+  filter "tcp port 443";
+}
+```
+
+Finally, we need a new setup function as well:
+
+```lua
+local app_graph = require('core.config')
+local pci = require('lib.hardware.pci')
+local pcap_filter = require('apps.packet_filter.pcap_filter')
+
+-- Function taking a snabb-pf-v2 configuration and
+-- returning a table mapping worker ID to app graph.
+return function (conf)
+   local workers = {}
+   for k, v in pairs(conf.worker) do
+      -- Load NIC driver for PCI address.
+      local device_info = pci.device_info(k.device)
+      local driver = require(device_info.driver).driver
+
+      -- Make a new app graph for this worker.
+      local graph = app_graph.new()
+      app_graph.app(graph, "nic", driver,
+                    {pciaddr=k.device, rxq=k.rss_queue,
+                     txq=k.rss_queue})
+      app_graph.app(graph, "filter", pcap_filter.PcapFilter,
+                    {filter=v.filter or conf.filter})
+      app_graph.link(graph, "nic."..device_info.tx.." -> filter.input")
+      app_graph.link(graph, "filter.output -> nic."..device_info.rx)
+
+      -- Use DEVICE/QUEUE as the worker ID.
+      local id = k.device..'/'..k.rss_queue
+
+      -- Add worker with the given ID and the given app graph.
+      workers[id] = graph
+   end
+   return workers
+end
+```
+
+If we place these into analogously named files, we have a multiprocess
+network function:
+
+```
+$ snabb ptree --name my-filter pf-v2.yang pf-v2.lua pf-v2.cfg
+```
+
+If you change the root filter string via `snabb config`, it propagates
+to all workers, except those that have their own overrides of course:
+
+```
+$ snabb config set my-filter /filter "'tcp port 666'"
+$ snabb config get my-filter /filter
+"tcp port 666"
+```
+
+The syntax to get at a particular worker is a little gnarly; it's based
+on XPath, for compatibility with existing NETCONF NCS systems.  See [the
+`snabb config` documentation](../config/README.md) for full details.
+
+```
+$ snabb config get my-filter '/worker[device=83:00.1][rss-queue=1]'
+filter "tcp port 443";
+```
+
+You can stop a worker with `snabb config remove`:
+
+```
+$ snabb config remove my-filter '/worker[device=83:00.1][rss-queue=1]'
+$ snabb config get my-filter /
+filter "tcp port 666";
+worker {
+  device 83:00.0;
+  rss-queue 0;
+}
+worker {
+  device 83:00.0;
+  rss-queue 1;
+}
+worker {
+  device 83:00.1;
+  rss-queue 0;
+  filter "tcp port 443";
+}
+```
+
+Start up a new one with `snabb config add`:
+
+```
+$ snabb config add my-filter /worker <<EOF
+{
+  device 83:00.1;
+  rss-queue 1;
+  filter "tcp port 8000";
+}
+EOF
+```
+
+Voilà!  Now your clients will think you are a wizard!

From 024b3ed7a61369a3acdd97d5982eb52346b5606f Mon Sep 17 00:00:00 2001
From: Andy Wingo <wingo@igalia.com>
Date: Mon, 11 Dec 2017 16:10:29 +0100
Subject: [PATCH 31/31] Finish "snabb ptree" docs

---
 src/program/ptree/README.md | 68 ++++++++++++++++++++++++++++++++++---
 1 file changed, 63 insertions(+), 5 deletions(-)

diff --git a/src/program/ptree/README.md b/src/program/ptree/README.md
index 19e9804532..dc9e77703b 100644
--- a/src/program/ptree/README.md
+++ b/src/program/ptree/README.md
@@ -5,9 +5,10 @@ functions.
 
 #### Overview
 
-The `lib.ptree` facility in Snabb allows network engineers to build a
-network function out of a tree of processes described by a YANG schema.
-The root process runs the management plane, and the leaf processes (the
+The [`lib.ptree`](../../lib/ptree/README.md) facility in Snabb allows
+network engineers to build a network function out of a tree of processes
+described by a [YANG schema](../../lib/yang/README.md).  The root
+process runs the management plane, and the leaf processes (the
 "workers") run the data plane.  The apps and links in the workers are
 declaratively created as a function of a YANG configuration.
 
@@ -143,11 +144,68 @@ See [`snabb ptree --help`](./README) for full details on arguments like
 
 #### Tuning
 
-(Document scheduling parameters here.)
+The `snabb ptree` program also takes a number of options that apply to
+the data-plane processes.
+
+— **--cpu** *cpus*
+
+Allocate *cpus* to the data-plane processes.  The manager of the process
+tree will allocate CPUs from this set to data-plane workers.  For
+example, For example, `--cpu 3-5,7-9` assigns CPUs 3, 4, 5, 7, 8, and 9
+to the network function.  The manager will try to allocate a CPU for a
+worker that is NUMA-local to the PCI devices used by the worker.
+
+— **--real-time**
+
+Use the `SCHED_FIFO` real-time scheduler for the data-plane processes.
+
+— **--on-ingress-drop** *action*
+
+If a data-plane process detects too many dropped packets (by default,
+100K packets over 30 seconds), perform *action*.  Available *action*s
+are `flush`, which tells Snabb to re-optimize the code; `warn`, which
+simply prints a warning and raises an alarm; and `off`, which does
+nothing.
+
+— **-j** *arg*
+
+Enable profiling in the data-plane.  Useful when you are trying to
+isolate a performance problem.  It is thought that this will be reworked
+when Snabb switches to RaptorJIT, so we leave this somewhat complicated
+option undocumented at present :)
 
 #### Reconfiguration
 
-(Document "snabb config" and NCS integration here.)
+The manager of a ptree-based Snabb network function also listens to
+configuration queries and updates on a local socket.  The user-facing
+side of this interface is [`snabb config`](../config/README.md).  A
+`snabb config` user can address a local ptree network function by PID,
+but it's easier to do so by name, so the above example passed `--name
+my-filter` to the `snabb ptree` invocation.
+
+For example, we can get the configuration of a running network function
+with `snabb config get`: 
+
+```
+$ snabb config get my-filter /
+device 83:00.0;
+rss-queue 0;
+filter "tcp port 80";
+```
+
+You can also update the configuration.  For example, to move this
+network function over to device `82:00.0`, do:
+
+```
+$ snabb config set my-filter /device 82:00.0
+$ snabb config get my-filter /
+device 82:00.0;
+rss-queue 0;
+filter "tcp port 80";
+```
+
+The ptree manager takes the necessary actions to update the dataplane to
+match the specified configuration.
 
 #### Multi-process