Feature/mesh file

documentation/source/configuration.rst

@@ -10,26 +10,27 @@ In the source code, this is implemented as a ``scarab::param_node``.
 Dripline Parameters
 ===================
 
-All dripline-based applications must have a common set of configuration parameters.  
-These parameters are indicated by the key ``dripline``. 
+All dripline-based applications must have a common set of mesh configuration parameters.  
+These parameters are grouped in the configuration with the key ``dripline_mesh``. 
 
 Explanation of Parameters
 -------------------------
 
-Here is the possible set of parameters for the ``dripline`` block, with a short description of each one:
+Here is the possible set of parameters for the ``dripline_mesh`` block, with a short description of each one:
 
 .. code-block:: YAML
 
-    dripline:
-        auth-file: (string) path to the authentications file (absolute paths are recommended)
-        requests-exchange: (string) name of the requests exchange
-        alerts-exchange: (string) name of the alerts exchange
-        max-payload-size: (unsigned int) maximum payload size in bytes
-        loop-timeout-ms: (unsigned int) time used in loops for checking for application shutdown in milliseconds
-        message-wait-ms: (unsigned int) timeout for waiting for a message in milliseconds
-        heartbeat-routing-key: (string) routing key for sending and receiving heartbeat messages
-        hearteat-interval-s: (unsigned int) interval for sending heartbeats in seconds
-        return-codes:
+    dripline_mesh:
+        broker: (string) address of the broker
+        broker_port: (unsigned int) port for exchanging AMQP messages with the broker
+        requests_exchange: (string) name of the requests exchange
+        alerts_exchange: (string) name of the alerts exchange
+        max_payload_size: (unsigned int) maximum payload size in bytes
+        loop_timeout_ms: (unsigned int) time used in loops for checking for application shutdown in milliseconds
+        message_wait_ms: (unsigned int) timeout for waiting for a message in milliseconds
+        heartbeat_routing_key: (string) routing key for sending and receiving heartbeat messages
+        hearteat_interval_s: (unsigned int) interval for sending heartbeats in seconds
+        return_codes:
           - name: (string) return-code name (must be unique)
             value: (unsigned int) return-code value (must be unique)
             description: (string) human-readable description of what the return-code means
@@ -43,19 +44,27 @@ The defaults for all of these parameters are given in the class ``dripline_confi
 .. code-block:: YAML
 
     dripline:
-        auth-file: DRIPLINE_AUTH_FILE
-        requests-exchange: requests
-        alerts-exchange: alerts
-        max-payload-size: DL_MAX_PAYLOAD_SIZE
-        loop-timeout-ms: 1000
-        message-wait-ms: 1000
-        heartbeat-routing-key: heartbeat
-        hearteat-interval-s: 60
-
-Note that the defaults for ``auth-file`` and ``max-payload-size`` are defined by preprocessor macros that 
+        broker: localhost
+        broker_port: 5672
+        requests_exchange: requests
+        alerts_exchange: alerts
+        max_payload_size: DL_MAX_PAYLOAD_SIZE
+        loop_timeout_ms: 1000
+        message_wait_ms: 1000
+        heartbeat_routing_key: heartbeat
+        hearteat_interval_s: 60
+
+Default parameters can be modified with a YAML file placed in the user's home directory.  
+Specifically, the file should be ``$HOME/.dripline_mesh.yaml``.  A common application of this 
+feature is to set the broker address.  To set the broker address to ``my-broker``, 
+the ``.dripline_mesh.yaml`` file should consist of:
+
+.. code-block:: YAML
+    broker: my-broker
+
+Note that the default ``max-payload-size`` are defined by preprocessor macros that 
 should be defined before building dripline-cpp.  ``DL_MAX_PAYLOAD_SIZE`` has a default 
-value within dripline-cpp, but ``DRIPLINE_AUTH_FILE`` must be defined by the client code/user, and 
-must be a valid file at runtime for the default to be placed in the ``dripline`` block.
+value within dripline-cpp.
 
 The default set of return codes are specified in ``return_codes.cc``.  There are no default return codes 
 in the class ``dripline_config``.
@@ -67,25 +76,60 @@ Application Parameters
 ======================
 
 Typically an application will have its own configuration parameters that will include 
-the ``dripline`` block.  For example, ``dl-agent`` is configured from ``agent_config``:
+the ``dripline_mesh`` block.  For example, ``dl-agent`` is configured from ``agent_config``:
 
 .. code-block:: YAML
 
     timeout: 10
-    dripline:
-        auth-file: DRIPLINE_AUTH_FILE
+    dripline_mesh:
+        broker: localhost
         . . .
 
 The application configuration parameters can be arbitrarily complicated, 
 according to the scarab application framework, 
-and for dripline purposes they just need to contain the ``dripline`` framework.
+and for dripline purposes they just need to contain the ``dripline_mesh`` block.
+
+Authentication
+==============
+
+Authentication information is handled separately from other configuation parameters since it's 
+usually sensitive information that shouldn't be exposed.  The authentication information is 
+specified through a combination of application-specific defaults and user overrides. 
+
+In the dripline context, the main authentication item is for the RabbitMQ broker: a username 
+and password are required.  Other authentication items may also be present: e.g. for database 
+access or posting messages to something like Slack.
+
+In order of precedence, with items lower on the list overriding those higher on the list, the 
+sources of authentication information are:
+
+    1. Application defaults.  For dripline, the default username and password are ``guest`` and ``guest``, 
+    which match the defaults used by the RabbitMQ broker.
+
+    2. Environment variables.  By default dripline uses ``DRIPLINE_USERNAME`` and ``DRIPLINE_PASSWORD`` to 
+    set the username and password for sending messages to the broker, respectively.  The user can change 
+    the variables used at runtime.  If the variable(s) are present, their values will be used; otherwise 
+    they will be ignored.
+
+    3. A user-supplied file.  A file can be provided that contains exactly the item in question.  This is most 
+    often used for passwords.  Some deployment methods use the concept of a "secrets file" that can be used to 
+    provide sensitive information like a password.  The file should contain exactly the value desired for the 
+    particular authentication parameter (e.g. watch out for unintentional new lines at the end of a file).   
+    There is no default setting for this -- if the user does not supply a filename, no action is taken.
+
+    4. A user-supplied value.  An authentication item can be supplied directly, overriding any other settings.  
+    Be aware that this can put the value of an item into one's CLI history or otherwise expose it, which 
+    can be problematic for passwords.  There is no default setting for this -- if the user does not supply a value, 
+    no action is taken.
 
 Specifying Parameters
 =====================
 
-The configuration process takes place in four stages:
+The configuration process takes place in five stages:
 
-    1. The default parameters are used to form the master configuration dictionary.
+    1. The default parameters are used to form the primary configuration dictionary.  If a dripline mesh 
+    configuration file exists in the user's home directory (i.e. ``$HOME/.dripline_mesh.yaml``), values 
+    present in that file are merged into the hard-coded defaults.
 
     2. If specified, a configuration file is parsed and merged with the stage-1 configuration.
 
@@ -94,22 +138,91 @@ The configuration process takes place in four stages:
 
     4. Any command-line options (i.e. ``--parameter value``) are merged with the stage-3 configuration.
 
-After stage four, the master configuration dictionary is passed to the application.
+    5. If any parameters have been specified to include environment variable values, the variables are checked and 
+    the values are inserted into the parameter values.
 
-Broker and Broker Port
-======================
+After stage five, the primary configuration dictionary is passed to the application.
 
-In a dripline application, a few parameters can be specified from the :ref:`authentication  <authentication>` file:
+Configuration File
+------------------
 
-.. code-block:: YAML
+A configuration file, written in YAML or JSON, can be provided on the command-line.  This file can specify any parameters 
+that the user wants to configure via the file.  Parameters not included will be set to their default values.
 
-    broker: [broker URL]
-    broker-port: [broker port]
+Keyword Arguments
+-----------------
 
-The values given in the authentications file form the default on top of which the master-config is applied.  
-In other words, if ``broker`` or ``broker-port`` are specified in the master configuration, they override 
-the values from the authentications file.  For that reason, ``dripline_config`` does not specify either of 
-those values.
+A keyword argument can modify any existing parameter value.  The format for the argument is ``key=value``.
+
+The ``key`` is used to address the particular parameter in the configuration hierarchy.  If the configuration 
+is viewed as a nested set of array-like and dictionary-like structures, any value in that structure can be 
+addressed with the following syntax: a combination of strings and integers, each of which indicates 
+a position in the nested dictionaries (string keys) and arrays (integer keys), separated by ``.``.  
+For example, given this configuration:
+
+.. code-block:: YAML
+    mercury:
+      moons: []
+      surface_temp: 167
+    venus:
+      moons: []
+      surface_temp: 464
+    earth:
+      moons:
+        - The moon
+      surface_temp: 18
+    mars:
+      moons:
+        - Phobos
+        - Deimos
+      surface_temp: -65
+
+You could fix the average temperature on early with ``earth.surface_temp=15`` or change the name 
+of Mars' second moon with ``mars.moons.1=moony``.
+
+Command-Line Options
+--------------------
+
+As a general principle, each application specifies the set of command-line (CL) options that it will use.  
+There is a default set of CL options that all dripline executables include:
+
+.. code-block::
+    -h,--help                     Print this help message and exit
+    -c,--config TEXT:FILE         Config file filename
+    --config-encoding TEXT        Config file encoding
+    -v,--verbose                  Increase verbosity
+    -q,--quiet                    Decrease verbosity
+    -V,--version                  Print the version message and exit
+    -u,--username TEXT            Specify the username for the rabbitmq broker
+    --password TEXT               Specify a password for the rabbitmq broker -- NOTE: this will be plain text on the command line and may end up in your command history!
+    --password-file TEXT          Specify a file (e.g. a secrets file) to be read in as the rabbitmq broker password
+    --auth-file TEXT              Set the authentication file path
+    -b,--broker TEXT              Set the dripline broker address
+    -p,--port UINT                Set the port for communication with the dripline broker
+    --requests-exchange TEXT      Set the name of the requests exchange
+    --alerts-exchange TEXT        Set the name of the alerts exchange
+    --max-payload UINT            Set the maximum payload size (in bytes)
+    --heartbeat-routing-key TEXT  Set the first token of heartbeat routing keys: [token].[origin]
+
+Specific applications will add further options.  For example, ``dl-agent`` adds options having to do with 
+sending messages, and ``dl-mon`` adds options having to do with monitoring messages.
+
+Environment Variables
+---------------------
+
+Environment variables can be used to substitute values into configuration parameters.  The syntax used in 
+the configuration parameter value is: ``ENV{<variable>}``.  That syntax needs to be inserted into or as a 
+configuration parameter value in one of the four previous configuration stages.
+
+If an environment variable is specified in the configuration but the variable does not exist in 
+the environment, an exception will be thrown.
+
+Here's an example configuration, shown in YAML format, where environment variable subsitution is requested:
+
+.. code-block:: YAML
+    dripline_mesh:
+      broker: ENV{DL_PREFIX}-broker
+      broker-port: ENV{DL_PORT}
 
-At the source-code level, clients of ``core`` can override ``broker`` and ``broker-port`` by specifying 
-parameters of the constructor.  There are a few other arguments that can be overridden in this way, as well.
+In this case the user wants a customized broker address specified at runtime by the contents of the ``DL_PREFIX`` 
+environment variable, and they want to specify the port with ``DL_PORT``.

library/core.cc

@@ -65,7 +65,7 @@ namespace dripline
         // So we need to assume no configuration values are supplied and we start again from dripline_config, then merge in a_config.
         dripline_config t_config;
         t_config.merge( a_config );
-        LDEBUG( dlog, "User-supplied config:\n" << a_config );
+        LDEBUG( dlog, "Received config:\n" << a_config );
         LDEBUG( dlog, "Dripline core being configured with:\n" << t_config );
 
 /* DO WE WANT TO USE ALTERNATIVE AUTH GROUPS?

library/dripline_config.cc

@@ -10,19 +10,21 @@
 #include "dripline_config.hh"
 
 #include "application.hh"
+#include "param_codec.hh"
 #include "param_helpers_impl.hh"
+#include "path.hh"
 
 #include "logger.hh"
 
 using scarab::param_node;
 using_param_args_and_kwargs;
 
-LOGGER( dlog, "agent_config" );
+LOGGER( dlog, "dripline_config" );
 
 namespace dripline
 {
 
-    dripline_config::dripline_config()
+    dripline_config::dripline_config( bool a_read_mesh_file )
     {
         // default dripline configuration
         add( "broker_port", 5672 );
@@ -32,6 +34,23 @@ namespace dripline
         add( "max_payload_size", DL_MAX_PAYLOAD_SIZE );
         add( "heartbeat_routing_key", "heartbeat" );
         add( "max_connection_attempts", 10 );
+
+        //LWARN( dlog, "in dripline_config constructor" );
+        if( a_read_mesh_file )
+        {
+            scarab::path t_file_path = scarab::path(getenv("HOME")) / scarab::path(".dripline_mesh.yaml");
+            if( boost::filesystem::exists( t_file_path ) )
+            {
+                LDEBUG( dlog, "Loading mesh file " << t_file_path );
+                scarab::param_translator t_translator;
+                scarab::param_ptr_t t_config_from_file = t_translator.read_file(t_file_path.native());
+                merge( t_config_from_file->as_node() );
+            }
+            else
+            {
+                LDEBUG( dlog, "Mesh file is not present: " << t_file_path );
+            }
+        }
     }
 
     void add_dripline_options( scarab::main_app& an_app )

library/dripline_config.hh

@@ -30,7 +30,8 @@ namespace dripline
     class DRIPLINE_API dripline_config : public scarab::param_node
     {
         public:
-            dripline_config();
+            /// Creates the mesh config object with default values merged (optionally) with anything in $HOME/.dripline_mesh.yaml
+            dripline_config( bool a_read_mesh_file=true );
             dripline_config( const dripline_config& ) = default;
             dripline_config( dripline_config&& ) = default;
             virtual ~dripline_config() = default;

testing/integration/dl-tests.sh

@@ -1,29 +1,33 @@
 #! /usr/bin/env bats
 
 @test "ping simple" {
-    dl-agent -vv -b rabbit-broker -u dripline --password-file /dl_pw.txt  cmd simple -s ping
+    dl-agent -vv cmd simple -s ping
 }
 
 @test "get simple" {
-    dl-agent -vv -b rabbit-broker -u dripline --password-file /dl_pw.txt  get simple
+    dl-agent -vv  get simple
 }
 
 @test "set simple" {
-    dl-agent -vv -b rabbit-broker -u dripline --password-file /dl_pw.txt  set simple 500
+    dl-agent -vv set simple 500
 }
 
 @test "user/pass on CL" {
-    dl-agent -vv -b rabbit-broker -u dripline --password dripline get simple
+    dl-agent -vv -u dripline --password dripline get simple
 }
 
 @test "user on CL/pass in file" {
-    dl-agent -vvv -b rabbit-broker -u dripline --password-file /dl_pw.txt get simple
+    dl-agent -vv -u dripline --password-file /dl_pw.txt get simple
 }
 
-@test "user/pass as environment variables" {
-    DRIPLINE_USER=dripline DRIPLINE_PASSWORD=dripline dl-agent -vvv -b rabbit-broker get simple
+@test "auth file" {
+    dl-agent -vv --auth-file /auths.json get simple
 }
 
-@test "auth file" {
-    dl-agent -vv -b rabbit-broker --auth-file /auths.json get simple
+@test "oscillator hub" {
+    dl-agent -vv get osc_svc_hub -s in-phase
+}
+
+@test "oscillator endpoints" {
+    dl-agent -vv get in_phase
 }

testing/integration/do-testing.sh

@@ -29,6 +29,8 @@ docker logs integration-test-1
 if [ -z ${TEST_EXIT_CODE+x} ] || [ "$TEST_EXIT_CODE" -ne 0 ] ; then
   docker logs integration-test-1
   docker logs integration-simple-service-1
+  docker logs integration-oscillator-hub-1
+  docker logs integration-oscillator-endpoints-1
   docker logs integration-rabbit-broker-1
   printf "${RED}Tests Failed${NC} - Exit Code: $TEST_EXIT_CODE\n"
 else

testing/integration/docker-compose-test.yaml

@@ -4,13 +4,19 @@ services:
     depends_on:
       - rabbit-broker
       - simple-service
-    volumes:
-      - ./dl-tests.sh:/root/dl-tests.sh
-      - ./rabbitmq_for_dl_collection.json:/root/rabbitmq_for_dl_collection.json
+      - oscillator-hub
+      - oscillator-endpoints
     command: >
       bash -c "sleep 1 &&
               newman run /root/rabbitmq_for_dl_collection.json &&
               /root/dl-tests.sh"
+    volumes:
+      - ./dl-tests.sh:/root/dl-tests.sh
+      - ./rabbitmq_for_dl_collection.json:/root/rabbitmq_for_dl_collection.json
+      - ./dripline_mesh.yaml:/root/.dripline_mesh.yaml
+    environment:
+      - DRIPLINE_USER=dripline
+      - DRIPLINE_PASSWORD=dripline 
     configs:
       - dl_pw.txt
       - auths.json