Skip to content

Commit

Permalink
docs: Update docs for better clarity around filters and logging
Browse files Browse the repository at this point in the history
  • Loading branch information
bmtcril committed Jun 6, 2023
1 parent 1050fc3 commit 2e33ac3
Show file tree
Hide file tree
Showing 2 changed files with 12 additions and 5 deletions.
4 changes: 4 additions & 0 deletions docs/getting_started.rst
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,10 @@ Two types of configuration are needed for the plugin:

By default, both ``xapi`` and ``caliper`` backends are already configured along with filters that allow all the supported events. ``caliper`` backend is disabled by default and can be enabled by setting ``CALIPER_EVENTS_ENABLED`` to ``True`` in plugin settings.

Additionally separate log streams for xAPI and Caliper are generated as events are transformed and can be configured to be saved or ignored. These can be configured as described in the `Django docs <https://docs.djangoproject.com/en/4.2/topics/logging/>`_ for the ``xapi_tracking`` and ``caliper_tracking`` loggers.



Router configuration
--------------------

Expand Down
13 changes: 8 additions & 5 deletions docs/howto/how_to_bulk_transform.rst
Original file line number Diff line number Diff line change
Expand Up @@ -37,15 +37,18 @@ The command can work in a few distinct ways.

**File(s) to learning record store (LRS)** - this will use the existing event-routing-backends configuration to route any log replays to **all** configured LRS backends just like the event was being emitted right now. This can be used to backfill old data, capture old events that didn't previously have transforms, or fix up lost data from downtime issues.

.. note:: Note that event filters for the LRS destination(s) in the `ERB router configuration <https://github.com/openedx/event-routing-backends/blob/master/docs/getting_started.rst#router-configuration>`_ will work as usual on these events.

**File(s) to file(s)** - This will perform the same transformations as usual, but instead of routing them to an LRS they can be saved as a file to any libcloud destination. In this mode all events are saved to a single file and no filters are applied.

.. note:: Note that NO event filters from the usual ERB configuration will impact these events, since they happen on a per-destination basis.
Additionally all generated statements are written to a Python logger which can be configured to be ignored, save to a file, write standard out, or a log forwarder like `Vector <https://vector.dev/>`__ for more statement handling options. The two loggers are named ``xapi_tracking`` and ``caliper_tracking``, and are always running.

**File(s) to logger** - For any destination you can use the ``--dry_run`` flag to perform tests on finding and transforming data before attempting to store it. Used in conjunction with loggers mentioned above, you can use Python log forwarding without the additional overhead of storing full files.

.. warning::
Events may be filtered differently in this command than in normal operation. Normally events pass through two layers of filters as described in `getting started <docs/getting_started.rst>`_.

Additionally you can run with the ``--log_statements`` option in either mode to send all generated statements to a Python logger which can be configured to save to a file, standard out, or a log forwarder like `Vector <https://vector.dev/>`__ for more statement handling options. These logs are not filtered.
First are the eventtracking AsyncRoutingBackend can have processor filters, which will be ignored when running this script (since these events have already passed through the eventtracking process).

**File(s) to logger** - For any destination you can use the ``--dry_run`` flag to perform tests on finding and transforming data before attempting to store it. Used in conjunction with ``--log_statements`` you can use Python log forwarding without the additional overhead of storing full files.
Second are the router configuration filters which work on a per-LRS basis. These are respected when the destination is LRS, but ignored when writing to a file and the loggers.


Examples
Expand Down

0 comments on commit 2e33ac3

Please sign in to comment.