From 29c4c3a664b7c10b1f1e73535dc652564b6aad6d Mon Sep 17 00:00:00 2001 From: Maria Grimaldi Date: Thu, 19 Dec 2024 17:54:47 +0100 Subject: [PATCH] docs: add docs about creating and consuming events --- docs/how-tos/consume-an-event.rst | 72 ++++++ docs/how-tos/create-a-new-event.rst | 224 ++++++++++++++++++ ...nt-bus-to-broadcast-and-consume-events.rst | 87 +++++++ 3 files changed, 383 insertions(+) create mode 100644 docs/how-tos/consume-an-event.rst create mode 100644 docs/how-tos/create-a-new-event.rst create mode 100644 docs/how-tos/use-the-event-bus-to-broadcast-and-consume-events.rst diff --git a/docs/how-tos/consume-an-event.rst b/docs/how-tos/consume-an-event.rst new file mode 100644 index 00000000..47abd56d --- /dev/null +++ b/docs/how-tos/consume-an-event.rst @@ -0,0 +1,72 @@ +Consume an Open edX Event +========================= + +You have two ways of consuming an Open edX event, within the same service or in a different service. In this guide, we will show you how to consume an event within the same service. For consuming events across services, see :doc:`../how-tos/use-the-event-bus-to-broadcast-and-consume-events`. + +.. note:: We encourage you to also consider the practices outlined in the :doc:`../decisions/0016-event-design-practices` ADR for event consumption. + +Throughout this guide, we will use an example of creating an event handler that will execute when a user enrolls in a course from the course about page to better illustrate the steps involved in creating a consumer for an event. + +Setup +----- + +To consume an event within the same service, follow these steps: + +Step 1: Install Open edX Events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +First, add the ``openedx-events`` plugin into your dependencies so the library's environment recognizes the event you want to consume. You can install ``openedx-events`` by running: + +.. code-block:: bash + + pip install openedx-events + +This will mainly make the events available for your CI/CD pipeline and local development environment. If you are using the Open edX platform, the library should be already be installed in the environment so no need to install it. + +Step 2: Create a Event Receiver and Connect it to the Event +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +An :term:`Event Receiver` is simply a function that listens for a specific event and executes custom logic in response to the event being triggered. You can create an event receiver by using the Django signal receivers decorator. Here's an example of an event receiver that listens for the ``COURSE_ENROLLMENT_CREATED`` event and creates a notification preference for the user: + +.. code-block:: python + + from openedx_events import COURSE_ENROLLMENT_CREATED + from django.dispatch import receiver + + @receiver(COURSE_ENROLLMENT_CREATED) + def create_notification_preference(signal, sender, enrollment, metadata, **kwargs): + # Custom logic to create a notification preference for the user + pass + +Now, the django dispatcher will call the ``create_notification_preference`` function when the ``COURSE_ENROLLMENT_CREATED`` event is triggered. In this case, that would be every time a user enrolls in a course. + +.. note:: Consider using asynchronous tasks to handle the event processing to avoid blocking the main thread and improve performance. Also, make sure to handle exceptions and errors gracefully to avoid silent failures and improve debugging. You should also consider not creating a tight coupling between receivers and other services, if doing so is necessary consider using the event bus to broadcast the event. + +Step 5: Test the Event Receiver +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Given the design of Open edX Events, you can include the events definitions in your test suite to ensure that the event receiver is working as expected. You can use the ``send_event`` method to trigger the event and test the event receiver. Here's an example of how you can test the event receiver: + +.. code-block:: python + + from openedx_events import send_event, COURSE_ENROLLMENT_CREATED + + def test_create_notification_preference(): + # Trigger the event + COURSE_ENROLLMENT_CREATED.connect(create_notification_preference) + COURSE_ENROLLMENT_CREATED.send_event( + user=UserData( + pii=UserPersonalData( + username='test_username', + email='test_email@example.com', + name='test_name', + ), + id=1, + is_active=True, + ), + ) + + # Assert that the notification preference was created + assert NotificationPreference.objects.filter(user=enrollment.user).exists() + +This way you can ensure that the event receiver is working as expected and that the custom logic is executed when the event is triggered. If the event definition or payload changes in any way, you can catch the error in the test suite instead of in production. diff --git a/docs/how-tos/create-a-new-event.rst b/docs/how-tos/create-a-new-event.rst new file mode 100644 index 00000000..380c6af7 --- /dev/null +++ b/docs/how-tos/create-a-new-event.rst @@ -0,0 +1,224 @@ +Create a New Open edX Event with Long-Term Support +================================================== + +This guide describes how to create a new Open edX event with long-term support by following the practices outlined in the :doc:`../decisions/0016-event-design-practices` ADR. + +Events design with long-support follow closely the practices described in the ADR to minimize breaking changes, maximize compatibility and support for future versions of Open edX. + +.. note:: Before starting, ensure you've reviewed the documentation on :doc:`docs.openedx.org:developers/concepts/hooks_extension_framework`, this documentation helps you decide if creating a new event is necessary. You should also review the documentation on :doc:`../decisions/0016-event-design-practices` to understand the practices that should be followed when creating a new event. + +Throughout this guide, we will use an example of creating a new event that will be triggered when a user enrolls in a course from the course about page to better illustrate the steps involved in creating a new event. + +Key Outlines from Event Design Practices +---------------------------------------- + +The :doc:`../decisions/0016-event-design-practices` outlines the following key practices to follow when creating a new event: + +- Clearly describe what happened and why. +- Self-descriptive and self-contained as much as possible. +- Avoid runtime dependencies with other services. +- Avoid ambiguous data fields or fields with multiple meaning. +- Appropriate types and formats. +- Events should have a single responsibility. +- Avoid combining multiple events into one. +- Maintain the right granularity: not too fine-grained or too coarse. +- Ensure the triggering logic is consistent and narrow. +- Keep the event size small. +- Avoid flow control information or business logic in the event. +- Consider the consumers' needs when designing the event. +- Avoid breaking changes. + +Step 1: Propose the Use Case to the Community +---------------------------------------------- + +Before contributing a new event, it is important to propose the event to the community to get feedback on the event's design and use case. For instance, you could create a post in Open edX Discuss Forum or create a new issue in the repository's issue tracker describing your use case for the new event. Here are some examples of community members that have taken this step: + +- `Add Extensibility Mechanism to IDV to Enable Integration of New IDV Vendor Persona`_ +- `Add Program Certificate events`_ + +.. note:: If your use case is too specific to your organization, you can implement them in your own library and use it within your services by adopting an organization-scoped approach leveraging the Apache 2.0 license. However, if you think that your use case could be beneficial to the community, you should propose it to the community for feedback and collaboration. + +In our example our use case proposal could be: + + I want to add an event that will be triggered when a user enrolls in a course from the course about page. This event will be useful for services that need to send notifications to the user when they enroll in a course. + +If you are confident that the event is beneficial to the community, you can proceed to the next steps and implement the event. + +Step 2: Place Your Event In an Architecture Subdomain +----------------------------------------------------- + +To implement the new event in the library, you should understand the purpose of the event and where it fits in the Open edX main architecture subdomains. This will help you place the event in the right architecture subdomain and ensure that the event is consistent with the framework's definitions. Fore more details on the Open edX Architectural Subdomains, refer to the :doc:`../reference/architecture-subdomains`. + +In our example, the event is related to the enrollment process, which is part of the ``learning`` subdomain. Therefore, the event should be placed in the ``/learning`` module in the library. The subdomain is also used as part of the :term:`event type `, which is used to identify the event. The event type should be unique and follow the naming convention for event types specified in the :doc:`../decisions/0002-events-naming-and-versioning` ADR. + +For the enrollment event, the event type could be ``org.openedx.learning.course.enrollment.v1``, where ``learning`` is the subdomain. + +.. note:: If you don't find a suitable subdomain for your event, you can propose a new subdomain to the community. However, new subdomains may require some discussion with the community. So we encourage you to start the conversation as soon as possible through any of the communication channels available. + +Step 3: Determine the Content of the Event +------------------------------------------- + +The content of the event should comply with the practices outlined in the :doc:`../decisions/0016-event-design-practices`. The event should be self-descriptive and self-contained as much as possible. The event should contain all the necessary information for consumers to react to the event without having to make additional calls to other services when possible. + +When determining the content of the event, consider the following: + +- What happened and why? +- What data is needed to describe the event? +- What data is needed to react to the event? + +In our specific example of the enrollment event this could be: + +- What happened: A user enrolled in a course. +- Why: The user enrolled in the course from the course about page. +- Data needed to describe the event: User information (who), course information (where), enrollment date and mode (output details). +- Data needed to react to the event: User information, course information, enrollment Date, enrollment Mode. For instance, a notification could send a welcome email to the user. + +As a rule of thumb, the event should contain the minimum amount of data required to describe the event and react to it. Try including data about each entity involved such that: + +- Consumers can identify the entities involved in the event. +- Key data about the entities is included in the event. +- The outcome of the event is clear. + +This will ensure that the event is self-descriptive and self-contained as much as possible. + +There has been cases where events also carry other contextual data not directly related to the event but useful for consumers. Although this is not recommended, if you need to include such data, ensure that the reasoning behind it is documented and does not introduce ambiguity. + +Step 4: Identify the Event Triggering Logic +------------------------------------------- + +The triggering logic for the event should be identified to ensure that the event is triggered in the right places and that the event is triggered consistently. We should identify the triggering logic to ensure that maximum coverage is achieved with minimal modifications. The goal is to focus on core, critical areas where the logic we want to modify executes, ensuring the event is triggered consistently. + +In our example, the triggering logic could be a place where all enrollment logic goes through. This could be the ``enroll`` method in the enrollment model in the LMS, which is called when a user enrolls in a course in all cases. + +Step 5: Write the Event Definition and Payload +---------------------------------------------- + +Implement the :term:`Event Definition` and :term:`Event Payload` for your event in the corresponding subdomain module. The event definition would be a signal that is triggered when the event takes place, and the event payload would be the data that is included in the event. + +.. note:: Ideally, the data that is included in the event payload should be available at the time the event is triggered, and it should be directly related to the event that took place. So before defining the payload, inspect the triggering logic to review the data that is available at the time the event is triggered. + +The event definition and payload must comply with the practices outlined in the :doc:`../decisions/0002-events-naming-and-versioning` and :doc:`../decisions/0003-events-payload` ADRs. Also, with the practices outlined in the :doc:`../decisions/0016-event-design-practices` ADR. Mainly: + +- The event should be self-descriptive and self-contained as much as possible. +- The event should contain all the necessary information directly related to the event that took place. + +Event Payload +~~~~~~~~~~~~~ + +The event payload is a data `attrs`_ class which defines the data that is included in the event that is defined in the corresponding subdomain module in the ``data.py`` file. The payload should contain all the necessary information directly related to the event that took place to ensure that consumers can react to the event without introducing new dependencies to understand the event. + +In our example, the event definition and payload for the enrollment event could be ``CourseEnrollmentData``. This class should contain all the necessary information about the enrollment event, such as user information, course information, enrollment mode, and other relevant data. + +.. code-block:: python + + # Location openedx_events/learning/data.py + @attr.s(frozen=True) + class CourseEnrollmentData: + """ + Attributes defined for Open edX Course Enrollment object. + + Arguments: + user (UserData): user associated with the Course Enrollment. + course (CourseData): course where the user is enrolled in. + mode (str): course mode associated with the course. + is_active (bool): whether the enrollment is active. + creation_date (datetime): creation date of the enrollment. + created_by (UserData): if available, who created the enrollment. + """ + + user = attr.ib(type=UserData) + course = attr.ib(type=CourseData) + mode = attr.ib(type=str) + is_active = attr.ib(type=bool) + creation_date = attr.ib(type=datetime) + created_by = attr.ib(type=UserData, default=None) + +.. note:: Try grouping the data into logical groups to make the event more readable and maintainable. For instance, in the above example, we have grouped the data into User, Course, and Enrollment data. + +Each field in the payload should be documented with a description of what the field represents and the data type it should contain. This will help consumers understand the payload and react to the event. You should be able to justify why each field is included in the payload and how it relates to the event. + +Event Definition +~~~~~~~~~~~~~~~~ + +The event definition should be defined in the corresponding subdomain module in the ``signals.py`` file. The :term:`Event Definition` should comply with: + +- It must be documented using in-line documentation with at least: ``event_type``, ``event_name``, ``event_description`` and ``event_data``. See :doc:`../reference/in-line-code-annotations-for-an-event` for more information. + +In our example, the event definition for the enrollment event could be: + +.. code-block:: python + + # Location openedx_events/learning/signals.py + # .. event_type: org.openedx.learning.course.enrollment.created.v1 + # .. event_name: COURSE_ENROLLMENT_CREATED + # .. event_description: emitted when the user's enrollment process is completed. + # .. event_data: CourseEnrollmentData + COURSE_ENROLLMENT_CREATED = OpenEdxPublicSignal( + event_type="org.openedx.learning.course.enrollment.created.v1", + data={ + "enrollment": CourseEnrollmentData, + } + ) + +Consumers will be able to access the event payload in their receivers to react to the event. The ``event_type`` is mainly used to identify the event. + +.. TODO: add reference to how to add event bus support to the event's payload + +Step 6: Send the Event +---------------------- + +After defining the event, you should trigger the event in the places we identified in the triggering logic. In our example, we identified that the event should be triggered when a user enrolls in a course so it should be triggered when the enrollment process completes successfully independent of the method of enrollment used. Therefore, we should trigger the event in the ``enroll`` method in the enrollment model in the LMS services. + +Here is how the integration could look like: + +.. code-block:: python + + # Location openedx/core/djangoapps/enrollments/models.py + from openedx_events.learning.signals import COURSE_ENROLLMENT_CREATED + + def enroll(cls, user, course_key, mode=None, **kwargs): + """ + Enroll a user in this course. + """ + # Enrollment logic here + ... + # .. event_implemented_name: COURSE_ENROLLMENT_CREATED + COURSE_ENROLLMENT_CREATED.send_event( + enrollment=CourseEnrollmentData( + user=UserData( + pii=UserPersonalData( + username=user.username, + email=user.email, + name=user.profile.name, + ), + id=user.id, + is_active=user.is_active, + ), + course=course_data, + mode=enrollment.mode, + is_active=enrollment.is_active, + creation_date=enrollment.created, + ) + ) + +.. note:: Ensure that the event is triggered consistently and only when the event should be triggered. Avoid triggering the event multiple times for the same event unless necessary, e.g., when there is no other way to ensure that the event is triggered consistently. + +Step 7: Test the Event +---------------------- + +You should test the event to ensure it triggers consistently and that its payload contains the necessary information. Add unit tests in the service that triggers the event. The main goal is to verify that the event triggers as needed, consumers can react to it, and it carries the expected information. + +In our example, we should add checks for all places where the event is triggered to ensure that the event is triggered consistently. We should also verify that the payload contains the necessary information for consumers to react to the event like user information, course information, enrollment mode, and other relevant data. + +Another way we suggest you test your events is by consuming them in a test environment. This will help you verify that the event is triggered and that the payload contains the necessary information. You can use follow the steps in :doc:`../how-tos/consume-an-event` to consume the event in a test environment with a Django Signal Receiver. Or you could also use the Open edX Event Bus to consume the event in a test environment. For more information on how to use the Open edX Event Bus, refer to the :doc:`../how-tos/use-the-event-bus-to-broadcast-and-consume-events`. + +Step 8: Continue the Contribution Process +----------------------------------------- + +After implementing the event, you should continue the contribution process by creating a pull request in the repository. The pull request should contain the changes you made to implement the event, including the event definition, payload, and the places where the event is triggered. + +For more details on how the contribution flow works, refer to the :doc:`docs.openedx.org:developers/concepts/hooks_extension_framework` documentation. + +.. _Add Extensibility Mechanism to IDV to Enable Integration of New IDV Vendor Persona: https://openedx.atlassian.net/wiki/spaces/OEPM/pages/4307386369/Proposal+Add+Extensibility+Mechanisms+to+IDV+to+Enable+Integration+of+New+IDV+Vendor+Persona +.. _Add Program Certificate events: https://github.com/openedx/openedx-events/issues/250 +.. _attrs: https://www.attrs.org/en/stable/ diff --git a/docs/how-tos/use-the-event-bus-to-broadcast-and-consume-events.rst b/docs/how-tos/use-the-event-bus-to-broadcast-and-consume-events.rst new file mode 100644 index 00000000..f7823d89 --- /dev/null +++ b/docs/how-tos/use-the-event-bus-to-broadcast-and-consume-events.rst @@ -0,0 +1,87 @@ +Use the Open edX Event Bus to Broadcast and Consume Events +========================================================== + +After creating a new Open edX Event, you might need to send it across services instead of just within the same process. For this kind of use-cases, you might want to use the Open edX Event Bus. Here :doc:`../concepts/event-bus`, you can find useful information to start getting familiar with the Open edX Event Bus. + +The Open edX Event Bus is a key component of the Open edX architecture, enabling services to communicate without direct dependencies on each other. This guide provides an overview of how to use the event bus to broadcast Open edX Events to multiple services, allowing them to react to changes or actions in the system. + +Setup +----- + +To start producing and consuming events using the Open edX Event Bus, follow these steps: + +Step 1: Install the Open edX Event Bus Plugin +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +First, you need to install the Open edX Event Bus plugin in both the producing and consuming services. The plugin is a Django app that provides the necessary tools and configurations to produce and consume events. You could install the Redis plugin by running: + +.. code-block:: bash + + pip install edx-event-bus-redis + +.. note:: There are currently two community-supported concrete implementations of the Open edX Events Bus, Redis (`event-bus-redis`_) and Kafka (`event-bus-kafka`_). Redis is the default plugin for the Open edX Event Bus, but you can also use Kafka depending on your requirements. If none of these implementations meet your needs, you can implement your own plugin by following the :doc:`../how-tos/add-new-event-bus-concrete-implementation` documentation. + +Step 2: Configure the Event Bus +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In :doc:`../reference/event-bus-configurations`, you can find the available configurations for the event bus that are used to set up the event bus in the Open edX platform. + +In both the producing and consuming services, you need to configure the event bus. This includes setting up the :term:`event types `, :term:`topics `, and other configurations for the :term:`Event Bus` to work with. In this step, you should configure the producer and consumer classes for the event bus implementation you are using: + +- In the :term:`producing ` service, configure the producer class by setting the ``EVENT_BUS_PRODUCER`` setting. E.g., ``edx_event_bus_redis.create_producer``. +- In the :term:`consuming ` service, configure the consumer class by setting the ``EVENT_BUS_CONSUMER`` setting. E.g., ``edx_event_bus_redis.RedisEventConsumer``. + +By configuring these settings, you are telling Open edX Events which concrete implementation to use for producing and consuming events. + +Step 3: Produce the Event +~~~~~~~~~~~~~~~~~~~~~~~~~ + +In the producing/host application, include ``openedx_events`` in ``INSTALLED_APPS`` settings if necessary and add ``EVENT_BUS_PRODUCER_CONFIG`` setting. This setting is a dictionary of :term:`event types ` to dictionaries for :term:`Topic` related configuration. Each :term:`Topic` configuration dictionary uses the topic as a key and contains: + +- A flag called ``enabled`` denoting whether the event will be published. +- The ``event_key_field`` which is a period-delimited string path to event data field to use as event key. + +.. note:: The topic names should not include environment prefix as it will be dynamically added based on ``EVENT_BUS_TOPIC_PREFIX`` setting. + +Here's an example of the producer configuration which will publish events for XBlock published and deleted events to the specified :term:`Topic`: + +.. code-block:: python + + EVENT_BUS_PRODUCER_CONFIG = { + 'org.openedx.content_authoring.xblock.published.v1': { + 'content-authoring-xblock-lifecycle': {'event_key_field': 'xblock_info.usage_key', 'enabled': True}, + 'content-authoring-xblock-published': {'event_key_field': 'xblock_info.usage_key', 'enabled': True} + }, + 'org.openedx.content_authoring.xblock.deleted.v1': { + 'content-authoring-xblock-lifecycle': {'event_key_field': 'xblock_info.usage_key', 'enabled': True}, + }, + } + +The ``EVENT_BUS_PRODUCER_CONFIG`` is read by ``openedx_events`` and a handler (`general_signal_handler`_) is attached which does the leg work of reading the configuration again and pushing to appropriate handlers. + +Step 4: Consume the Event +~~~~~~~~~~~~~~~~~~~~~~~~~ + +In the consuming service, include ``openedx_events`` in ``INSTALLED_APPS`` settings if necessary and add ``EVENT_BUS_CONSUMER_CONFIG`` setting. Then, you should implement a receiver for the event type you are interested in. In this example, we are interested in the XBlock deleted event: + +.. code-block:: python + + @receiver(XBLOCK_DELETED) + def update_some_data(sender, **kwargs): + ... do things with the data in kwargs ... + ... log the event for debugging purposes ... + +Step 5: Run the Consumer +~~~~~~~~~~~~~~~~~~~~~~~~ + +To consume events, Open edX Events provides a management command called `consume_events`_ which can be called from the command line, how to run this command will depend on your deployment strategy. This command will start a process that listens to the message broker for new messages, processes them and emits the event. Here is an example using of a `consumer using Tutor hosted in Kubernetes`_. + +You can find more a concrete example of how to produce and consume events in the `event-bus-redis`_ documentation. + +.. _consume_events: https://github.com/openedx/openedx-events/blob/main/openedx_events/management/commands/consume_events.py +.. _event-bus-redis: https://github.com/openedx/event-bus-redis +.. _event-bus-kafka: https://github.com/openedx/event-bus-kafka +.. _run the consumer locally without tutor: https://github.com/openedx/event-bus-redis/?tab=readme-ov-file#testing-locally +.. _run the consumer locally with tutor: https://github.com/openedx/event-bus-redis/blob/main/docs/tutor_installation.rst#setup-example-with-openedx-course-discovery-and-tutor +.. _general_signal_handler: https://github.com/openedx/openedx-events/blob/main/openedx_events/apps.py#L16-L44 +.. _consumer using Tutor hosted in Kubernetes: https://github.com/openedx/tutor-contrib-aspects/blob/master/tutoraspects/patches/k8s-deployments#L535-L588