feat: pluggable local/dev/k8s do <job> commands

We introduce a new filter to implement custom commands in arbitrary containers. It becomes easy to write convenient ad-hoc commands that users will then be able to run either on Kubernetes or locally using a documented CLI. Pluggable jobs are declared as Click commands and are responsible for parsing their own arguments. See the new CLI_DO_COMMANDS filter. Close openedx-unsupported/wg-developer-experience#75
overhangio · Nov 8, 2022 · 84e08fe · 84e08fe
1 parent 109dd34
commit 84e08fe
Show file tree

Hide file tree

Showing 18 changed files with 268 additions and 182 deletions.
diff --git a/CHANGELOG-nightly.md b/CHANGELOG-nightly.md
@@ -8,6 +8,8 @@ will be backported to the master branch at every major release.
 When backporting changes to master, we should keep only the entries that correspond to user-
 facing changes.
 -->
+- 💥[Feature] Add an extensible `local/dev/k8s do ...` command to trigger custom job commands. These commands are used to run a series of bash scripts in designated containers. Any plugin can add custom jobs thanks to the `CLI_DO_COMMANDS` filter. This causes the following breaking changes:
+    - The "init", "createuser", "settheme", "importdemocourse" commands were all migrated to this new interface. For instance,  `tutor local init` was replaced by `tutor local do init`.
 - 💥[Improvement] Remove the `local/dev bindmount` commands, which have been marked as deprecated for some time. The `--mount` option should be used instead.
 - 💥[Bugfix] Fix local installation requirements. Plugins that implemented the "openedx-dockerfile-post-python-requirements" patch and that needed access to the edx-platform repo will no longer work. Instead, these plugins should implement the "openedx-dockerfile-pre-assets" patch. This scenario should be very rare, though. (by @regisb)
 - 💥[Improvement] Rename the implementation of tutor <mode> quickstart to tutor <mode> launch. (by @Carlos-Muniz)

diff --git a/docs/local.rst b/docs/local.rst
@@ -80,7 +80,7 @@ Service initialisation
 
 ::
 
-    tutor local init
+    tutor local do init
 
 This command should be run just once. It will initialise all applications in a running platform. In particular, this will create the required databases tables and apply database migrations for all applications.
 
@@ -120,7 +120,7 @@ Creating a new user with staff and admin rights
 
 You will most certainly need to create a user to administer the platform. Just run::
 
-    tutor local createuser --staff --superuser yourusername user@email.com
+    tutor local do createuser --staff --superuser yourusername user@email.com
 
 You will be asked to set the user password interactively.
 
@@ -131,7 +131,7 @@ Importing the demo course
 
 After a fresh installation, your platform will not have a single course. To import the `Open edX demo course <https://github.com/openedx/edx-demo-course>`_, run::
 
-    tutor local importdemocourse
+    tutor local do importdemocourse
 
 .. _settheme:
 
@@ -140,7 +140,7 @@ Setting a new theme
 
 The default Open edX theme is rather bland, so Tutor makes it easy to switch to a different theme::
 
-    tutor local settheme mytheme
+    tutor local do settheme mytheme
 
 Out of the box, only the default "open-edx" theme is available. We also developed `Indigo, a beautiful, customizable theme <https://github.com/overhangio/indigo>`__ which is easy to install with Tutor.
 

diff --git a/docs/tutorials/arm64.rst b/docs/tutorials/arm64.rst
@@ -48,9 +48,9 @@ Finish setup and start Tutor
 From this point on, use Tutor as normal. For example, start Open edX and run migrations with::
 
     tutor local start -d
-    tutor local init
+    tutor local do init
 
 Or for a development environment::
 
     tutor dev start -d
-    tutor dev init
+    tutor dev do init
diff --git a/docs/tutorials/plugin.rst b/docs/tutorials/plugin.rst
@@ -218,7 +218,7 @@ You can now run the "myservice" container which will execute the ``CMD`` stateme
 Declaring initialisation tasks
 ------------------------------
 
-Services often need to run specific tasks before they can be started. For instance, the LMS and the CMS need to apply database migrations. These commands are written in shell scripts that are executed whenever we run ``launch``. We call these scripts "init tasks". To add a new local init task, we must first add the corresponding service to the ``docker-compose-jobs.yml`` file by implementing the :patch:`local-docker-compose-jobs-services` patch::
+Services often need to run specific tasks before they can be started. For instance, the LMS and the CMS need to apply database migrations. These commands are written in shell scripts that are executed whenever we run ``launch``. We call these scripts "init tasks". To add a new local initialisation task, we must first add the corresponding service to the ``docker-compose-jobs.yml`` file by implementing the :patch:`local-docker-compose-jobs-services` patch::
 
     hooks.Filters.ENV_PATCHES.add_item(
         (
@@ -251,7 +251,7 @@ Add our init task script to the :py:data:`tutor.hooks.Filters.COMMANDS_INIT` fil
 
 Run this initialisation task with::
 
-    $ tutor local init --limit=myplugin
+    $ tutor local do init --limit=myplugin
     ...
     Running init task: myplugin/tasks/init.sh
     ...

diff --git a/docs/tutorials/theming.rst b/docs/tutorials/theming.rst
@@ -48,7 +48,7 @@ Then, run a local webserver::
 
 The LMS can then be accessed at http://local.overhang.io:8000. You will then have to :ref:`enable that theme <settheme>`::
 
-    tutor dev settheme mythemename
+    tutor dev do settheme mythemename
 
 Watch the themes folders for changes (in a different terminal)::
 

diff --git a/tests/commands/base.py b/tests/commands/base.py
@@ -17,7 +17,9 @@ def invoke(args: t.List[str]) -> click.testing.Result:
             return TestCommandMixin.invoke_in_root(root, args)
 
     @staticmethod
-    def invoke_in_root(root: str, args: t.List[str]) -> click.testing.Result:
+    def invoke_in_root(
+        root: str, args: t.List[str], catch_exceptions: bool = True
+    ) -> click.testing.Result:
         """
         Use this method for commands that all need to run in the same root:
 
@@ -32,4 +34,6 @@ def invoke_in_root(root: str, args: t.List[str]) -> click.testing.Result:
                 "TUTOR_IGNORE_DICT_PLUGINS": "1",
             }
         )
-        return runner.invoke(cli, args, obj=TestContext(root))
+        return runner.invoke(
+            cli, args, obj=TestContext(root), catch_exceptions=catch_exceptions
+        )
diff --git a/tests/commands/test_jobs.py b/tests/commands/test_jobs.py
@@ -1,71 +1,74 @@
-import re
-import unittest
-from io import StringIO
 from unittest.mock import patch
 
-from tests.helpers import TestContext, temporary_root
-from tutor import config as tutor_config
+from tests.helpers import PluginsTestCase, temporary_root
 from tutor.commands import jobs
 
+from .base import TestCommandMixin
 
-class JobsTests(unittest.TestCase):
-    @patch("sys.stdout", new_callable=StringIO)
-    def test_initialise(self, mock_stdout: StringIO) -> None:
+
+class JobsTests(PluginsTestCase, TestCommandMixin):
+    def test_initialise(self) -> None:
         with temporary_root() as root:
-            context = TestContext(root)
-            config = tutor_config.load_full(root)
-            runner = context.job_runner(config)
-            jobs.initialise(runner)
-            output = mock_stdout.getvalue().strip()
-            self.assertTrue(output.startswith("Initialising all services..."))
-            self.assertTrue(output.endswith("All services initialised."))
+            self.invoke_in_root(root, ["config", "save"])
+            result = self.invoke_in_root(root, ["local", "do", "init"])
+            self.assertIsNone(result.exception)
+            self.assertEqual(0, result.exit_code)
+            self.assertIn("All services initialised.", result.output)
 
-    def test_create_user_command_without_staff(self) -> None:
-        command = jobs.create_user_template("superuser", False, "username", "email", "p4ssw0rd")
+    def test_create_user_template_without_staff(self) -> None:
+        command = jobs.create_user_template(
+            "superuser", False, "username", "email", "p4ssw0rd"
+        )
         self.assertNotIn("--staff", command)
         self.assertIn("set_password", command)
 
-    def test_create_user_command_with_staff(self) -> None:
-        command = jobs.create_user_template("superuser", True, "username", "email", "p4ssw0rd")
+    def test_create_user_template_with_staff(self) -> None:
+        command = jobs.create_user_template(
+            "superuser", True, "username", "email", "p4ssw0rd"
+        )
         self.assertIn("--staff", command)
 
-    @patch("sys.stdout", new_callable=StringIO)
-    def test_import_demo_course(self, mock_stdout: StringIO) -> None:
+    def test_import_demo_course(self) -> None:
         with temporary_root() as root:
-            context = TestContext(root)
-            config = tutor_config.load_full(root)
-            runner = context.job_runner(config)
-            runner.run_job_from_str("cms", jobs.import_demo_course_template())
-
-            output = mock_stdout.getvalue()
-            service = re.search(r"Service: (\w*)", output)
-            commands = re.search(r"(-----)([\S\s]+)(-----)", output)
-            assert service is not None
-            assert commands is not None
-            self.assertEqual(service.group(1), "cms")
-            self.assertTrue(
-                commands.group(2)
-                .strip()
-                .startswith('echo "Loading settings $DJANGO_SETTINGS_MODULE"')
-            )
+            self.invoke_in_root(root, ["config", "save"])
+            with patch("tutor.utils.docker_compose") as mock_docker_compose:
+                result = self.invoke_in_root(root, ["local", "do", "importdemocourse"])
+                dc_args, _dc_kwargs = mock_docker_compose.call_args
+        self.assertIsNone(result.exception)
+        self.assertEqual(0, result.exit_code)
+        self.assertIn("cms-job", dc_args)
+        self.assertTrue(
+            dc_args[-1]
+            .strip()
+            .startswith('echo "Loading settings $DJANGO_SETTINGS_MODULE"')
+        )
 
-    @patch("sys.stdout", new_callable=StringIO)
-    def test_set_theme(self, mock_stdout: StringIO) -> None:
+    def test_set_theme(self) -> None:
         with temporary_root() as root:
-            context = TestContext(root)
-            config = tutor_config.load_full(root)
-            runner = context.job_runner(config)
-            command = jobs.set_theme_template("sample_theme", ["domain1", "domain2"])
-            runner.run_job_from_str("lms", command)
+            self.invoke_in_root(root, ["config", "save"])
+            with patch("tutor.utils.docker_compose") as mock_docker_compose:
+                result = self.invoke_in_root(
+                    root,
+                    [
+                        "local",
+                        "do",
+                        "settheme",
+                        "--domain",
+                        "domain1",
+                        "--domain",
+                        "domain2",
+                        "beautiful",
+                    ],
+                )
+                dc_args, _dc_kwargs = mock_docker_compose.call_args
 
-            output = mock_stdout.getvalue()
-            service = re.search(r"Service: (\w*)", output)
-            commands = re.search(r"(-----)([\S\s]+)(-----)", output)
-            assert service is not None
-            assert commands is not None
-            self.assertEqual(service.group(1), "lms")
+            self.assertIsNone(result.exception)
+            self.assertEqual(0, result.exit_code)
+            self.assertIn("lms-job", dc_args)
             self.assertTrue(
-                commands.group(2)
+                dc_args[-1]
                 .strip()
                 .startswith('echo "Loading settings $DJANGO_SETTINGS_MODULE"')
             )
+            self.assertIn("assign_theme('beautiful', 'domain1')", dc_args[-1])
+            self.assertIn("assign_theme('beautiful', 'domain2')", dc_args[-1])
diff --git a/tests/test_env.py b/tests/test_env.py
@@ -83,7 +83,7 @@ def test_render_file(self) -> None:
         tutor_config.render_full(config)
 
         config["MYSQL_ROOT_PASSWORD"] = "testpassword"
-        rendered = env.render_file(config, "hooks", "mysql", "init")
+        rendered = env.render_file(config, "jobs", "init", "mysql.sh")
         self.assertIn("testpassword", rendered)
 
     @patch.object(tutor_config.fmt, "echo")

diff --git a/tutor/commands/compose.py b/tutor/commands/compose.py
@@ -297,19 +297,16 @@ def restart(context: BaseComposeContext, services: t.List[str]) -> None:
     context.job_runner(config).docker_compose(*command)
 
 
-@click.command(help="Initialise all applications")
-@click.option("-l", "--limit", help="Limit initialisation to this service or plugin")
+@click.group("do")
 @mount_option
 @click.pass_obj
-def init(
-    context: BaseComposeContext,
-    limit: str,
-    mounts: t.Tuple[t.List[MountParam.MountType]],
+def do(
+    context: BaseComposeContext, mounts: t.Tuple[t.List[MountParam.MountType]]
 ) -> None:
+    """
+    Run custom jobs in the right containers.
+    """
     mount_tmp_volumes(mounts, context)
-    config = tutor_config.load(context.root)
-    runner = context.job_runner(config)
-    jobs.initialise(runner, limit_to=limit)
 
 
 @click.command(
@@ -470,11 +467,14 @@ def add_commands(command_group: click.Group) -> None:
     command_group.add_command(stop)
     command_group.add_command(restart)
     command_group.add_command(reboot)
-    command_group.add_command(init)
     command_group.add_command(dc_command)
     command_group.add_command(run)
     command_group.add_command(copyfrom)
     command_group.add_command(execute)
     command_group.add_command(logs)
     command_group.add_command(status)
-    jobs.add_commands(command_group)
+
+    @hooks.Actions.PLUGINS_LOADED.add()
+    def _add_do_commands() -> None:
+        jobs.add_job_commands(do)
+        command_group.add_command(do)
diff --git a/tutor/commands/dev.py b/tutor/commands/dev.py
@@ -105,7 +105,7 @@ def launch(
     context.invoke(compose.start, detach=True)
 
     click.echo(fmt.title("Database creation and migrations"))
-    context.invoke(compose.init)
+    context.invoke(compose.do.commands["init"])
 
     fmt.echo_info(
         """The Open edX platform is now running in detached mode