diff --git a/CHANGELOG-nightly.md b/CHANGELOG-nightly.md index ee725c91ccc..9907b70a366 100644 --- a/CHANGELOG-nightly.md +++ b/CHANGELOG-nightly.md @@ -8,6 +8,8 @@ will be backported to the master branch at every major release. When backporting changes to master, we should keep only the entries that correspond to user- facing changes. --> +- 💥[Feature] Add an extensible `local/dev/k8s do ...` command to trigger custom job commands. These commands are used to run a series of bash scripts in designated containers. Any plugin can add custom jobs thanks to the `CLI_DO_COMMANDS` filter. This causes the following breaking changes: + - The "init", "createuser", "settheme", "importdemocourse" commands were all migrated to this new interface. For instance, `tutor local init` was replaced by `tutor local do init`. - 💥[Improvement] Remove the `local/dev bindmount` commands, which have been marked as deprecated for some time. The `--mount` option should be used instead. - 💥[Bugfix] Fix local installation requirements. Plugins that implemented the "openedx-dockerfile-post-python-requirements" patch and that needed access to the edx-platform repo will no longer work. Instead, these plugins should implement the "openedx-dockerfile-pre-assets" patch. This scenario should be very rare, though. (by @regisb) - 💥[Improvement] Rename the implementation of tutor quickstart to tutor launch. (by @Carlos-Muniz) diff --git a/docs/local.rst b/docs/local.rst index 590692fce86..d9c4e1331af 100644 --- a/docs/local.rst +++ b/docs/local.rst @@ -80,7 +80,7 @@ Service initialisation :: - tutor local init + tutor local do init This command should be run just once. It will initialise all applications in a running platform. In particular, this will create the required databases tables and apply database migrations for all applications. @@ -120,7 +120,7 @@ Creating a new user with staff and admin rights You will most certainly need to create a user to administer the platform. Just run:: - tutor local createuser --staff --superuser yourusername user@email.com + tutor local do createuser --staff --superuser yourusername user@email.com You will be asked to set the user password interactively. @@ -131,7 +131,7 @@ Importing the demo course After a fresh installation, your platform will not have a single course. To import the `Open edX demo course `_, run:: - tutor local importdemocourse + tutor local do importdemocourse .. _settheme: @@ -140,7 +140,7 @@ Setting a new theme The default Open edX theme is rather bland, so Tutor makes it easy to switch to a different theme:: - tutor local settheme mytheme + tutor local do settheme mytheme Out of the box, only the default "open-edx" theme is available. We also developed `Indigo, a beautiful, customizable theme `__ which is easy to install with Tutor. diff --git a/docs/tutorials/arm64.rst b/docs/tutorials/arm64.rst index aa172c61b93..5978b9ba3e7 100644 --- a/docs/tutorials/arm64.rst +++ b/docs/tutorials/arm64.rst @@ -48,9 +48,9 @@ Finish setup and start Tutor From this point on, use Tutor as normal. For example, start Open edX and run migrations with:: tutor local start -d - tutor local init + tutor local do init Or for a development environment:: tutor dev start -d - tutor dev init + tutor dev do init diff --git a/docs/tutorials/plugin.rst b/docs/tutorials/plugin.rst index c4ca5b0a159..e04844af04b 100644 --- a/docs/tutorials/plugin.rst +++ b/docs/tutorials/plugin.rst @@ -218,7 +218,7 @@ You can now run the "myservice" container which will execute the ``CMD`` stateme Declaring initialisation tasks ------------------------------ -Services often need to run specific tasks before they can be started. For instance, the LMS and the CMS need to apply database migrations. These commands are written in shell scripts that are executed whenever we run ``launch``. We call these scripts "init tasks". To add a new local init task, we must first add the corresponding service to the ``docker-compose-jobs.yml`` file by implementing the :patch:`local-docker-compose-jobs-services` patch:: +Services often need to run specific tasks before they can be started. For instance, the LMS and the CMS need to apply database migrations. These commands are written in shell scripts that are executed whenever we run ``launch``. We call these scripts "init tasks". To add a new local initialisation task, we must first add the corresponding service to the ``docker-compose-jobs.yml`` file by implementing the :patch:`local-docker-compose-jobs-services` patch:: hooks.Filters.ENV_PATCHES.add_item( ( @@ -251,7 +251,7 @@ Add our init task script to the :py:data:`tutor.hooks.Filters.COMMANDS_INIT` fil Run this initialisation task with:: - $ tutor local init --limit=myplugin + $ tutor local do init --limit=myplugin ... Running init task: myplugin/tasks/init.sh ... diff --git a/docs/tutorials/theming.rst b/docs/tutorials/theming.rst index de2c7a2d21c..224a5595a23 100644 --- a/docs/tutorials/theming.rst +++ b/docs/tutorials/theming.rst @@ -48,7 +48,7 @@ Then, run a local webserver:: The LMS can then be accessed at http://local.overhang.io:8000. You will then have to :ref:`enable that theme `:: - tutor dev settheme mythemename + tutor dev do settheme mythemename Watch the themes folders for changes (in a different terminal):: diff --git a/tests/commands/base.py b/tests/commands/base.py index a39d934b09c..e2f4ed6d5ef 100644 --- a/tests/commands/base.py +++ b/tests/commands/base.py @@ -17,7 +17,9 @@ def invoke(args: t.List[str]) -> click.testing.Result: return TestCommandMixin.invoke_in_root(root, args) @staticmethod - def invoke_in_root(root: str, args: t.List[str]) -> click.testing.Result: + def invoke_in_root( + root: str, args: t.List[str], catch_exceptions: bool = True + ) -> click.testing.Result: """ Use this method for commands that all need to run in the same root: @@ -32,4 +34,6 @@ def invoke_in_root(root: str, args: t.List[str]) -> click.testing.Result: "TUTOR_IGNORE_DICT_PLUGINS": "1", } ) - return runner.invoke(cli, args, obj=TestContext(root)) + return runner.invoke( + cli, args, obj=TestContext(root), catch_exceptions=catch_exceptions + ) diff --git a/tests/commands/test_jobs.py b/tests/commands/test_jobs.py index 687fc2452f2..f5f8bb3ce2e 100644 --- a/tests/commands/test_jobs.py +++ b/tests/commands/test_jobs.py @@ -1,71 +1,74 @@ -import re -import unittest -from io import StringIO from unittest.mock import patch -from tests.helpers import TestContext, temporary_root -from tutor import config as tutor_config +from tests.helpers import PluginsTestCase, temporary_root from tutor.commands import jobs +from .base import TestCommandMixin -class JobsTests(unittest.TestCase): - @patch("sys.stdout", new_callable=StringIO) - def test_initialise(self, mock_stdout: StringIO) -> None: + +class JobsTests(PluginsTestCase, TestCommandMixin): + def test_initialise(self) -> None: with temporary_root() as root: - context = TestContext(root) - config = tutor_config.load_full(root) - runner = context.job_runner(config) - jobs.initialise(runner) - output = mock_stdout.getvalue().strip() - self.assertTrue(output.startswith("Initialising all services...")) - self.assertTrue(output.endswith("All services initialised.")) + self.invoke_in_root(root, ["config", "save"]) + result = self.invoke_in_root(root, ["local", "do", "init"]) + self.assertIsNone(result.exception) + self.assertEqual(0, result.exit_code) + self.assertIn("All services initialised.", result.output) - def test_create_user_command_without_staff(self) -> None: - command = jobs.create_user_template("superuser", False, "username", "email", "p4ssw0rd") + def test_create_user_template_without_staff(self) -> None: + command = jobs.create_user_template( + "superuser", False, "username", "email", "p4ssw0rd" + ) self.assertNotIn("--staff", command) self.assertIn("set_password", command) - def test_create_user_command_with_staff(self) -> None: - command = jobs.create_user_template("superuser", True, "username", "email", "p4ssw0rd") + def test_create_user_template_with_staff(self) -> None: + command = jobs.create_user_template( + "superuser", True, "username", "email", "p4ssw0rd" + ) self.assertIn("--staff", command) - @patch("sys.stdout", new_callable=StringIO) - def test_import_demo_course(self, mock_stdout: StringIO) -> None: + def test_import_demo_course(self) -> None: with temporary_root() as root: - context = TestContext(root) - config = tutor_config.load_full(root) - runner = context.job_runner(config) - runner.run_job_from_str("cms", jobs.import_demo_course_template()) - - output = mock_stdout.getvalue() - service = re.search(r"Service: (\w*)", output) - commands = re.search(r"(-----)([\S\s]+)(-----)", output) - assert service is not None - assert commands is not None - self.assertEqual(service.group(1), "cms") - self.assertTrue( - commands.group(2) - .strip() - .startswith('echo "Loading settings $DJANGO_SETTINGS_MODULE"') - ) + self.invoke_in_root(root, ["config", "save"]) + with patch("tutor.utils.docker_compose") as mock_docker_compose: + result = self.invoke_in_root(root, ["local", "do", "importdemocourse"]) + dc_args, _dc_kwargs = mock_docker_compose.call_args + self.assertIsNone(result.exception) + self.assertEqual(0, result.exit_code) + self.assertIn("cms-job", dc_args) + self.assertTrue( + dc_args[-1] + .strip() + .startswith('echo "Loading settings $DJANGO_SETTINGS_MODULE"') + ) - @patch("sys.stdout", new_callable=StringIO) - def test_set_theme(self, mock_stdout: StringIO) -> None: + def test_set_theme(self) -> None: with temporary_root() as root: - context = TestContext(root) - config = tutor_config.load_full(root) - runner = context.job_runner(config) - command = jobs.set_theme_template("sample_theme", ["domain1", "domain2"]) - runner.run_job_from_str("lms", command) + self.invoke_in_root(root, ["config", "save"]) + with patch("tutor.utils.docker_compose") as mock_docker_compose: + result = self.invoke_in_root( + root, + [ + "local", + "do", + "settheme", + "--domain", + "domain1", + "--domain", + "domain2", + "beautiful", + ], + ) + dc_args, _dc_kwargs = mock_docker_compose.call_args - output = mock_stdout.getvalue() - service = re.search(r"Service: (\w*)", output) - commands = re.search(r"(-----)([\S\s]+)(-----)", output) - assert service is not None - assert commands is not None - self.assertEqual(service.group(1), "lms") + self.assertIsNone(result.exception) + self.assertEqual(0, result.exit_code) + self.assertIn("lms-job", dc_args) self.assertTrue( - commands.group(2) + dc_args[-1] .strip() .startswith('echo "Loading settings $DJANGO_SETTINGS_MODULE"') ) + self.assertIn("assign_theme('beautiful', 'domain1')", dc_args[-1]) + self.assertIn("assign_theme('beautiful', 'domain2')", dc_args[-1]) diff --git a/tests/test_env.py b/tests/test_env.py index f0de1747e89..a752c69c1e5 100644 --- a/tests/test_env.py +++ b/tests/test_env.py @@ -83,7 +83,7 @@ def test_render_file(self) -> None: tutor_config.render_full(config) config["MYSQL_ROOT_PASSWORD"] = "testpassword" - rendered = env.render_file(config, "hooks", "mysql", "init") + rendered = env.render_file(config, "jobs", "init", "mysql.sh") self.assertIn("testpassword", rendered) @patch.object(tutor_config.fmt, "echo") diff --git a/tutor/commands/compose.py b/tutor/commands/compose.py index f8000ff5ad7..b918d0caa18 100644 --- a/tutor/commands/compose.py +++ b/tutor/commands/compose.py @@ -297,19 +297,16 @@ def restart(context: BaseComposeContext, services: t.List[str]) -> None: context.job_runner(config).docker_compose(*command) -@click.command(help="Initialise all applications") -@click.option("-l", "--limit", help="Limit initialisation to this service or plugin") +@click.group("do") @mount_option @click.pass_obj -def init( - context: BaseComposeContext, - limit: str, - mounts: t.Tuple[t.List[MountParam.MountType]], +def do( + context: BaseComposeContext, mounts: t.Tuple[t.List[MountParam.MountType]] ) -> None: + """ + Run custom jobs in the right containers. + """ mount_tmp_volumes(mounts, context) - config = tutor_config.load(context.root) - runner = context.job_runner(config) - jobs.initialise(runner, limit_to=limit) @click.command( @@ -470,11 +467,14 @@ def add_commands(command_group: click.Group) -> None: command_group.add_command(stop) command_group.add_command(restart) command_group.add_command(reboot) - command_group.add_command(init) command_group.add_command(dc_command) command_group.add_command(run) command_group.add_command(copyfrom) command_group.add_command(execute) command_group.add_command(logs) command_group.add_command(status) - jobs.add_commands(command_group) + + @hooks.Actions.PLUGINS_LOADED.add() + def _add_do_commands() -> None: + jobs.add_job_commands(do) + command_group.add_command(do) diff --git a/tutor/commands/dev.py b/tutor/commands/dev.py index 46dfb057a74..6176656ff3c 100644 --- a/tutor/commands/dev.py +++ b/tutor/commands/dev.py @@ -105,7 +105,7 @@ def launch( context.invoke(compose.start, detach=True) click.echo(fmt.title("Database creation and migrations")) - context.invoke(compose.init) + context.invoke(compose.do.commands["init"]) fmt.echo_info( """The Open edX platform is now running in detached mode diff --git a/tutor/commands/jobs.py b/tutor/commands/jobs.py index eef81c3b67e..e6c591b2d07 100644 --- a/tutor/commands/jobs.py +++ b/tutor/commands/jobs.py @@ -1,19 +1,24 @@ """ Common jobs that must be added both to local, dev and k8s commands. """ - +import functools import typing as t import click +from typing_extensions import ParamSpec from tutor import config as tutor_config -from tutor import fmt, hooks, jobs +from tutor import env, fmt, hooks -from .context import BaseJobContext -BASE_OPENEDX_COMMAND = """ -echo "Loading settings $DJANGO_SETTINGS_MODULE" -""" +def add_job_commands(do_command_group: click.Group) -> None: + """ + This is meant to be called with the `local/dev/k8s do` group commands, to add the + different `do` subcommands. + """ + subcommands: t.Iterator[click.Command] = hooks.Filters.CLI_DO_COMMANDS.iterate() + for subcommand in subcommands: + do_command_group.add_command(subcommand) @hooks.Actions.CORE_READY.add() @@ -25,16 +30,18 @@ def _add_core_init_tasks() -> None: the --limit argument. """ with hooks.Contexts.APP("mysql").enter(): - hooks.Filters.COMMANDS_INIT.add_item(("mysql", ("hooks", "mysql", "init"))) + hooks.Filters.COMMANDS_INIT.add_item(("mysql", ("jobs", "init", "mysql.sh"))) with hooks.Contexts.APP("lms").enter(): - hooks.Filters.COMMANDS_INIT.add_item(("lms", ("hooks", "lms", "init"))) + hooks.Filters.COMMANDS_INIT.add_item(("lms", ("jobs", "init", "lms.sh"))) with hooks.Contexts.APP("cms").enter(): - hooks.Filters.COMMANDS_INIT.add_item(("cms", ("hooks", "cms", "init"))) + hooks.Filters.COMMANDS_INIT.add_item(("cms", ("jobs", "init", "cms.sh"))) -def initialise(runner: jobs.BaseJobRunner, limit_to: t.Optional[str] = None) -> None: +@click.command("init", help="Initialise all applications") +@click.option("-l", "--limit", help="Limit initialisation to this service or plugin") +def initialise(limit: t.Optional[str]) -> t.Iterator[t.Tuple[str, str]]: fmt.echo_info("Initialising all services...") - filter_context = hooks.Contexts.APP(limit_to).name if limit_to else None + filter_context = hooks.Contexts.APP(limit).name if limit else None # Pre-init tasks iter_pre_init_tasks: t.Iterator[ @@ -42,7 +49,7 @@ def initialise(runner: jobs.BaseJobRunner, limit_to: t.Optional[str] = None) -> ] = hooks.Filters.COMMANDS_PRE_INIT.iterate(context=filter_context) for service, path in iter_pre_init_tasks: fmt.echo_info(f"Running pre-init task: {'/'.join(path)}") - runner.run_job_from_template(service, *path) + yield service, env.read_template_file(*path) # Init tasks iter_init_tasks: t.Iterator[ @@ -50,7 +57,7 @@ def initialise(runner: jobs.BaseJobRunner, limit_to: t.Optional[str] = None) -> ] = hooks.Filters.COMMANDS_INIT.iterate(context=filter_context) for service, path in iter_init_tasks: fmt.echo_info(f"Running init task: {'/'.join(path)}") - runner.run_job_from_template(service, *path) + yield service, env.read_template_file(*path) fmt.echo_info("All services initialised.") @@ -67,49 +74,14 @@ def initialise(runner: jobs.BaseJobRunner, limit_to: t.Optional[str] = None) -> ) @click.argument("name") @click.argument("email") -@click.pass_obj def createuser( - context: BaseJobContext, superuser: str, staff: bool, password: str, name: str, email: str, -) -> None: - run_job( - context, "lms", create_user_template(superuser, staff, name, email, password) - ) - - -@click.command(help="Import the demo course") -@click.pass_obj -def importdemocourse(context: BaseJobContext) -> None: - run_job(context, "cms", import_demo_course_template()) - - -@click.command( - help="Assign a theme to the LMS and the CMS. To reset to the default theme , use 'default' as the theme name." -) -@click.option( - "-d", - "--domain", - "domains", - multiple=True, - help=( - "Limit the theme to these domain names. By default, the theme is " - "applied to the LMS and the CMS, both in development and production mode" - ), -) -@click.argument("theme_name") -@click.pass_obj -def settheme(context: BaseJobContext, domains: t.List[str], theme_name: str) -> None: - run_job(context, "lms", set_theme_template(theme_name, domains)) - - -def run_job(context: BaseJobContext, service: str, command: str) -> None: - config = tutor_config.load(context.root) - runner = context.job_runner(config) - runner.run_job_from_str(service, command) +) -> t.Iterable[t.Tuple[str, str]]: + yield ("lms", create_user_template(superuser, staff, name, email, password)) def create_user_template( @@ -120,9 +92,7 @@ def create_user_template( opts += " --superuser" if staff: opts += " --staff" - return ( - BASE_OPENEDX_COMMAND - + f""" + return f""" ./manage.py lms manage_user {opts} {username} {email} ./manage.py lms shell -c " from django.contrib.auth import get_user_model @@ -130,20 +100,36 @@ def create_user_template( u.set_password('{password}') u.save()" """ - ) -def import_demo_course_template() -> str: - return ( - BASE_OPENEDX_COMMAND - + """ +@click.command(help="Import the demo course") +def importdemocourse() -> t.Iterable[t.Tuple[str, str]]: + template = """ # Import demo course git clone https://github.com/openedx/edx-demo-course --branch {{ OPENEDX_COMMON_VERSION }} --depth 1 ../edx-demo-course python ./manage.py cms import ../data ../edx-demo-course # Re-index courses ./manage.py cms reindex_course --all --setup""" - ) + yield ("cms", template) + + +@click.command( + help="Assign a theme to the LMS and the CMS. To reset to the default theme , use 'default' as the theme name." +) +@click.option( + "-d", + "--domain", + "domains", + multiple=True, + help=( + "Limit the theme to these domain names. By default, the theme is " + "applied to the LMS and the CMS, both in development and production mode" + ), +) +@click.argument("theme_name") +def settheme(domains: t.List[str], theme_name: str) -> t.Iterable[t.Tuple[str, str]]: + yield ("lms", set_theme_template(theme_name, domains)) def set_theme_template(theme_name: str, domain_names: t.List[str]) -> str: @@ -179,9 +165,72 @@ def assign_theme(name, domain): ] for domain_name in domain_names: python_command += f"assign_theme('{theme_name}', '{domain_name}')\n" - return BASE_OPENEDX_COMMAND + f'./manage.py lms shell -c "{python_command}"' + return f'./manage.py lms shell -c "{python_command}"' + + +hooks.Filters.CLI_DO_COMMANDS.add_items( + [ + createuser, + importdemocourse, + initialise, + settheme, + ] +) + + +def do_callback(service_commands: t.Iterable[t.Tuple[str, str]]) -> None: + """ + This function must be added as a callback to all `do` subcommands. + + `do` subcommands don't actually run any task. They just yield tuples of (service + name, unrendered script string). This function is responsible for actually running + the scripts. It does the following: + + - Prefix the script with a base command + - Render the script string + - Run a job in the right container + + In order to be added as a callback to the do subcommands, the + `_patch_do_commands_callbacks` must be called. + """ + context = click.get_current_context().obj + config = tutor_config.load(context.root) + runner = context.job_runner(config) + base_openedx_command = """ +echo "Loading settings $DJANGO_SETTINGS_MODULE" +""" + for service, command in service_commands: + runner.run_job_from_str(service, base_openedx_command + command) + + +@hooks.Actions.PLUGINS_LOADED.add() +def _patch_do_commands_callbacks() -> None: + """ + After plugins have been loaded, patch `do` subcommands such that their output is + forwarded to `do_callback`. + """ + subcommands: t.Iterator[click.Command] = hooks.Filters.CLI_DO_COMMANDS.iterate() + for subcommand in subcommands: + # Modify the subcommand callback such that job results are processed by do_callback + if subcommand.callback is None: + raise ValueError("Cannot patch None callback") + subcommand.callback = _patch_callback(subcommand.callback) + + +P = ParamSpec("P") + + +def _patch_callback( + func: t.Callable[P, t.Iterable[t.Tuple[str, str]]] +) -> t.Callable[P, None]: + """ + Modify a subcommand callback function such that its results are processed by `do_callback`. + """ + + def new_callback(*args: P.args, **kwargs: P.kwargs) -> None: + do_callback(func(*args, **kwargs)) + # Make the new callback behave like the old one + functools.update_wrapper(new_callback, func) -def add_commands(command_group: click.Group) -> None: - for job_command in [createuser, importdemocourse, settheme]: - command_group.add_command(job_command) + return new_callback diff --git a/tutor/commands/k8s.py b/tutor/commands/k8s.py index 9b3e85d20e4..2e2477a0ad2 100644 --- a/tutor/commands/k8s.py +++ b/tutor/commands/k8s.py @@ -6,7 +6,7 @@ from tutor import config as tutor_config from tutor import env as tutor_env -from tutor import exceptions, fmt +from tutor import exceptions, fmt, hooks from tutor import interactive as interactive_config from tutor import serialize, utils from tutor.commands import jobs @@ -22,7 +22,8 @@ class K8sClients: def __init__(self) -> None: # Loading the kubernetes module here to avoid import overhead - from kubernetes import client, config # pylint: disable=import-outside-toplevel + # pylint: disable=import-outside-toplevel + from kubernetes import client, config config.load_kube_config() self._batch_api = None @@ -341,17 +342,27 @@ def delete(context: K8sContext, yes: bool) -> None: ) -@click.command(help="Initialise all applications") -@click.option("-l", "--limit", help="Limit initialisation to this service or plugin") +@click.group("do") @click.pass_obj -def init(context: K8sContext, limit: Optional[str]) -> None: +def do(context: K8sContext) -> None: + """ + Run custom jobs in the right containers. + + We make sure that some essential containers (databases, proxy) are up before we + launch the jobs. + """ config = tutor_config.load(context.root) - runner = context.job_runner(config) wait_for_deployment_ready(config, "caddy") for name in ["elasticsearch", "mysql", "mongodb"]: if tutor_config.is_service_activated(config, name): wait_for_deployment_ready(config, name) - jobs.initialise(runner, limit_to=limit) + + +@click.command(help="Initialise all applications") +@click.option("-l", "--limit", help="Limit initialisation to this service or plugin") +@click.pass_context +def init(context: click.Context, limit: Optional[str]) -> None: + context.invoke(do.commands["init"], limit=limit) @click.command(help="Scale the number of replicas of a given deployment") @@ -540,4 +551,9 @@ def k8s_namespace(config: Config) -> str: k8s.add_command(upgrade) k8s.add_command(apply_command) k8s.add_command(status) -jobs.add_commands(k8s) + + +@hooks.Actions.PLUGINS_LOADED.add() +def _add_k8s_do_commands() -> None: + jobs.add_job_commands(do) + k8s.add_command(do) diff --git a/tutor/commands/local.py b/tutor/commands/local.py index e4614c3f3ed..6ba52b14e85 100644 --- a/tutor/commands/local.py +++ b/tutor/commands/local.py @@ -142,7 +142,7 @@ def launch( click.echo(fmt.title("Starting the platform in detached mode")) context.invoke(compose.start, detach=True) click.echo(fmt.title("Database creation and migrations")) - context.invoke(compose.init) + context.invoke(compose.do.commands["init"]) config = tutor_config.load(context.obj.root) fmt.echo_info( diff --git a/tutor/hooks/consts.py b/tutor/hooks/consts.py index 9a09cb9702a..7cb73b9e592 100644 --- a/tutor/hooks/consts.py +++ b/tutor/hooks/consts.py @@ -104,6 +104,20 @@ def your_filter(items): return items """ + #: List of command line interface (CLI) commands. + #: + #: :parameter list commands: commands are instances of ``click.Command``. They will + #: all be added as subcommands of the main ``tutor`` command. + CLI_COMMANDS = filters.get("cli:commands") + + # List of `do ...` commands. + #: + #: :parameter list commands: see :py:data:`CLI_COMMANDS`. These commands will be + #: added as subcommands to the `local/dev/k8s do` commands. They must return a list of + #: ("service name", "service command") tuples. Each "service command" will be executed + #: in the "service" container, both in local, dev and k8s mode. + CLI_DO_COMMANDS = filters.get("cli:commands:do") + #: List of commands to be executed during initialization. These commands typically #: include database migrations, setting feature flags, etc. #: @@ -111,7 +125,7 @@ def your_filter(items): #: #: - ``service`` is the name of the container in which the task will be executed. #: - ``path`` is a tuple that corresponds to a template relative path. - #: Example: ``("myplugin", "hooks", "myservice", "pre-init")`` (see:py:data:`IMAGES_BUILD`). + #: Example: ``("myplugin", "hooks", "myservice", "pre-init")`` (see :py:data:`IMAGES_BUILD`). #: The command to execute will be read from that template, after it is rendered. COMMANDS_INIT = filters.get("commands:init") @@ -159,40 +173,6 @@ def your_filter(items): #: Same as :py:data:`COMPOSE_LOCAL_TMP` but for jobs COMPOSE_LOCAL_JOBS_TMP = filters.get("compose:local-jobs:tmp") - #: List of images to be built when we run ``tutor images build ...``. - #: - #: :parameter list[tuple[str, tuple[str, ...], str, tuple[str, ...]]] tasks: list of ``(name, path, tag, args)`` tuples. - #: - #: - ``name`` is the name of the image, as in ``tutor images build myimage``. - #: - ``path`` is the relative path to the folder that contains the Dockerfile. - #: For instance ``("myplugin", "build", "myservice")`` indicates that the template will be read from - #: ``myplugin/build/myservice/Dockerfile`` - #: - ``tag`` is the Docker tag that will be applied to the image. It will be - #: rendered at runtime with the user configuration. Thus, the image tag could - #: be ``"{{ DOCKER_REGISTRY }}/myimage:{{ TUTOR_VERSION }}"``. - #: - ``args`` is a list of arguments that will be passed to ``docker build ...``. - #: :parameter dict config: user configuration. - IMAGES_BUILD = filters.get("images:build") - - #: List of images to be pulled when we run ``tutor images pull ...``. - #: - #: :parameter list[tuple[str, str]] tasks: list of ``(name, tag)`` tuples. - #: - #: - ``name`` is the name of the image, as in ``tutor images pull myimage``. - #: - ``tag`` is the Docker tag that will be applied to the image. (see :py:data:`IMAGES_BUILD`). - #: :parameter dict config: user configuration. - IMAGES_PULL = filters.get("images:pull") - - #: List of images to be pushed when we run ``tutor images push ...``. - #: Parameters are the same as for :py:data:`IMAGES_PULL`. - IMAGES_PUSH = filters.get("images:push") - - #: List of command line interface (CLI) commands. - #: - #: :parameter list commands: commands are instances of ``click.Command``. They will - #: all be added as subcommands of the main ``tutor`` command. - CLI_COMMANDS = filters.get("cli:commands") - #: Declare new default configuration settings that don't necessarily have to be saved in the user #: ``config.yml`` file. Default settings may be overridden with ``tutor config save --set=...``, in which #: case they will automatically be added to ``config.yml``. @@ -298,6 +278,34 @@ def your_filter(items): #: :parameter filters: list of (name, value) tuples. ENV_TEMPLATE_VARIABLES = filters.get("env:templates:variables") + #: List of images to be built when we run ``tutor images build ...``. + #: + #: :parameter list[tuple[str, tuple[str, ...], str, tuple[str, ...]]] tasks: list of ``(name, path, tag, args)`` tuples. + #: + #: - ``name`` is the name of the image, as in ``tutor images build myimage``. + #: - ``path`` is the relative path to the folder that contains the Dockerfile. + #: For instance ``("myplugin", "build", "myservice")`` indicates that the template will be read from + #: ``myplugin/build/myservice/Dockerfile`` + #: - ``tag`` is the Docker tag that will be applied to the image. It will be + #: rendered at runtime with the user configuration. Thus, the image tag could + #: be ``"{{ DOCKER_REGISTRY }}/myimage:{{ TUTOR_VERSION }}"``. + #: - ``args`` is a list of arguments that will be passed to ``docker build ...``. + #: :parameter dict config: user configuration. + IMAGES_BUILD = filters.get("images:build") + + #: List of images to be pulled when we run ``tutor images pull ...``. + #: + #: :parameter list[tuple[str, str]] tasks: list of ``(name, tag)`` tuples. + #: + #: - ``name`` is the name of the image, as in ``tutor images pull myimage``. + #: - ``tag`` is the Docker tag that will be applied to the image. (see :py:data:`IMAGES_BUILD`). + #: :parameter dict config: user configuration. + IMAGES_PULL = filters.get("images:pull") + + #: List of images to be pushed when we run ``tutor images push ...``. + #: Parameters are the same as for :py:data:`IMAGES_PULL`. + IMAGES_PUSH = filters.get("images:push") + #: List of installed plugins. In order to be added to this list, a plugin must first #: be discovered (see :py:data:`Actions.CORE_READY`). #: diff --git a/tutor/templates/hooks/cms/init b/tutor/templates/jobs/init/cms.sh similarity index 100% rename from tutor/templates/hooks/cms/init rename to tutor/templates/jobs/init/cms.sh diff --git a/tutor/templates/hooks/lms/init b/tutor/templates/jobs/init/lms.sh similarity index 100% rename from tutor/templates/hooks/lms/init rename to tutor/templates/jobs/init/lms.sh diff --git a/tutor/templates/hooks/mysql/init b/tutor/templates/jobs/init/mysql.sh similarity index 100% rename from tutor/templates/hooks/mysql/init rename to tutor/templates/jobs/init/mysql.sh diff --git a/tutor/utils.py b/tutor/utils.py index 122a9b3a521..f2d6b805497 100644 --- a/tutor/utils.py +++ b/tutor/utils.py @@ -203,6 +203,10 @@ def is_a_tty() -> bool: def execute(*command: str) -> int: click.echo(fmt.command(_shlex_join(*command))) + return execute_silent(*command) + + +def execute_silent(*command: str) -> int: with subprocess.Popen(command) as p: try: result = p.wait(timeout=None)