Hooks/ Plugins support to run arbitrary scripts

[mattes]

It would be nice if docker-compose up|down|.. would look for custom scripts in the current directory, like:

docker-compose.before.up
docker-compose.after.up

I guess this is mostly helpful when docker-compose is used to run the development environment. These scripts should not be meant to configure containers.

My current use-cases include:

• set up hostnames in /etc/hosts after container startup
• run docker-rsync after container started

Eventually docker-compose should allow these scripts to run in the background and stream stdout/stderr to docker-compose logs.

Related #74 #57

[mattes]

Instead of using files scripts could be baked into the docker-compose.yml file as well.

[TheFeshy]

This would also be useful for using pipework to give containers rout-able IP addresses.

It should be possible to specify whether or not Compose should wait for the script to finish before proceeding with the other containers.

[lorenz]

+1

[akoenig]

That would be a nice feature. Especially for the case when you need to execute arbitrary scripts after the application cluster is up (initializing database etc.).

So a huge +1 :)

[aanand]

In general, I'm -1 on adding functionality to Compose which runs scripts on the host machine. It breaks the abstraction - this should be the job of a Makefile, or some other script/tool which wraps Compose.

[digitalkaoz]

+1 from here, would be useful for provision the vms with non persitent data

[dnephin]

I suspect you can already use docker events (https://docs.docker.com/reference/api/docker_remote_api_v1.18/#monitor-dockers-events) to do this, without having to make any changes to compose.

[missedone]

agreed, it will be really helpful with this sort of hooks support.
we can define some hook actions as below in docker-compose.yml:

web:
  build: .
  ports:
   - "5000:5000"
  volumes:
   - .:/code
  links:
   - redis
  hooks:
   - pre_start:  xx
   - post_start: xx
redis:
  image: redis

[emersonf]

+1 ... executing database scripts against a postgres container would be brilliant, e.g.

[arioch]

+1

[hodonsky]

@aanand I believe that it is entirely up to what ever the developer runs in that script.

Plus the reason I would +1 this, is as a container spins up, it gets a self-assigned IP, and the only way to fetch that externally and apply it to another container say, for linking of some kind, is through a script or to communicate with the host machine from another container before it spins up, which isn't possible as far as I know.

This MAY allow for a more dynamic docker-compose without breaking abstraction as a practice, but leaving it up to the developer to make the mistakes.

At this point I feel like I'd have to have nested docker-compose setups... run docker compose that spins up gulp, and watches process ID's, then have gulp manage my processes instead of the host machine, I'm already feeling dirty...

[dnephin]

What about #1510, which would let you react to things without compose having to run anything directly.

[hodonsky]

@dnephin That would totally work for me, -1 here, and +1 for #1510

[mitar]

+1

[levivm]

+1

[waltervargas]

+1

[ykhrustalev]

+1, with composer I no longer want tons of Makefiles around, I want hooks, plugins and sugar in yaml

it is 21st century already

[dnephin]

Now that #1510 is in, is there still a need for this? Events give you a really easy way to react to any of the docker events.

[hodonsky]

+1 #voteToClose

[vovimayhem]

Hi guys!

I was actually considering to develop an app that operates on top of docker-compose to aid me whenever I use compose to developing my apps, making some tasks easier, such as:

• Attach to processes started with compose using the names defined in the docker-compose.yml file - I actually have use plain docker to find the container name and attach to it manually.
• Generate some env files automatically, whenever they are referenced on the docker-compose.yml file, but not present in the project folder - that would be awesome for using a bash one-liner to clone a project and start it smoothly with compose!
• Run either docker exec -ti [container_full_name] [command] or docker-compose run --rm [container_name] [command] depending if a running container exists or not.

But after seeing this discussion, I'd prefer to implement these as would-be plugins run by docker-compose itself...

What do you think?

[hodonsky]

@vovimayhem Look at the docker events link that @dnephin posted about 14 days ago as the second to last comment. I believe it has the answer you're looking for, but not in the form of an extra layer... give it a peak.

[vovimayhem]

@relicmelex I can't find the post your'e referring me to, but I guess it has something to do with Docker events...

If that's the case, Docker events must have a container starting-up, running, killed or etc ... I don't see how being able to subscribe to those events could help implementing (for example) the .env file generation before attempting to create+start the containers defined at docker-compose.yml... (not that docker events are not useful for other stuff I'm thinking about)

Do you have the link to @dnephin 's post comment?

[husobee]

I feel like docker events integration is more reactionary, versus the proactive route of having scripting hooks that are performed on docker up/down. For example sometimes I find myself running a configuration template through jinja2 and then wanting that generated configuration volume mounted into the container. With the docker events scheme that was merged in, it seems that I get the docker event that the container was created, but I can not promise that my configuration is generated prior to the entrypoint of the container running, it seems. Here is a link to a proof of concept for implementing pre_create and pre_start hooks that I find really handy in my environment: master...husobee:master

[dnephin]

I believe you could accomplish the same thing by running a bash script (or some other tool) outside of Compose, right? It feels like this doesn't need to be invoked from Compose itself. Compose already has a set of responsibilities and a change like this would unnecessarily increase those responsibilities.

[vovimayhem]

Sadly (at least for me) I must agree with @dnephin... adding more responsibilities to compose can bite our asses later on...

[jdoubleu]

A bit late but I'm still struggling with some problems.

To answer to @dnephin

Now that #1510 is in, is there still a need for this? Events give you a really easy way to react to any of the docker events.

I think docker events won't solve all problems. The main difference between the events and a hook is, that an event is fired after some task run, while a hook can be run before a sepcific task is run. And here is the point.
Imagine you have a database container (in your development environment set up with docker-compose) and you want to create a databasedump when the container will be killed.
You'll only receive the events when the container died and was killed, so far.
A hook to run a script (or whatever) before the container dies would be nice at this point. Of course this problem can be solved by adding more events.

[dnephin]

Why would you need a hook for that? Just write a script that does a dump, then shuts down the database.

[jdoubleu]

That is an option.
But I would like to use docker-compose only if possible and this would be a great feature.

If you're using docker in large scale or want to create scheduled backups in the cloud the solution with a script may not be possible.

[gw0]

@dnephin In the lines of your argument... Why would anyone need something like Docker? Just write a script that sets up an overlay fs, configures virtual network devices and iptables, drops some permissions, sets up cgroups and accouting, and finally runs the given command inside a chroot.

[dnephin]

Sure, that's one option. Sometimes a two line script is appropriate (my example), and sometimes it's less appropriate.

[feffi]

-1 as it breaks abstraction, as mentioned above

[electrofelix]

Seems PR #3905 would allow for such behaviour to remain outside of docker-compose while still providing a common entry point for those that have use cases outside of the core remit.

[JamesBewley]

+1 Hooks for bringing down would be great. I would use this to dump a database before bringing everything down.

[shin-]

Expressed my feelings about this proposal in #3905 - it's unfortunately too much of a burden to support at that stage of the project, and I'm afraid we just won't be doing it.

[Muhd]

Of course one could always write a script to fix any problem with docker, but once you use a script you are losing a lot of the value of docker and compose. The greatest value of having docker compose is standardization—no matter what team you are developing with, if they are using docker standard dockerfiles and compose files you should be able to figure out the whole development setup. Having use cases where you are forced to write custom scripts as entry points for the developer running a container means this is no longer true—there may be scripts that need to be run that aren't documented in the compose file and thus anyone using docker will have additional training overhead for developers new to their team, even developers already familiar with docker, since there is no standardized way of covering certain use cases.

[lucile-sticky]

So sad that this issue have been closed because of some refractoriness to evolution 😞

The greatest value of having docker compose is standardization

That's the point. If we could "just" write a .sh file or whatever to do the job without using Docker Compose, why Docker Compose is existing? 😕

We can understand that is a big job, as @shin- said:

it's unfortunately too much of a burden to support at that stage of the project

❤️

But you can't say "Make a script" what means "Hey, that's too hard, we're not gonna make it".

If it's hard to do it, just say "Your idea is interesting, and it fills some needs, but it's really difficult to do and we don't have resources to do it at this time... Maybe could you develop it and ask a pull request" or something like that 💡

[dopry]

At the very least it would be good to see these things in a backlog and not just dismissed.

[vovimayhem]

Dudes, I started a while ago a project called plis, and something I wanted to do is to be able to do stuff before firing up a docker-compose project, in particular interest to be able to write dotenv files to achieve a true 'clone & run' flow for development environments.

I still haven't done that, but you might want to give it a try nonetheless.

[dnephin]

Running scripts on the host is out-of-scope for the docker/compose project, so it would be misleading to add it to a backlog.

docker-compose is not a build automation tool. It's "a tool for defining and running multi-container Docker application".

dobi is a tool for build automation that I have been working on. It may solve some of these problems by running tasks in containers.

[gw0]

@lucile-sticky Afaik running an external command on the host is pretty trivial in any programming language. So it is not too hard, they are just stubborn and fo not want to do it. Maybe it is time to fork this project?

@dnephin Exactly, it is a tool for defining multi-container Docker applications. Unfortunately some applications need to make custom adjustments on the host to work (eg. setup special networks, configure iptables, enable kernel modules). Just adding a simple host pre- and post-command for expert use, that can be used only if you specify run Docker Compose with a special parameter eg. docker-compose --dangerous, would solve all issues.

[lucile-sticky]

@dnephin I didn't read anything in the original post about

Running scripts on the host

😕

But after reading it again, I saw that @mattes wrote:

These scripts should not be meant to configure containers.

Sorry for the misunderstood ^-^'

Maybe the original post could be edited to make it more clearer? (I'm not an English native speaker, so those little sentences hidden in a paragraph are more difficult to spot than if it was said at the beginning of the issue 😶)

In this case, I agree that if those script have to be run on the host and not on the containers, that's not the job of Docker Compose to deal with it.

[route1rodent]

Case scenario: you have other projects installed via a package manager, e.g. with yarn, under node_modules which contain docker-compose.yml files you want to extend in your own yml file.

Without a pre-hook that would install the npm/yarn dependencies it's not possible to run docker-compose up, because the yml files we want to extend may not exist... or is there any other good practice in order to accomplish this?

I see a pre/post-hook as a special container, that will be executed before the services part of the yml file is processed and started. This way these hooks will require a container, and won't be executed in the host. That could go in a hooks / hook_services section, having the same syntax as any other service.

What do you think @dnephin ?

[lucile-sticky]

@Metaunicorn it's exactly the use case described here: #1809 (comment)

Aaaaand it's a nope from the dev team, I guess.

[38leinaD]

+1

[pedro93]

As @gw0 mentioned some applications require special pre-bootup modifications based on information accessible only on the host during the docker-compose up process.

I'm currently trying to creating a setup framework with docker-compose for a project in my company which has special networks needs such as multi-clustering communication using iptables to communicate out of the container (containers are merely one of our deployment solutions, not only for used testing though that is a great boon) which must be defined before the entry point stage.

However, this information is dynamically generated by certain dependency services when they are up and running during the docker-compose up process. Without pre-entry point hooks I have no way to configure these services as I won't know in the dockerfile the information I need.

[gdsoftwares8]

HI all,
I am getting this error "Unsupported config option for services.master_crawler: 'hooks'"
below is my docker compose file is

version: "2"
services:
master_crawler:
image: test:test
volumes:
- master/app-resources.yml:/app-resources.yml
ports:
- "4444"
- "9080"
hooks:
- pre_start: xx
- post_start: xx

which version we need to mention in docker compose in order to use hooks

[JoseFMP]

@gdsoftwares8 the hooks thing was like a suggestion to be adder into docker compose =) I do not think is there if you read the thread.

[shirakaba]

Would really enjoy having a 'before up' lifecycle hook to create an external network if it doesn't already exist, solving #2846

[duzun]

I have to share this here, it is genial!
#1510 comment @hanoii