feat: add environment variable representing the path to the build context

[mdelapenya]

What does this PR do?

This PR adds an environment variable to the build context of the Docker images for metricbeat's integrations. This variable, named after service's name appending '_PATH' (uppercase, i.e. APACHE_PATH), will have a default value of '.', which is the current value to all of them.

Why is it important?

We'd like to align the docker compose files in the end-2-end tests PoC (https://github.com/elastic/metricbeat-tests-poc) with those located in the Beats repo, so that we consume the same artifacts with same configurations.

The reason why it is not possible before these changes is because of Docker Compose's expected behavior: when using multiple docker-compose files (as we do in the PoC), any relative path in the build-chain must be discoverable by the first -f compose.yml element (See docker/compose#3568 and docker/compose#3874). With current approach, existing compose files cannot be consumed by anybody else but Beats, as it expects having access to the ./_meta from the build context. (See https://github.com/elastic/beats/blob/master/metricbeat/module/apache/docker-compose.yml#L1-L11)

Because of that, from PoC perspective, where we want to consume same compose files, we need a manner to discover each _meta directory from a first compose file, and the easiest manner is to override an environment variable with the path to the compose, which is known in runtime by the PoC, so it's able to put the proper value, resolving each _meta directory for each integration.

Checklist

• My code follows the style guidelines of this project
  - [ ] I have commented my code, particularly in hard-to-understand areas
  - [ ] I have made corresponding changes to the documentation
  - [ ] I have made corresponding change to the default configuration files
  - [ ] I have added tests that prove my fix is effective or that my feature works

Author's Checklist

• I have built the docker images using Beats' build system

How to test this PR locally

From metricbeat directory, run mage compose:buildSupportedVersions, the process should finish successfully.

Build Log

$ mage compose:buildSupportedVersions
2020/02/13 17:30:50 Found Elastic Beats dir at /Users/mdelapenya/sourcecode/src/github.com/elastic/beats
>> compose: Building docker images for supported versions
>> compose: Using compose file module/apache/docker-compose.yml
>> compose: Building images for variant map[APACHE_VERSION:2.4.12]
>> compose: Building images for variant map[APACHE_VERSION:2.4.20]
>> compose: OK
>> compose: Using compose file module/envoyproxy/docker-compose.yml
>> compose: Building images for variant map[ENVOYPROXY_VERSION:1.7.0]
>> compose: Building images for variant map[ENVOYPROXY_VERSION:1.12.0]
>> compose: OK
>> compose: Using compose file module/etcd/docker-compose.yml
>> compose: Building images for variant map[ETCD_VERSION:3.2.25]
>> compose: Building images for variant map[ETCD_VERSION:3.3.10]
>> compose: OK
>> compose: Using compose file module/haproxy/docker-compose.yml
>> compose: Building images for variant map[HAPROXY_VERSION:2.0.9]
>> compose: Building images for variant map[HAPROXY_VERSION:1.9.12]
>> compose: Building images for variant map[HAPROXY_VERSION:1.8.22]
>> compose: Building images for variant map[HAPROXY_VERSION:1.7.12]
>> compose: Building images for variant map[HAPROXY_VERSION:1.6.15]
>> compose: OK
>> compose: Using compose file module/kafka/docker-compose.yml
>> compose: Building images for variant map[KAFKA_VERSION:2.1.1]
>> compose: Building images for variant map[KAFKA_VERSION:2.0.0]
>> compose: Building images for variant map[KAFKA_VERSION:1.1.0]
>> compose: Building images for variant map[KAFKA_VERSION:0.10.2.2]
>> compose: OK
>> compose: Using compose file module/mysql/docker-compose.yml
>> compose: Building images for variant map[MYSQL_VARIANT:mysql MYSQL_VERSION:5.7.12]
>> compose: Building images for variant map[MYSQL_VARIANT:mysql MYSQL_VERSION:8.0.13]
>> compose: Building images for variant map[MYSQL_VARIANT:percona MYSQL_VERSION:5.7.24]
>> compose: Building images for variant map[MYSQL_VARIANT:percona MYSQL_VERSION:8.0.13-4]
>> compose: Building images for variant map[MYSQL_VARIANT:mariadb MYSQL_VERSION:10.2.23]
>> compose: Building images for variant map[MYSQL_VARIANT:mariadb MYSQL_VERSION:10.3.14]
>> compose: Building images for variant map[MYSQL_VARIANT:mariadb MYSQL_VERSION:10.4.4]
>> compose: OK
>> compose: Using compose file module/redis/docker-compose.yml
>> compose: Building images for variant map[REDIS_VERSION:3.2.12]
>> compose: Building images for variant map[REDIS_VERSION:4.0.11]
>> compose: Building images for variant map[REDIS_VERSION:5.0.5]
>> compose: OK

Related Issues

• Relates Reuse beats' docker-compose files in the PoC e2e-testing#79

[mdelapenya]

jenkins run the tests please

[jsoriano]

@mdelapenya I would prefer to avoid having to add these artificial variables to the docker compose files in the beats repo.

I have tought in a couple of possible alternatives (only slightly tested):

• We don't need to build images in the PoC, only to pull them, so we don't care what the _meta directory contains. We could create an empty dummy _meta directory in the same directory where the first compose file resides. This way docker compose will read the file and start the services with the pulled images.
• If for any reason previous option doesn't work because we need to build, another option would be to generate an additional compose file, overriding the context paths of the original services, and pass it as an additional compose file to the call to docker-compose. For example if we want to start the redis service, we generate a file like the following one, and pass it additionaly to the other ones:

  version: '2.3'
  
  services:
    redis:
      build:
        context: ${REDIS_PATH_IN_BEATS}/_meta
  

  If needed, the list of services and the version could be easily parsed from the original docker-compose.yml file.

[mdelapenya]

When I firstly thought about this, I tried to escape from massaging the compose (your 2nd choice), but it seems that it's a balanced approach. Therefore, I'll go that way:

• sync the PoC compose files from Beats, and remove the build section in the YML

wdyt? If so, let's close this one

Thanks for your insights!