Bug: Remote Servers Certificates - Persistence Issue

[Felix83000]

Actual behavior

After adding authentication certificates to one of our remote server and reloading the container the certificates disapears in the folder /var/www/MISP/app/files/certs/

It seems that this function is executed at each container startup core/files/entrypoint_nginx.sh:

update_misp_data_files(){
    for DIR in $(ls /var/www/MISP/app/files.dist); do
        echo "... rsync -azh --delete \"/var/www/MISP/app/files.dist/$DIR\" \"/var/www/MISP/app/files/\""
        rsync -azh --delete "/var/www/MISP/app/files.dist/$DIR" "/var/www/MISP/app/files/"
    done
}

--delete: This option tells rsync to delete files in the destination directory (/var/www/MISP/app/files/) that do not exist in the source directory (/var/www/MISP/app/files.dist/certs).
This result deleting all custom certificates in (/var/www/MISP/app/files/certs).

In the startup logs:

rsync -azh --delete "/var/www/MISP/app/files.dist/certs" "/var/www/MISP/app/files/"

Expected behavior

The Certificates should remain in /var/www/MISP/app/files/certs/ after rebooting the MISP Docker container.

Steps to reproduce

1. Set server/client certificates for a remote server.
2. docker compose down
3. docker compose up
4. Try any sync process, a connection test to a remote server after

You would get a Connection Timeout:

Operating System

Centos

Operating System version

8

[ostefano]

Oh interesting, that is not a use case that was in our mind when designing this image.

What if you use a bind mount for that directory? Have you tried?

[Felix83000]

Yes I already mounted the certs directory within the docker-compose.yml, and the permissions are alrights:

    volumes:
      - "./files/:/var/www/MISP/app/files"
      - "./img/:/var/www/MISP/app/webroot/img/custom"

I also mounted this directory: /var/www/MISP/app/webroot/img/custom to keep the persistence for the custom logos. And for this one, it works like a charm (same permissions than /certs/ folder).

[Felix83000]

Basically, everything in each subfolders of /var/www/MISP/app/files/ will start from scratch at each startup.
Maybe just removing the rsync --delete option will do the thing. my 2 cents

[ostefano]

I am not going to remove that command since it's needed.
My question was whether you tried to map the certs directory directly (aka, adding the following)

    volumes:
      - "./certs/:/var/www/MISP/app/files/certs"

[Felix83000]

I did not ask you to remove that command. I proposed to remove the --delete option. Can you tell why this option is needed to merge the files directories?

As proposed, I tried to mount only the certs directory and the same thing happens.
The certificates are removed in the dirctory because they are not located in the /var/www/MISP/app/files.dist archive (thanks to --delete option). Which make sense because my authentication certificates are not stored in your image archive...

[ostefano]

Because I want to propagate if a new version of MISP deletes some of the directories inside files.
I am thinking of adding an exception specifically for certs to accommodate this use case.

[ostefano]

Oh and now I see the issue. A few months ago the directory was not part of the MISP, now it is an empty directory, that is the issue, and the rsync behavior changed.

@Felix83000 I think adding an exception in the script is the way to go; can you create a PR with the fix?

[Felix83000]

I can propose you something like this then:

update_misp_data_files(){
    for DIR in $(ls /var/www/MISP/app/files.dist); do
        if [ "$DIR" = "certs" ]; then
            echo "... rsync -azh \"/var/www/MISP/app/files.dist/$DIR\" \"/var/www/MISP/app/files/\""
            rsync -azh "/var/www/MISP/app/files.dist/$DIR" "/var/www/MISP/app/files/"
        else
            echo "... rsync -azh --delete \"/var/www/MISP/app/files.dist/$DIR\" \"/var/www/MISP/app/files/\""
            rsync -azh --delete "/var/www/MISP/app/files.dist/$DIR" "/var/www/MISP/app/files/"
        fi
    done
}

I will make the PR as soon as possible. I need to test it first. You will be able to modify it if needed.

[Felix83000]

When do you plan to merge it? Will it be available for 2.4.185?

[ostefano]

Thank you for your PR and I appreciate you editing/deleting your previous messages.
I will test this and merge in time for the next release.