in_calyptia_fleet: support multiple files for configurations.

[pwhelan]

Summary

Add support for extra fleet files. These go along with the main configuration and can be referred to using relative paths. The intention is to use them for:

• Parsers
• Includes
• Lua Scripts

But they are not restricted to just those uses. They can also be used as general files for plugins.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build.
• Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[edsiper]

fix conflict pls

[pwhelan]

There is also #7969 that is relevant to fleet, especially for windows.

[pwhelan]

fix conflict pls

The branch has been rebased again off of master and all conflicts fixed.

[pwhelan]

@edsiper all tests besides the log timing test have passed.

[pwhelan]

There is still an out of scope memory leak but it seems to be a single allocation over a long time frame:

[0] dummy: [[1699309358.993836448, {}], {"powered_by"=>"calyptia", "message"=>"unscripted #10"}]
[0] dummy: [[1699309359.993809536, {}], {"powered_by"=>"calyptia", "message"=>"unscripted #10"}]
[0] dummy: [[1699309360.993855547, {}], {"powered_by"=>"calyptia", "message"=>"unscripted #10"}]
[0] dummy: [[1699309361.993810751, {}], {"powered_by"=>"calyptia", "message"=>"unscripted #10"}]
[0] dummy: [[1699309363.003043522, {}], {"powered_by"=>"calyptia", "message"=>"unscripted #10"}]
[0] dummy: [[1699309363.993884022, {}], {"powered_by"=>"calyptia", "message"=>"unscripted #10"}]
[0] dummy: [[1699309364.993818176, {}], {"powered_by"=>"calyptia", "message"=>"unscripted #10"}]
[0] dummy: [[1699309365.993820324, {}], {"powered_by"=>"calyptia", "message"=>"unscripted #10"}]
[0] dummy: [[1699309366.993877346, {}], {"powered_by"=>"calyptia", "message"=>"unscripted #10"}]
[0] dummy: [[1699309367.993850950, {}], {"powered_by"=>"calyptia", "message"=>"unscripted #10"}]
[0] dummy: [[1699309368.993814938, {}], {"powered_by"=>"calyptia", "message"=>"unscripted #10"}]
[0] dummy: [[1699309369.993866200, {}], {"powered_by"=>"calyptia", "message"=>"unscripted #10"}]
^C[2023/11/06 19:22:51] [engine] caught signal (SIGINT)
[2023/11/06 19:22:51] [ warn] [engine] service will shutdown in max 5 seconds
[0] dummy: [[1699309370.993816413, {}], {"powered_by"=>"calyptia", "message"=>"unscripted #10"}]
[2023/11/06 19:22:51] [ info] [input] pausing dummy.0
[2023/11/06 19:22:51] [ info] [input] pausing fluentbit_metrics.1
[2023/11/06 19:22:51] [ info] [input] pausing calyptia_fleet.2
[2023/11/06 19:22:51] [ info] [engine] service has stopped (0 pending tasks)
[2023/11/06 19:22:51] [ info] [input] pausing dummy.0
[2023/11/06 19:22:51] [ info] [input] pausing fluentbit_metrics.1
[2023/11/06 19:22:51] [ info] [input] pausing calyptia_fleet.2
[2023/11/06 19:22:51] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/11/06 19:22:51] [ info] [output:stdout:stdout.0] thread worker #0 stopped

=================================================================
==349931==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 27 byte(s) in 1 object(s) allocated from:
    #0 0x7ffff78e1359 in __interceptor_malloc /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0x55555575acff in flb_malloc /home/pwhelan/Projects/work/fluent-bit.git/pwhelan-fleet-header/include/fluent-bit/flb_mem.h:80
    #2 0x55555575b2e6 in sds_alloc /home/pwhelan/Projects/work/fluent-bit.git/pwhelan-fleet-header/src/flb_sds.c:41
    #3 0x55555575b4b4 in flb_sds_create_len /home/pwhelan/Projects/work/fluent-bit.git/pwhelan-fleet-header/src/flb_sds.c:62
    #4 0x55555575b5fe in flb_sds_create /home/pwhelan/Projects/work/fluent-bit.git/pwhelan-fleet-header/src/flb_sds.c:88
    #5 0x55555571871e in flb_service_conf_path_set /home/pwhelan/Projects/work/fluent-bit.git/pwhelan-fleet-header/src/fluent-bit.c:736
    #6 0x5555557187b9 in service_configure /home/pwhelan/Projects/work/fluent-bit.git/pwhelan-fleet-header/src/fluent-bit.c:762
    #7 0x55555571ab29 in flb_main /home/pwhelan/Projects/work/fluent-bit.git/pwhelan-fleet-header/src/fluent-bit.c:1298
    #8 0x55555571b20a in main /home/pwhelan/Projects/work/fluent-bit.git/pwhelan-fleet-header/src/fluent-bit.c:1456
    #9 0x7ffff6cbaccf  (/usr/lib/libc.so.6+0x27ccf) (BuildId: 8bfe03f6bf9b6a6e2591babd0bbc266837d8f658)

SUMMARY: AddressSanitizer: 27 byte(s) leaked in 1 allocation(s)

That is the end of a run of about 4 hours with several configuration updates.

[edsiper]

@pwhelan pls fix the conflict; re the leak maybe its related to config->conf_path_file ?