feat: Provider to load rule sets from cloud blobs

[dadrus]

closes #84
closes #85
closes #282

This PR enables loading of rule sets from cloud blobs. Following new property with following configuration options is available under rules.provider:

cloud_blob:
  watch_interval: 2m
  buckets:
    - url: gs://my-bucket
      prefix: service1
      rules_path_prefix: /service1
    - url: gs://my-bucket
      prefix: service2
      rules_path_prefix: /service2
    - url: s3://my-bucket/my-rule-set

By default the watch_interval is disabled (set to 0), which means the communication to the configured buckets happens only once. The configuration of the buckets within the bucket property supports

• an url (required) to load the blobs from,
• an optional prefix, which indicates that only blobs with a key starting with this prefix should be retrieved, and
• an optional rules_path_prefix, which can be used to create kind of a namespace for the rule sets retrieved from the blobs stored in the corresponding buckets. If set, the provider checks whether the urls specified in all rules retrieved from the referenced bucket have the defined path prefix. If that rule is violated, a warning is emitted and the rule set is ignored. This can be used to ensure a rule retrieved from one endpoint does not override a rule from another endpoint.

This provider supports rule sets in yaml, as well as in json format. The differentiation happens based on the Content-Type set in the blob meta data. If application/yaml or application/json is set, the body of the response is read using a yaml, respectively a json parser, otherwise an error is logged and the response from the endpoint is ignored.

The implementation uses Go CDK - Blob and supports AWS S3, Google Cloud Storage, as well as Azure Blobs. Implementations, which are compatible with the corresponding APIs are supported as well. Examples can be found in the referenced documentation.

The loading and removal of rules happens as follows:

• if the response status code is an HTTP 200 OK and contains a rule set in a known format (see above), the corresponding rules are loaded (if the definitions are valid)
• in case of network issues, like dns errors, timeouts and alike, the rule sets previously received from the corresponding buckets are preserved.
• in any other case related to network communication (like, not 200 status code, empty response body, unsupported format, etc), the corresponding rules are removed if these were previously loaded.

TODOs:

• Implement the actual retrieval or the rule sets (incl tests)
• Implement path prefix validation (incl test)
• Update documentation
• Update configuration validation schema

[codecov]

Codecov Report

Merging #283 (adc0792) into main (850d3bc) will increase coverage by 0.26%.
The diff coverage is 93.09%.

@@            Coverage Diff             @@
##             main     #283      +/-   ##
==========================================
+ Coverage   85.92%   86.18%   +0.26%     
==========================================
  Files         164      172       +8     
  Lines        6863     7161     +298     
==========================================
+ Hits         5897     6172     +275     
- Misses        826      844      +18     
- Partials      140      145       +5     

Impacted Files	Coverage Δ
internal/rules/provider/module.go	0.00% <0.00%> (ø)
...l/rules/provider/cloudblob/mapstructure_decoder.go	75.00% <75.00%> (ø)
...nternal/rules/provider/cloudblob/config_decoder.go	78.57% <78.57%> (ø)
...nal/rules/provider/cloudblob/provider_registrar.go	90.47% <90.47%> (ø)
...ernal/rules/provider/cloudblob/ruleset_endpoint.go	93.75% <93.75%> (ø)
internal/rules/provider/cloudblob/provider.go	95.03% <95.03%> (ø)
internal/rules/provider/filesystem/provider.go	74.54% <100.00%> (ø)
...al/rules/provider/httpendpoint/ruleset_endpoint.go	79.59% <100.00%> (-7.59%)	⬇️
internal/rules/provider/pathprefix/path_prefix.go	100.00% <100.00%> (ø)
...ternal/rules/provider/rulesetparser/json_parser.go	100.00% <100.00%> (ø)
... and 3 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more