Support more than one directory for settings management

[BrettMoan]

very similar to pydantic/pydantic#2284

I would love to be able to leverage built-in settings configuration, but in order to do that, I need support for both ConfigMaps and Secrets

i don't know if a file per setting is absolutely necessary, but it would be great if instead of a secrets_dir: str|None you had a directories: List[str]|None or search_dirs: List[str]|None argument, where settings were loaded by iterating over 1..N directories using much the same logic as today.

beyond just allowing use of both ConfigMaps and Secrets, this could then produce conflicts, which instead of a defect, would actually be a very handy a feature, it allows for overrides (such as when running locally) this can be accomplished by having the expectation that the last found match wins.

class Settings(BaseSettings):
    my_secret_data: str
    my_not_secret_data: str
    my_overridden_data: str


    class Config:
        search_dirs = [ '/run/config' ,   '/run/secrets', ]

with files

/run/config/my_not_secret_data # foo
/run/config/my_overridden_data # bar
/run/secrets/my_overridden_data # baz
/run/secrets/my_secret_data # fu

pragmatic expectation:
all 4 files get opened/read/parsed/validated

resulting object is

s = Settings() # s.my_secret_data == 'fu', s.my_not_secret_data == 'foo', s.my_overridden_data == 'baz', 

[samuelcolvin]

Thanks, looks reasonable to me 👍.

[notpushkin]

May I throw in 2¢ – perhaps a more flexible approach would be to expose sources as a Config parameter? E. g. something like:

from pydantic_settings import default_sources, BaseSettings

class Settings(BaseSettings):
    my_secret_data: str
    class Config:
        sources = [
            *default_sources,
            SecretsSettingsSource('/run/config'),
            SecretsSettingsSource('/run/secrets'),
        ]

This would need some changes in how settings sources are initialized (I think we can pass settings_cls at the time Config is defined, but it would look a bit clunky?).

As an upside, it will allow specifying third-party sources more easily:

class Config:
    sources = [*default_sources, MyFeatureFlagsProviderSource(...)]

[stereobutter]

I'd also second that sources should be configurable via Config as having to implement .settings_customise_sources() is very verbose for simple use cases.

[makukha]

Please take a look at the package that I'm going to use in production, it implements this feature:
https://pypi.org/project/pydantic-file-secrets/

[hramezani]

Added in 12d85cf