Settings Merging from files doesn't allow for partial overrides

[rfenner]

I wrote a class to simulate the .dotenv loading of toml files where it generated the list of toml files to load in.
The problem I discovered is that the file merging is shallow ie if you have two files that both specify the same header key if you stry to override just one setting in the header from the previous file's version it wipes everything else out.

example

main.toml

[db]
host="some_host"
user="some_user"
pss="pass"
port=3306
db="soem_db

override.toml

[db]
db="some_other_db"

The data that gets loaded and then validated is

[db]
db="some_other_db"

instead of

[db]
host="some_host"
user="some_user"
pss="pass"
port=3306
db="soem_other_db

This is because the _read_files in ConfigFileSourceMixin uses the update methods of the dict which just replaces the value if the key already exists.

It would be nice to add a function that could recessively merge the dicts together that may one could do partial overrides. Thi affects all the other loaders like yaml and json as well.

[hramezani]

Thanks @rfenner for reporting this.

This is because the _read_files in ConfigFileSourceMixin uses the update methods of the dict which just replaces the value if the key already exists.

Yes

It would be nice to add a function that could recessively merge the dicts together that may one could do partial overrides. Thi affects all the other loaders like yaml and json as well.

Agree, we can add a flag for this.

Also, you can create a custom toml settings source like bellow:

from typing import Any
from pathlib import Path

from pydantic._internal._utils import deep_update
from pydantic_settings import (
    BaseSettings,
    PydanticBaseSettingsSource,
    SettingsConfigDict,
    TomlConfigSettingsSource,
)
from pydantic_settings.sources import PathType

class MyTomlConfigSettingsSource(TomlConfigSettingsSource):
    def _read_files(self, files: PathType | None) -> dict[str, Any]:
        vars: dict[str, Any] = {}
        for file in files:
            file_path = Path(file).expanduser()
            if file_path.is_file():
                vars = deep_update(vars, self._read_file(file_path))
        return vars

class Settings(BaseSettings):
    db: dict[str, Any]
    model_config = SettingsConfigDict(toml_file=('main.toml', 'override.toml'))

    @classmethod
    def settings_customise_sources(
        cls,
        settings_cls: type[BaseSettings],
        init_settings: PydanticBaseSettingsSource,
        env_settings: PydanticBaseSettingsSource,
        dotenv_settings: PydanticBaseSettingsSource,
        file_secret_settings: PydanticBaseSettingsSource,
    ) -> tuple[PydanticBaseSettingsSource, ...]:
        return (MyTomlConfigSettingsSource(settings_cls),)

print(Settings())

[rfenner]

Thanks for pointing out how to do it. I was able to implement the merging from your suggestion. I had another idea on how to do it but one that would have loaded it outside and provide the data to the settings class and you solution works better

[espdev]

I would also note that it would be nice to be able to merge data from config files and init_settings. Right now the data from files overwrites any collections in init_settings.

For example:

class Settings(BaseSettings):
    ...  # add yaml config file source

    data: dict[str, int] = {
        'foo': 10,
        'bar': 20,
    }

# settings.yaml config file

data:
    bar: 30

And finally we get:

>>> s = Settings()
>>> print(s)
Settings(data={'bar': 30})

But I'd like to get:

Settings(data={'foo': 10, 'bar': 30})

It would be nice to be able to customize merge strategies for different collections such as lists and dictionaries.