Wandb Config Updates are not synced

[MoH-assan]

Hi,

When I intialized wandb with specific configuration (e.g. config_global in the code snippet below.
Then when I update these configuration and trigger sync, these updates are not reflected in the online run.
See the code and and image below.

Is there is a way around this?

Thanks,
Mohamed

os.environ['WANDB_MODE'] = 'offline'
os.environ['WANDB_DIR'] = wandb_dir
import wandb
config_global={}
config_global['dummy1'] = 0
wandb_run = wandb.init(config=config_global, project='SM_H_S')
wandb.config['dummy2']=0
TriggerWandbSyncHook(communication_dir=wandb_ohs_dir)()

[klieret]

Thanks for your question @MoH-assan.

TriggerWandbSyncHook really only calls wandb sync on the directory that is created by wandb. So my conclusion is that the wandb.config['dummy2']=0 was not written to your_wandb_run/files/config.yml. Could you check that?

Also, could you try wandb_run.config['dummy2'] = 0 instead? (I'm on the go, so I can't check)

[MoH-assan]

@klieret
Thanks for your prompt response.

wandb.config['dummy2']=0 was not written to your_wandb_run/files/config.yml

You are correct, and it seems to be a bit complicated how wandb handle creating and writing to config.yml.
Based on the test below, I think when wandb is offline, it only creates config.yml when the sync is triggered.

Also if sync is triggered before running wandb.finish() a copy of the config file used in wandb.init(config=config) is written to config.yml.

If sync is triggered after running wandb.finish() the most recent version of wandb.config() is written to config.yml.,

So if the process is killed before running wandb.finish(), config.yml will not be updated, which is the issue I am experiencing.

wandb_dir = f"{root_path}"
wandb_ohs_dir = f"{wandb_dir}/wandb/ohs"


import os
import yaml
import wandb


def read_yaml(yaml_path):
    try:
        with open(yaml_path, 'r') as f:
            loaded_yaml = yaml.safe_load(f)
            print(loaded_yaml,flush=True)
    except Exception as e:
            print(e,flush=True)
    print('-------------------',flush=True)
    print('-------------------',flush=True)
    print('-------------------',flush=True)

os.environ['WANDB_MODE'] = 'offline'
os.environ["WANDB_SILENT"] = "true"
os.environ['WANDB_DIR'] = wandb_dir


config_global={}
config_global['dummy1'] = 0
wandb_run = wandb.init(config=config_global, project='SM_H_S')
run_dir = wandb_run.dir
comfig_yaml_path = f'{run_dir}/config.yaml'




wandb.config['dummy2']=0
wandb.config.update({'dummy3': 0})
wandb_run.config['dummy4'] = 0

for counter in range(1000):
    wandb.log({'dummy_counter': counter})
    print(f'counter: {counter}', flush=True)

print('First Attemp to read config.yaml', flush=True)
read_yaml(comfig_yaml_path)


os.system(f'wandb sync {os.path.dirname(run_dir)}')
print('Second Attemp to read config.yaml', flush=True)
read_yaml(comfig_yaml_path)


wandb.finish()
os.system(f'wandb sync {os.path.dirname(run_dir)}')
print('Third attemp to read config.yaml', flush=True)
read_yaml(comfig_yaml_path)

First Attemp to read config.yaml
[Errno 2] No such file or directory: '/gpfs/fs0/scratch/c/cglee/abubakr5/AM/wandb/offline-run-20240103_171425-tszvuffc/files/config.yaml'
-------------------
-------------------
-------------------
Find logs at: /gpfs/fs0/scratch/c/cglee/abubakr5/AM/wandb/debug-cli.abubakr5.log
Syncing: https://wandb.ai/uoft/SM_H_S/runs/tszvuffc ... done.
Second Attemp to read config.yaml
{'wandb_version': 1, 'dummy1': {'desc': None, 'value': 0}, '_wandb': {'desc': None, 'value': {'python_version': '3.9.10', 'cli_version': '0.16.1', 'is_jupyter_run': False, 'is_kaggle_kernel': False, 'start_time': 1704320065.394158, 't': {1: [55], 3: [4, 16, 23, 42], 4: '3.9.10', 5: '0.16.1', 8: [5], 13: 'linux-x86_64'}}}}
-------------------
-------------------
-------------------
Find logs at: /gpfs/fs0/scratch/c/cglee/abubakr5/AM/wandb/debug-cli.abubakr5.log
Syncing: https://wandb.ai/uoft/SM_H_S/runs/tszvuffc ... done.
Third attemp to read config.yaml
{'wandb_version': 1, 'dummy1': {'desc': None, 'value': 0}, '_wandb': {'desc': None, 'value': {'python_version': '3.9.10', 'cli_version': '0.16.1', 'is_jupyter_run': False, 'is_kaggle_kernel': False, 'start_time': 1704320065.394158, 't': {1: [55], 2: [55], 3: [2, 4, 16, 19, 23, 37, 42], 4: '3.9.10', 5: '0.16.1', 8: [5], 13: 'linux-x86_64'}}}, 'dummy2': {'desc': None, 'value': 0}, 'dummy3': {'desc': None, 'value': 0}, 'dummy4': {'desc': None, 'value': 0}}
-------------------
-------------------
-------------------

[klieret]

This is all very interesting! Thank you for looking at this closer. I especially appreciate that you added the code snippets to reproduce your experiments.

[klieret]

I think this might be something to raise to the wandb team, especially because I think that the same applies to the summary metrics.

If you want I can collect a bit more information and raise an issue in the wandb repo. Else I could also support you opening one.

I suspect that this issue is also what causes the wandb pytorch lightning plugin to only synchronize the config at the very end when wandb is run in offline mode (and I've been annoyed by this for a very long time)

[MoH-assan]

@klieret I tried to figure out what method wandb.finish() use to update the config.ymal so I can manually trigger, but no luck.

I think that the same applies to the summary metrics.

I believe so

If you want I can collect a bit more information and raise an issue in the wandb repo.

This would be great, thank you

[MoH-assan]

I think this might be something to raise to the wandb team, especially because I think that the same applies to the summary metrics.

For summary metric, I guess this might be needed

wandb_dir = Path(wandb.run.dir).parent.resolve()
wandb.finish()
time.sleep(5) # might need more time depending on how large are your logs and summaries
trigger_sync(wandb_dir)

[klieret]

Sorry for being not responsive! I'll take another look tomorrow :)

[MoH-assan]

No worries, I tested it I think sleep is necessary before triggering the sync.
Otherwise the hook will command the last sync before the summary are actually stored in the wandb logging directory.
In this case summary will only exist locally.