train script ??? help

[fxnie]

#!/bin/bash

On AWS, the EFA and OFI paths enable NCCL to use optimized networking.

export LD_LIBRARY_PATH=/opt/nccl/build/lib:/opt/amazon/efa/lib:/opt/amazon/openmpi/lib:/opt/aws-ofi-nccl/lib:/usr/local/cuda/lib:/usr/local/cuda/lib64:/usr/local/cuda:/usr/local/cuda/targets/x86_64-linux/lib/:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/lib:$LD_LIBRARY_PATH

export FI_EFA_SET_CUDA_SYNC_MEMOPS=0

torchrun --nnodes=8 --node_rank=0 --nproc_per_node=8
main_training_mamba.py
--tokenizer_path="/workspace/mnt/xxx/models/Bamba-9B"
--data_path="/workspace/mnt/cm-nfx/datasets/case7/no_preprocess"
--datasets="dataset=algorithmic_corpus,dataset=synthetic_code_snippet,dataset=synthetic_qa"
--weights="1,1,1"
--col_name="text"
--file_type="arrow"
--num_workers=12
--seq_length=4096
--vocab_size=128256
--logical_shards=960
--ckpt_load_path="/workspace/mnt/xxx/models/Bamba-9B"
--ckpt_save_path="/workspace/mnt/xxx/ckpt/bamba-fms"
--sharding_strategy="fsdp"
--batch_size=2
--learning_rate=3e-4
--num_steps=1000
--report_interval=10
--checkpoint_interval=10
--strip_tokens: str = ""
--seed=2023
--bos_token=None
--bol_token=None
--eol_token=None
--eos_token=0 \

I am using this script for training, but it has been stuck. What is the reason? Thank you

W0109 16:19:56.623000 139977270769472 torch/distributed/run.py:779]
W0109 16:19:56.623000 139977270769472 torch/distributed/run.py:779] *****************************************
W0109 16:19:56.623000 139977270769472 torch/distributed/run.py:779] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
W0109 16:19:56.623000 139977270769472 torch/distributed/run.py:779] *****************************************
Traceback (most recent call last):
File "/opt/conda/envs/bamba/bin/torchrun", line 8, in
sys.exit(main())
File "/opt/conda/envs/bamba/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 348, in wrapper
return f(*args, **kwargs)
File "/opt/conda/envs/bamba/lib/python3.10/site-packages/torch/distributed/run.py", line 901, in main
run(args)
File "/opt/conda/envs/bamba/lib/python3.10/site-packages/torch/distributed/run.py", line 892, in run
elastic_launch(
File "/opt/conda/envs/bamba/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 133, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/opt/conda/envs/bamba/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 255, in launch_agent
result = agent.run()
File "/opt/conda/envs/bamba/lib/python3.10/site-packages/torch/distributed/elastic/metrics/api.py", line 124, in wrapper
result = f(*args, **kwargs)
File "/opt/conda/envs/bamba/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 680, in run
result = self._invoke_run(role)
File "/opt/conda/envs/bamba/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 829, in _invoke_run
self._initialize_workers(self._worker_group)
File "/opt/conda/envs/bamba/lib/python3.10/site-packages/torch/distributed/elastic/metrics/api.py", line 124, in wrapper
result = f(*args, **kwargs)
File "/opt/conda/envs/bamba/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 652, in _initialize_workers
self._rendezvous(worker_group)
File "/opt/conda/envs/bamba/lib/python3.10/site-packages/torch/distributed/elastic/metrics/api.py", line 124, in wrapper
result = f(*args, **kwargs)
File "/opt/conda/envs/bamba/lib/python3.10/site-packages/torch/distributed/elastic/agent/server/api.py", line 489, in _rendezvous
rdzv_info = spec.rdzv_handler.next_rendezvous()
File "/opt/conda/envs/bamba/lib/python3.10/site-packages/torch/distributed/elastic/rendezvous/static_tcp_rendezvous.py", line 66, in next_rendezvous
self._store = TCPStore( # type: ignore[call-arg]
torch.distributed.DistStoreError: Timed out after 901 seconds waiting for clients. 1/8 clients joined.

[lchu6]

It seems the error is related to the torchrun command on dist init. so this hasn't got into the application code yet.

Are you running 64 gpus?

[fxnie]

@lchu6 No, it's written in the script that 8 GPUs were used。May I ask if the dataset needs any additional processing? Can datasets in Parquet format be processed directly?
It would be even better if we could provide a faster way to contact you. thank you.

[fxnie]

@lchu6 The above problem has been solved, but I have encountered a new one:
How should the loading method of the dataset be set?bFloat16 enabled for mixed precision - using bfSixteen policy
Sharding strategy = fsdp

--> model has 9780.235392 Million params

Constructing datasets...
Datasets constructed!
--> enabling torch compile...
No valid checkpoint detected at /workspace/mnt/xxx/models/Bamba-9B/checkpoints/, starting from scratch.
Training for 1000 steps
[rank4]: Traceback (most recent call last):
[rank4]: File "/workspace/mnt/xxx/fms-fsdp/main_training_mamba.py", line 177, in
[rank4]: fire.Fire(main)
[rank4]: File "/opt/conda/envs/bamba/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
[rank4]: component_trace = _Fire(component, args, parsed_flag_args, context, name)
[rank4]: File "/opt/conda/envs/bamba/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
[rank4]: component, remaining_args = _CallAndUpdateTrace(
[rank4]: File "/opt/conda/envs/bamba/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
[rank4]: component = fn(*varargs, **kwargs)
[rank4]: File "/workspace/mnt/xxx/fms-fsdp/main_training_mamba.py", line 156, in main
[rank4]: train(
[rank4]: File "/workspace/mnt/xxx/fms-fsdp/fms_fsdp/utils/train_utils.py", line 81, in train
[rank4]: for batch_idx, (input, label) in enumerate(train_loader, start=start_step + 1):
[rank4]: File "/opt/conda/envs/bamba/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 630, in next
[rank4]: data = self._next_data()
[rank4]: File "/opt/conda/envs/bamba/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1344, in _next_data
[rank4]: return self._process_data(data)
[rank4]: File "/opt/conda/envs/bamba/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1370, in _process_data
[rank4]: data.reraise()
[rank4]: File "/opt/conda/envs/bamba/lib/python3.10/site-packages/torch/_utils.py", line 706, in reraise
[rank4]: raise exception
[rank4]: AssertionError: Caught AssertionError in DataLoader worker process 0.
[rank4]: Original Traceback (most recent call last):
[rank4]: File "/opt/conda/envs/bamba/lib/python3.10/site-packages/torch/utils/data/_utils/worker.py", line 309, in _worker_loop
[rank4]: data = fetcher.fetch(index) # type: ignore[possibly-undefined]
[rank4]: File "/opt/conda/envs/bamba/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 33, in fetch
[rank4]: data.append(next(self.dataset_iter))
[rank4]: File "/workspace/mnt/xxx/fms-fsdp/fms_fsdp/utils/dataset_utils.py", line 542, in iter
[rank4]: yield next(dataset)
[rank4]: File "/workspace/mnt/xxx/fms-fsdp/fms_fsdp/utils/dataset_utils.py", line 487, in iter
[rank4]: out = next(dataset)
[rank4]: File "/workspace/mnt/xxxfms-fsdp/fms_fsdp/utils/dataset_utils.py", line 487, in iter
[rank4]: out = next(dataset)
[rank4]: File "/workspace/mnt/xxx/fms-fsdp/fms_fsdp/utils/dataset_utils.py", line 660, in iter
[rank4]: self.buffer[self.buffer_size] = next(dataset)
[rank4]: File "/workspace/mnt/xxx/fms-fsdp/fms_fsdp/utils/dataset_utils.py", line 792, in iter
[rank4]: out, buffer = self._get_buffer(dataset, self.len, self.buffer)
[rank4]: File "/workspace/mnt/xxx/fms-fsdp/fms_fsdp/utils/dataset_utils.py", line 753, in _get_buffer
[rank4]: new = next(iterable)
[rank4]: File "/workspace/mnt/xxx/fms-fsdp/fms_fsdp/utils/dataset_utils.py", line 1373, in iter
[rank4]: out = next(data[self.current_iterator])
[rank4]: File "/workspace/mnt/xxx/fms-fsdp/fms_fsdp/utils/dataset_utils.py", line 1241, in iter
[rank4]: sum(self.n_docs_remaining) > 0
[rank4]: AssertionError: No documents detected in /workspace/mnt/cm-nfx/datasets/case7/preprocess/algorithmic_corpus/dataset=data-00002-of-00033.parquet

[lchu6]

@fxnie grad you solved the early problem.

Now, this issue is related to how to prepare the dataset to be used with our dataloader.

adding @daviswer here to help check further.

[daviswer]

Thanks @lchu6! @fxnie, detailed directions on using the dataloader can be found here - see in particular the end section on using your own data and different file types. Note that the config args related to datasets should be specified in addition to the ones related to training in the general training example. It looks like you're already doing this since your job got to the point of opening particular data files, but including this anyways since in your initial post you have --file_type arrow despite using parquet files.

In this case it looks like we missed cleaning up a bit of code with file path handling specific to how we did our data preprocessing. I'll hunt that down and see about getting it removed.

[daviswer]

Or not? @fxnie can you post the --datapath and --datasets values you used to produce the above? I'm not seeing how the ones in your initial post could be producing the file path described in your error.

Do note that when specifying subdatasets, you only need to provide the subfolder names. The reason we have dataset=cc_en_head etc is because our subfolders were literally named that i.e. [datapath]/dataset=cc_en_head/[file].arrow. I suspect you may be running into issues because your subfolders don't have this prefix. Try --datasets="algorithmic_corpus,synthetic_code_snippet,synthetic_qa"

[daviswer]

Regardless, I've added a section on the expected directory layout towards the end of the dataloader readme that hopefully makes some of this clearer

[fxnie]

@daviswer Thank you very much for your answer, which solved the problem of data loading.

But there is a checkpoint error here. My checkpoint directory is shown in the picture. But why is there still an invalid checkpoint displayed? My training steps and error messages are as follows

script：

#!/bin/bash

export LD_LIBRARY_PATH=/opt/nccl/build/lib:/opt/amazon/efa/lib:/opt/amazon/openmpi/lib:/opt/aws-ofi-nccl/lib:/usr/local/cuda/lib:/usr/local/cuda/lib64:/usr/local/cuda:/usr/local/cuda/targets/x86_64-linux/lib/:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/lib:$LD_LIBRARY_PATH

export FI_EFA_SET_CUDA_SYNC_MEMOPS=0

torchrun --nnodes=1 --node_rank=0 --nproc_per_node=8
main_training_mamba.py
--model_variant=mamba_9.8b
--tokenizer_path="/workspace/mnt/xxx/models/Bamba-9B/checkpoints"
--data_path="/workspace/mnt/cm-nfx/datasets/case7/preprocess"
--datasets="algorithmic_corpus,synthetic_code_snippet,synthetic_qa"
--weights="1,1,1"
--col_name="text"
--file_type="hf_parquet"
--num_workers=12
--seq_length=4096
--vocab_size=128256
--logical_shards=960
--ckpt_load_path="/workspace/mnt/xxx/models/Bamba-9B/checkpoints"
--ckpt_save_path="/workspace/mnt/xxx/ckpt/bamba-fms"
--sharding_strategy="fsdp"
--batch_size=2
--learning_rate=3e-4
--num_steps=1000
--report_interval=10
--checkpoint_interval=10
--strip_tokens: str = ""
--seed=2023
--bos_token=None
--bol_token=None
--eol_token=None
--eos_token=0 \

error:
...
...
Sharding strategy = fsdp

--> model has 9780.235392 Million params

Constructing datasets...
Datasets constructed!
--> enabling torch compile...
No valid checkpoint detected at /workspace/mnt/xxx/models/Bamba-9B/checkpoints/checkpoints/, starting from scratch.
Training for 1000 steps
Dataset: No valid checkpoint detected at /workspace/mnt/xxx/ckpt/bamba-fms/checkpoints, dataset starting from scratch.
/workspace/mnt/xxx/fms-fsdp/fms_fsdp/utils/dataloader_utils.py:29: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
data_seq = torch.tensor(data_seq, dtype=torch.int)
/workspace/mnt/xxx/fms-fsdp/fms_fsdp/utils/dataloader_utils.py:29: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
data_seq = torch.tensor(data_seq, dtype=torch.int)
/workspace/mnt/xxx/fms-fsdp/fms_fsdp/utils/dataloader_utils.py:29: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
data_seq = torch.tensor(data_seq, dtype=torch.int)
/workspace/mnt/xxx/fms-fsdp/fms_fsdp/utils/dataloader_utils.py:29: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
data_seq = torch.tensor(data_seq, dtype=torch.int)
/workspace/mnt/xxx/fms-fsdp/fms_fsdp/utils/dataloader_utils.py:29: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
data_seq = torch.tensor(data_seq, dtype=torch.int)
/workspace/mnt/xxx/fms-fsdp/fms_fsdp/utils/dataloader_utils.py:29: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
data_seq = torch.tensor(data_seq, dtype=torch.int)
/workspace/mnt/xxx/fms-fsdp/fms_fsdp/utils/dataloader_utils.py:29: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
data_seq = torch.tensor(data_seq, dtype=torch.int)
/workspace/mnt/xxx/fms-fsdp/fms_fsdp/utils/dataloader_utils.py:29: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
data_seq = torch.tensor(data_seq, dtype=torch.int)
/workspace/mnt/xxx/fms-fsdp/fms_fsdp/utils/dataloader_utils.py:29: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
data_seq = torch.tensor(data_seq, dtype=torch.int)
/workspace/mnt/xxx/fms-fsdp/fms_fsdp/utils/dataloader_utils.py:29: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
data_seq = torch.tensor(data_seq, dtype=torch.int)
/workspace/mnt/xxx/fms-fsdp/fms_fsdp/utils/dataloader_utils.py:29: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
data_seq = torch.tensor(data_seq, dtype=torch.int)
/workspace/mnt/xxx/fms-fsdp/fms_fsdp/utils/dataloader_utils.py:29: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
data_seq = torch.tensor(data_seq, dtype=torch.int)
/workspace/mnt/xxxfms-fsdp/fms_fsdp/utils/dataloader_utils.py:29: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
data_seq = torch.tensor(data_seq, dtype=torch.int)
/workspace/mnt/xxx/fms-fsdp/fms_fsdp/utils/dataloader_utils.py:29: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
data_seq = torch.tensor(data_seq, dtype=torch.int)
/workspace/mnt/xxx/fms-fsdp/fms_fsdp/utils/dataloader_utils.py:29: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
data_seq = torch.tensor(data_seq, dtype=torch.int)
/workspace/mnt/xxx/fms-fsdp/fms_fsdp/utils/dataloader_utils.py:29: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
data_seq = torch.tensor(data_seq, dtype=torch.int)
/workspace/mnt/xxx/fms-fsdp/fms_fsdp/utils/dataloader_utils.py:29: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
data_seq = torch.tensor(data_seq, dtype=torch.int)
/workspace/mnt/xxx/fms-fsdp/fms_fsdp/utils/dataloader_utils.py:29: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
data_seq = torch.tensor(data_seq, dtype=torch.int)
[rank0]:W0114 10:38:46.343000 139781756962624 torch/_logging/internal.py:1034] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
[rank5]:W0114 10:38:46.345000 139797034338112 torch/logging/internal.py:1034] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
/workspace/mnt/xxx/fms-fsdp/fms_fsdp/utils/dataloader_utils.py:29: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad(True), rather than torch.tensor(sourceTensor).
data_seq = torch.tensor(data_seq, dtype=torch.int)
/workspace/mnt/xxx/fms-fsdp/fms_fsdp/utils/dataloader_utils.py:29: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad(True), rather than torch.tensor(sourceTensor).
data_seq = torch.tensor(data_seq, dtype=torch.int)
/workspace/mnt/xxx/fms-fsdp/fms_fsdp/utils/dataloader_utils.py:29: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad(True), rather than torch.tensor(sourceTensor).
data_seq = torch.tensor(data_seq, dtype=torch.int)
[rank1]:W0114 10:38:48.718000 139778707380032 torch/_logging/_internal.py:1034] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
.........etc

No valid checkpoint detected at /workspace/mnt/xxx/models/Bamba-9B/checkpoints/checkpoints/, starting from scratch.
Dataset: No valid checkpoint detected at /workspace/mnt/xxx/ckpt/bamba-fms/checkpoints, dataset starting from scratch.

My checkpoint is already in this directory, why can't find it? And why does dataset search for cpkt_stave_path directory? I am very confused

[daviswer]

Oh the reason this is coming up is because we released the model in HF format, but we didn't train it in HF format, we used custom PyTorch model code. So it's looking for a pure PyTorch checkpoint and because only the HF format is in your path, it registers as not finding anything. Same deal for the dataloader. The dataloader is stateful and handles its own checkpoint saving/loading, so it also looks in the checkpoint directory for a dataloader checkpoint. And because the HF format doesn't include a dataloader checkpoint, it also doesn't find anything.

Let me see about accessing the training checkpoints in the original format.

[lchu6]

you could access the original ckpt (in dcp format) as instructed in https://huggingface.co/ibm-fms/Bamba-9B.

e.g.

aws --endpoint-url https://s3.us-east.cloud-object-storage.appdomain.cloud s3 ls s3://bamba-public/checkpoints/pretraining/phase_two/2_2t/step_140000_ckp/

[fxnie]

@lchu6
1、I am a user from China and need a foreign credit card to register. Is there any other way to obtain the checkpoint? esteem it a favor.
2、Is there a way to convert HF format to PyTorch format to weight conversion?

[fxnie]

@daviswer

[fxnie]

@daviswer @lchu6 May I ask where I can obtain the file 'generation.config. json'?

[fxnie]

@daviswer @lchu6 help