training and validation Dice scores always 0

[weixiaoxkq]

I am training an auxiliary labeling tool using a public open-fracture dataset and a Segmentation model. The training process runs normally and the loss decreasing, but both the training and validation Dice scores always 0 throughout the process.

I have uploaded my modified segmentation.py file, and the training logs.

My training dataset contains four label classes (excluding background), all saved in NIfTI (.nii.gz) format. I have already confirmed that the labels and images have exactly the same spacing, orientation, and other metadata.

Could someone help me check what might be causing this issue?

segmentation.py file

class Segmentation(TaskConfig):
    def init(self, name: str, model_dir: str, conf: Dict[str, str], planner: Any, **kwargs):
        super().init(name, model_dir, conf, planner, **kwargs)

        # Labels
        self.labels = {
            "buckle fracture": 1,
            "nondisplaced fracture": 2,
            "displaced fracture": 3,
            "segmental fracture": 4
        }
       # self.labels = {
        #    "fracture":1,
        #}

        # Model Files
        self.path = [
            os.path.join(self.model_dir, f"pretrained_{name}.pt"),  # pretrained
            os.path.join(self.model_dir, f"{name}.pt"),  # published
        ]

        # Download PreTrained Model
        if strtobool(self.conf.get("use_pretrained_model", "false")):
            url = f"{self.conf.get('pretrained_path', self.PRE_TRAINED_PATH)}"
            url = f"{url}/radiology_segmentation_segresnet_multilabel.pt"
            download_file(url, self.path[0])

        self.target_spacing = (1.5, 1.5, 1.5)  # target space for image
        # Setting ROI size - This is for the image padding
        self.roi_size = (96, 96, 96)

        # Network
        self.network = SegResNet(
            spatial_dims=3,
            in_channels=1,
            out_channels=len(self.labels) + 1,  # labels plus background,
            init_filters=32,
            blocks_down=(1, 2, 2, 4),
            blocks_up=(1, 1, 1),
            dropout_prob=0.2,
        )

    def infer(self) -> Union[InferTask, Dict[str, InferTask]]:
        task: InferTask = lib.infers.Segmentation(
            path=self.path,
            network=self.network,
            roi_size=self.roi_size,
            target_spacing=self.target_spacing,
            labels=self.labels,
            preload=strtobool(self.conf.get("preload", "false")),
            config={"largest_cc": True if has_cp and has_cucim else False},
        )
        return task

    def trainer(self) -> Optional[TrainTask]:
        output_dir = os.path.join(self.model_dir, self.name)
        load_path = self.path[0] if os.path.exists(self.path[0]) else self.path[1]

        task: TrainTask = lib.trainers.Segmentation(
            model_dir=output_dir,
            network=self.network,
            roi_size=self.roi_size,
            target_spacing=self.target_spacing,
            load_path=load_path,
            publish_path=self.path[1],
            description="Train Segmentation Model",
            labels=self.labels,
        )
        return task

train log

[2025-11-16 14:31:50,730] [6492] [MainThread] [INFO] (__main__:285) - USING:: version = False
[2025-11-16 14:31:50,730] [6492] [MainThread] [INFO] (__main__:285) - USING:: app = /home/ys2/apps/radiology
[2025-11-16 14:31:50,730] [6492] [MainThread] [INFO] (__main__:285) - USING:: studies = /home/ys2/monailabel-ribfra-data/imagerTrTest
[2025-11-16 14:31:50,730] [6492] [MainThread] [INFO] (__main__:285) - USING:: verbose = INFO
[2025-11-16 14:31:50,730] [6492] [MainThread] [INFO] (__main__:285) - USING:: conf = [['models', 'segmentation']]
[2025-11-16 14:31:50,730] [6492] [MainThread] [INFO] (__main__:285) - USING:: host = 0.0.0.0
[2025-11-16 14:31:50,730] [6492] [MainThread] [INFO] (__main__:285) - USING:: port = 8000
[2025-11-16 14:31:50,730] [6492] [MainThread] [INFO] (__main__:285) - USING:: uvicorn_app = monailabel.app:app
[2025-11-16 14:31:50,730] [6492] [MainThread] [INFO] (__main__:285) - USING:: ssl_keyfile = None
[2025-11-16 14:31:50,730] [6492] [MainThread] [INFO] (__main__:285) - USING:: ssl_certfile = None
[2025-11-16 14:31:50,730] [6492] [MainThread] [INFO] (__main__:285) - USING:: ssl_keyfile_password = None
[2025-11-16 14:31:50,730] [6492] [MainThread] [INFO] (__main__:285) - USING:: ssl_ca_certs = None
[2025-11-16 14:31:50,730] [6492] [MainThread] [INFO] (__main__:285) - USING:: workers = None
[2025-11-16 14:31:50,730] [6492] [MainThread] [INFO] (__main__:285) - USING:: limit_concurrency = None
[2025-11-16 14:31:50,730] [6492] [MainThread] [INFO] (__main__:285) - USING:: access_log = False
[2025-11-16 14:31:50,730] [6492] [MainThread] [INFO] (__main__:285) - USING:: root_path =
[2025-11-16 14:31:50,730] [6492] [MainThread] [INFO] (__main__:285) - USING:: log_level = info
[2025-11-16 14:31:50,730] [6492] [MainThread] [INFO] (__main__:285) - USING:: log_config = None
[2025-11-16 14:31:50,730] [6492] [MainThread] [INFO] (__main__:285) - USING:: dryrun = False
[2025-11-16 14:31:50,730] [6492] [MainThread] [INFO] (__main__:285) - USING:: action = start_server
[2025-11-16 14:31:50,731] [6492] [MainThread] [INFO] (__main__:296) -
Allow Origins: ['*']
[2025-11-16 14:31:51,524] [6492] [MainThread] [INFO] (uvicorn.error:82) - Started server process [6492]
[2025-11-16 14:31:51,524] [6492] [MainThread] [INFO] (uvicorn.error:48) - Waiting for application startup.
App Init...
[2025-11-16 14:31:51,524] [6492] [MainThread] [INFO] (monailabel.interfaces.utils.app:37) - Initializing App from: /home/ys2/apps/radiology; studies: /home/ys2/monailabel-ribfra-data/imagerTrTest; conf: {'models': 'segmentation'}
[2025-11-16 14:31:51,607] [6492] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for MONAILabelApp Found: <class 'main.MyApp'>
[2025-11-16 14:31:51,612] [6492] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.localization_spine.LocalizationSpine'>
[2025-11-16 14:31:51,612] [6492] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_3d.Deepgrow3D'>
[2025-11-16 14:31:51,614] [6492] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_spleen.SegmentationSpleen'>
[2025-11-16 14:31:51,614] [6492] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_vertebra.SegmentationVertebra'>
[2025-11-16 14:31:51,614] [6492] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation.Segmentation'>
[2025-11-16 14:31:51,615] [6492] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepedit.DeepEdit'>
[2025-11-16 14:31:51,615] [6492] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_2d.Deepgrow2D'>
[2025-11-16 14:31:51,615] [6492] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation.Segmentation'>
[2025-11-16 14:31:51,616] [6492] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.localization_vertebra.LocalizationVertebra'>
[2025-11-16 14:31:51,616] [6492] [MainThread] [INFO] (main:95) - +++ Adding Model: segmentation => lib.configs.segmentation.Segmentation
[2025-11-16 14:31:51,749] [6492] [MainThread] [INFO] (main:98) - +++ Using Models: ['segmentation']
[2025-11-16 14:31:51,750] [6492] [MainThread] [INFO] (monailabel.interfaces.app:135) - Init Datastore for: /home/ys2/monailabel-ribfra-data/imagerTrTest
[2025-11-16 14:31:51,750] [6492] [MainThread] [INFO] (monailabel.datastore.local:130) - Auto Reload: True; Extensions: ['*.nii.gz', '*.nii', '*.nrrd', '*.jpg', '*.png', '*.tif', '*.svs', '*.xml']
[2025-11-16 14:31:51,752] [6492] [MainThread] [INFO] (monailabel.datastore.local:577) - Invalidate count: 0
[2025-11-16 14:31:51,752] [6492] [MainThread] [INFO] (monailabel.datastore.local:151) - Start observing external modifications on datastore (AUTO RELOAD)
[2025-11-16 14:31:51,756] [6492] [MainThread] [INFO] (main:128) - +++ Adding Inferer:: segmentation => <lib.infers.segmentation.Segmentation object at 0x7f3be603aa60>
[2025-11-16 14:31:51,756] [6492] [MainThread] [INFO] (main:194) - {'segmentation': <lib.infers.segmentation.Segmentation object at 0x7f3be603aa60>, 'Histogram+GraphCut': <monailabel.scribbles.infer.HistogramBasedGraphCut object at 0x7f3bdb4320d0>, 'GMM+GraphCut': <monailabel.scribbles.infer.GMMBasedGraphCut object at 0x7f3bd9e45400>}
[2025-11-16 14:31:51,756] [6492] [MainThread] [INFO] (main:209) - +++ Adding Trainer:: segmentation => <lib.trainers.segmentation.Segmentation object at 0x7f3bd9e45340>
[2025-11-16 14:31:51,756] [6492] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: /home/ys2/.cache/monailabel/sessions
[2025-11-16 14:31:51,756] [6492] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2025-11-16 14:31:51,756] [6492] [MainThread] [INFO] (monailabel.interfaces.app:469) - App Init - completed
[2025-11-16 14:31:51,757] [timeloop] [INFO] Starting Timeloop..
[2025-11-16 14:31:51,757] [6492] [MainThread] [INFO] (timeloop:60) - Starting Timeloop..
[2025-11-16 14:31:51,757] [timeloop] [INFO] Registered job <function MONAILabelApp.on_init_complete.<locals>.run_scheduler at 0x7f3bde0a6ca0>
[2025-11-16 14:31:51,757] [6492] [MainThread] [INFO] (timeloop:42) - Registered job <function MONAILabelApp.on_init_complete.<locals>.run_scheduler at 0x7f3bde0a6ca0>
[2025-11-16 14:31:51,757] [timeloop] [INFO] Timeloop now started. Jobs will run based on the interval set
[2025-11-16 14:31:51,757] [6492] [MainThread] [INFO] (timeloop:63) - Timeloop now started. Jobs will run based on the interval set
[2025-11-16 14:31:51,757] [6492] [MainThread] [INFO] (uvicorn.error:62) - Application startup complete.
[2025-11-16 14:31:51,757] [6492] [MainThread] [INFO] (uvicorn.error:214) - Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
[2025-11-16 14:31:54,361] [6492] [MainThread] [INFO] (monailabel.utils.async_tasks.task:41) - Train request: {'model': 'segmentation', 'name': 'train_01', 'pretrained': True, 'device': 'NVIDIA GeForce RTX 2080 Ti:0', 'max_epochs': 5, 'early_stop_patience': -1, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1, 'multi_gpu': True, 'gpus': 'all', 'dataset': 'Dataset', 'dataloader': 'ThreadDataLoader', 'tracking': 'mlflow', 'tracking_uri': '', 'tracking_experiment_name': '', 'client_id': 'user-xyz'}
[2025-11-16 14:31:54,362] [6492] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:49) - Before:: /home/ys2/anaconda3/envs::/home/ys2/apps/radiology
[2025-11-16 14:31:54,363] [6492] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:53) - After:: /home/ys2/anaconda3/envs::/home/ys2/apps/radiology
[2025-11-16 14:31:54,363] [6492] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:65) - COMMAND:: /home/ys2/anaconda3/envs/monailabel/bin/python -m monailabel.interfaces.utils.app -m train -r {"model":"segmentation","name":"train_01","pretrained":true,"device":"NVIDIA GeForce RTX 2080 Ti:0","max_epochs":5,"early_stop_patience":-1,"val_split":0.2,"train_batch_size":1,"val_batch_size":1,"multi_gpu":true,"gpus":"all","dataset":"Dataset","dataloader":"ThreadDataLoader","tracking":"mlflow","tracking_uri":"","tracking_experiment_name":"","client_id":"user-xyz"}
[2025-11-16 14:31:54,640] [6540] [MainThread] [INFO] (__main__:37) - Initializing App from: /home/ys2/apps/radiology; studies: /home/ys2/monailabel-ribfra-data/imagerTrTest; conf: {'models': 'segmentation'}
[2025-11-16 14:31:57,327] [6540] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for MONAILabelApp Found: <class 'main.MyApp'>
[2025-11-16 14:31:57,331] [6540] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.localization_spine.LocalizationSpine'>
[2025-11-16 14:31:57,331] [6540] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_3d.Deepgrow3D'>
[2025-11-16 14:31:57,333] [6540] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_spleen.SegmentationSpleen'>
[2025-11-16 14:31:57,333] [6540] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_vertebra.SegmentationVertebra'>
[2025-11-16 14:31:57,334] [6540] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation.Segmentation'>
[2025-11-16 14:31:57,334] [6540] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepedit.DeepEdit'>
[2025-11-16 14:31:57,334] [6540] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_2d.Deepgrow2D'>
[2025-11-16 14:31:57,335] [6540] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation.Segmentation'>
[2025-11-16 14:31:57,335] [6540] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.localization_vertebra.LocalizationVertebra'>
[2025-11-16 14:31:57,335] [6540] [MainThread] [INFO] (main:95) - +++ Adding Model: segmentation => lib.configs.segmentation.Segmentation
[2025-11-16 14:31:57,469] [6540] [MainThread] [INFO] (main:98) - +++ Using Models: ['segmentation']
[2025-11-16 14:31:57,469] [6540] [MainThread] [INFO] (monailabel.interfaces.app:135) - Init Datastore for: /home/ys2/monailabel-ribfra-data/imagerTrTest
[2025-11-16 14:31:57,470] [6540] [MainThread] [INFO] (monailabel.datastore.local:130) - Auto Reload: False; Extensions: ['*.nii.gz', '*.nii', '*.nrrd', '*.jpg', '*.png', '*.tif', '*.svs', '*.xml']
[2025-11-16 14:31:57,472] [6540] [MainThread] [INFO] (monailabel.datastore.local:577) - Invalidate count: 0
[2025-11-16 14:31:57,474] [6540] [MainThread] [INFO] (main:128) - +++ Adding Inferer:: segmentation => <lib.infers.segmentation.Segmentation object at 0x7fb0e0e8d700>
[2025-11-16 14:31:57,474] [6540] [MainThread] [INFO] (main:194) - {'segmentation': <lib.infers.segmentation.Segmentation object at 0x7fb0e0e8d700>, 'Histogram+GraphCut': <monailabel.scribbles.infer.HistogramBasedGraphCut object at 0x7fb0d8e42fd0>, 'GMM+GraphCut': <monailabel.scribbles.infer.GMMBasedGraphCut object at 0x7fb0d54a0250>}
[2025-11-16 14:31:57,474] [6540] [MainThread] [INFO] (main:209) - +++ Adding Trainer:: segmentation => <lib.trainers.segmentation.Segmentation object at 0x7fb0d54a0070>
[2025-11-16 14:31:57,475] [6540] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: /home/ys2/.cache/monailabel/sessions
[2025-11-16 14:31:57,475] [6540] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2025-11-16 14:31:57,475] [6540] [MainThread] [INFO] (monailabel.tasks.train.basic_train:436) - Train Request (input): {'model': 'segmentation', 'name': 'train_01', 'pretrained': True, 'device': 'NVIDIA GeForce RTX 2080 Ti:0', 'max_epochs': 5, 'early_stop_patience': -1, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1, 'multi_gpu': True, 'gpus': 'all', 'dataset': 'Dataset', 'dataloader': 'ThreadDataLoader', 'tracking': 'mlflow', 'tracking_uri': '', 'tracking_experiment_name': '', 'client_id': 'user-xyz', 'local_rank': 0}
[2025-11-16 14:31:57,475] [6540] [MainThread] [INFO] (monailabel.tasks.train.basic_train:449) - CUDA_VISIBLE_DEVICES: None
[2025-11-16 14:31:57,475] [6540] [MainThread] [INFO] (monailabel.tasks.train.basic_train:459) - Distributed/Multi GPU Training = TRUE
[2025-11-16 14:31:57,475] [6540] [MainThread] [INFO] (monailabel.tasks.train.basic_train:465) - Total proces to spawn: 2
[2025-11-16 14:32:00.433][ INFO](monailabel.tasks.train.basic_train) - Main Worker: 0
[2025-11-16 14:32:00.434][ INFO](monailabel.tasks.train.basic_train) - 0 - Train Request (final): {'name': 'train_01', 'pretrained': True, 'device': 'cuda:0', 'max_epochs': 5, 'early_stop_patience': -1, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1, 'multi_gpu': True, 'gpus': 'all', 'dataset': 'Dataset', 'dataloader': 'ThreadDataLoader', 'tracking': 'mlflow', 'tracking_uri': '', 'tracking_experiment_name': '', 'model': 'segmentation', 'client_id': 'user-xyz', 'local_rank': 0, 'run_id': '20251116_143157'}
[2025-11-16 14:32:00.434][ INFO](monailabel.tasks.train.basic_train) - Main Worker: 1
[2025-11-16 14:32:00.434][ INFO](monailabel.tasks.train.basic_train) - 1 - Train Request (final): {'name': 'train_01', 'pretrained': True, 'device': 'cuda:0', 'max_epochs': 5, 'early_stop_patience': -1, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1, 'multi_gpu': True, 'gpus': 'all', 'dataset': 'Dataset', 'dataloader': 'ThreadDataLoader', 'tracking': 'mlflow', 'tracking_uri': '', 'tracking_experiment_name': '', 'model': 'segmentation', 'client_id': 'user-xyz', 'local_rank': 0, 'run_id': '20251116_143157'}
[2025-11-16 14:32:00.458][ INFO](torch.distributed.distributed_c10d) - Added key: store_based_barrier_key:1 to store for rank: 0
[2025-11-16 14:32:00.467][ INFO](torch.distributed.distributed_c10d) - Added key: store_based_barrier_key:1 to store for rank: 1
[2025-11-16 14:32:00.468][ INFO](torch.distributed.distributed_c10d) - Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
[2025-11-16 14:32:00.468][ INFO](torch.distributed.distributed_c10d) - Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
[2025-11-16 14:32:00.478][ INFO](torch.distributed.distributed_c10d) - Added key: store_based_barrier_key:2 to store for rank: 1
[2025-11-16 14:32:00.478][ INFO](torch.distributed.distributed_c10d) - Added key: store_based_barrier_key:2 to store for rank: 0
[2025-11-16 14:32:00.479][ INFO](torch.distributed.distributed_c10d) - Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
[2025-11-16 14:32:00.479][ INFO](torch.distributed.distributed_c10d) - Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
[2025-11-16 14:32:00.499][ INFO](monailabel.tasks.train.basic_train) - ++++ Rank:0 => Using GPU-0
[2025-11-16 14:32:00.499][ INFO](monailabel.tasks.train.basic_train) - ++++ Rank:1 => Using GPU-1
[2025-11-16 14:32:00.508][ INFO](monailabel.tasks.train.basic_train) - 0 - Using Device: cuda:0; IDX: 0
[2025-11-16 14:32:00.508][ INFO](monailabel.tasks.train.basic_train) - 1 - Using Device: cuda:1; IDX: 1
[2025-11-16 14:32:00.508][ INFO](monailabel.tasks.train.basic_train) - Run/Output Path: /home/ys2/apps/radiology/model/segmentation/train_01
[2025-11-16 14:32:00.509][ INFO](monailabel.tasks.train.basic_train) - Run/Output Path: /home/ys2/apps/radiology/model/segmentation/train_01
[2025-11-16 14:32:00.509][ INFO](monailabel.tasks.train.basic_train) - Tracking: mlflow
[2025-11-16 14:32:00.509][ INFO](monailabel.tasks.train.basic_train) - Tracking URI: file:///home/ys2/apps/radiology/model/segmentation/train_01/mlruns;
[2025-11-16 14:32:00.509][ INFO](monailabel.tasks.train.basic_train) - Tracking Experiment Name: segmentation; Run Name: run_20251116_143157
[2025-11-16 14:32:00.509][ INFO](monailabel.tasks.train.basic_train) - Tracking: mlflow
[2025-11-16 14:32:00.509][ INFO](monailabel.tasks.train.basic_train) - Tracking URI: file:///home/ys2/apps/radiology/model/segmentation/train_01/mlruns;
[2025-11-16 14:32:00.509][ INFO](monailabel.tasks.train.basic_train) - Total Records for Training: 4
[2025-11-16 14:32:00.509][ INFO](monailabel.tasks.train.basic_train) - Tracking Experiment Name: segmentation; Run Name: run_20251116_143157
[2025-11-16 14:32:00.509][ INFO](monailabel.tasks.train.basic_train) - Total Records for Validation: 1
[2025-11-16 14:32:02.737][ INFO](monailabel.tasks.train.basic_train) - 1 - Records for Validation: 1
[2025-11-16 14:32:02.739][ INFO](monailabel.tasks.train.basic_train) - 1 - Adding Validation to run every '1' interval
[2025-11-16 14:32:02.739][ INFO](monailabel.tasks.train.basic_train) - 1 - Load Path /home/ys2/apps/radiology/model/segmentation/train_01/model.pt
[2025-11-16 14:32:02.739][ INFO](lib.trainers.segmentation) - xkq test output{'buckle fracture': 1, 'nondisplaced fracture': 2, 'displaced fracture': 3, 'segmental fracture': 4}
[2025-11-16 14:32:02.739][ INFO](monailabel.tasks.train.basic_train) - 0 - Records for Validation: 1
[2025-11-16 14:32:02.740][ INFO](monailabel.tasks.train.basic_train) - 1 - Records for Training: 2
[2025-11-16 14:32:02.742][ INFO](ignite.engine.engine.SupervisedTrainer) - Engine run resuming from iteration 0, epoch 0 until 5 epochs
[2025-11-16 14:32:02.743][ INFO](monailabel.tasks.train.basic_train) - 0 - Adding Validation to run every '1' interval
[2025-11-16 14:32:02.744][ INFO](monailabel.tasks.train.basic_train) - 0 - Load Path /home/ys2/apps/radiology/model/segmentation/train_01/model.pt
[2025-11-16 14:32:02.744][ INFO](lib.trainers.segmentation) - xkq test output{'buckle fracture': 1, 'nondisplaced fracture': 2, 'displaced fracture': 3, 'segmental fracture': 4}
[2025-11-16 14:32:02.745][ INFO](monailabel.tasks.train.basic_train) - 0 - Records for Training: 2
[2025-11-16 14:32:02.748][ INFO](ignite.engine.engine.SupervisedTrainer) - Engine run resuming from iteration 0, epoch 0 until 5 epochs
[2025-11-16 14:32:02.780][ INFO](ignite.engine.engine.SupervisedTrainer) - Restored all variables from /home/ys2/apps/radiology/model/segmentation/train_01/model.pt
[2025-11-16 14:32:02.814][ INFO](ignite.engine.engine.SupervisedTrainer) - Restored all variables from /home/ys2/apps/radiology/model/segmentation/train_01/model.pt
2025-11-16 14:32:10,274 - INFO - Epoch: 1/5, Iter: 1/2 -- train_loss: 1.6281
[2025-11-16 14:32:14.235][ INFO](torch.nn.parallel.distributed) - Reducer buckets have been rebuilt in this iteration.
[2025-11-16 14:32:14.625][ INFO](torch.nn.parallel.distributed) - Reducer buckets have been rebuilt in this iteration.
2025-11-16 14:32:15,170 - INFO - Epoch: 1/5, Iter: 2/2 -- train_loss: 1.4654
[2025-11-16 14:32:15.176][ INFO](ignite.engine.engine.SupervisedTrainer) - Got new best metric of train_mean_dice: 0.0
[2025-11-16 14:32:15.176][ INFO](ignite.engine.engine.SupervisedTrainer) - Got new best metric of train_mean_dice: 0.0
[2025-11-16 14:32:15.176][ INFO](ignite.engine.engine.SupervisedEvaluator) - Engine run resuming from iteration 0, epoch 0 until 1 epochs
2025-11-16 14:32:15,176 - INFO - Epoch[1] Metrics -- train_buckle fracture_mean_dice: 0.0000 train_displaced fracture_mean_dice: 0.0000 train_mean_dice: 0.0000 train_nondisplaced fracture_mean_dice: 0.0000 train_segmental fracture_mean_dice: 0.0000
2025-11-16 14:32:15,176 - INFO - Key metric: train_mean_dice best value: 0.0 at epoch: 1
[2025-11-16 14:32:15.178][ INFO](ignite.engine.engine.SupervisedEvaluator) - Engine run resuming from iteration 0, epoch 0 until 1 epochs
[2025-11-16 14:32:32.961][ INFO](ignite.engine.engine.SupervisedEvaluator) - Got new best metric of val_mean_dice: 0.0
[2025-11-16 14:32:32.961][ INFO](ignite.engine.engine.SupervisedEvaluator) - Got new best metric of val_mean_dice: 0.0
[2025-11-16 14:32:32.961][ INFO](ignite.engine.engine.SupervisedEvaluator) - Epoch[1] Complete. Time taken: 00:00:17.785
2025-11-16 14:32:32,961 - INFO - Epoch[1] Metrics -- val_buckle fracture_mean_dice: 0.0000 val_displaced fracture_mean_dice: 0.0000 val_mean_dice: 0.0000 val_nondisplaced fracture_mean_dice: 0.0000 val_segmental fracture_mean_dice: 0.0000
2025-11-16 14:32:32,961 - INFO - Key metric: val_mean_dice best value: 0.0 at epoch: 1
[2025-11-16 14:32:32.961][ INFO](ignite.engine.engine.SupervisedEvaluator) - Engine run complete. Time taken: 00:00:17.785
[2025-11-16 14:32:32.961][ INFO](ignite.engine.engine.SupervisedTrainer) - Epoch[1] Complete. Time taken: 00:00:30.182
[2025-11-16 14:32:33.110][ INFO](monailabel.tasks.train.handler) - New Model published: /home/ys2/apps/radiology/model/segmentation/train_01/model.pt => /home/ys2/apps/radiology/model/segmentation.pt
[2025-11-16 14:32:33.111][ INFO](ignite.engine.engine.SupervisedEvaluator) - Epoch[1] Complete. Time taken: 00:00:17.917
[2025-11-16 14:32:33.112][ INFO](ignite.engine.engine.SupervisedEvaluator) - Engine run complete. Time taken: 00:00:17.934
[2025-11-16 14:32:33.290][ INFO](ignite.engine.engine.SupervisedTrainer) - Epoch[1] Complete. Time taken: 00:00:30.476
2025-11-16 14:32:39,817 - INFO - Epoch: 2/5, Iter: 1/2 -- train_loss: 1.3654
2025-11-16 14:32:45,137 - INFO - Epoch: 2/5, Iter: 2/2 -- train_loss: 1.3366
2025-11-16 14:32:45,144 - INFO - Epoch[2] Metrics -- train_buckle fracture_mean_dice: 0.0000 train_displaced fracture_mean_dice: 0.0000 train_mean_dice: 0.0000 train_nondisplaced fracture_mean_dice: 0.0000 train_segmental fracture_mean_dice: 0.0000
2025-11-16 14:32:45,144 - INFO - Key metric: train_mean_dice best value: 0.0 at epoch: 1
[2025-11-16 14:32:45.144][ INFO](ignite.engine.engine.SupervisedEvaluator) - Engine run resuming from iteration 0, epoch 1 until 2 epochs
[2025-11-16 14:32:45.146][ INFO](ignite.engine.engine.SupervisedEvaluator) - Engine run resuming from iteration 0, epoch 1 until 2 epochs
2025-11-16 14:33:02,749 - INFO - Epoch[2] Metrics -- val_buckle fracture_mean_dice: 0.0000 val_displaced fracture_mean_dice: 0.0000 val_mean_dice: 0.0000 val_nondisplaced fracture_mean_dice: 0.0000 val_segmental fracture_mean_dice: 0.0000
[2025-11-16 14:33:02.749][ INFO](ignite.engine.engine.SupervisedEvaluator) - Epoch[2] Complete. Time taken: 00:00:17.605
2025-11-16 14:33:02,749 - INFO - Key metric: val_mean_dice best value: 0.0 at epoch: 1
[2025-11-16 14:33:02.749][ INFO](ignite.engine.engine.SupervisedEvaluator) - Engine run complete. Time taken: 00:00:17.605
[2025-11-16 14:33:02.749][ INFO](ignite.engine.engine.SupervisedTrainer) - Epoch[2] Complete. Time taken: 00:00:29.788
[2025-11-16 14:33:02.791][ INFO](ignite.engine.engine.SupervisedEvaluator) - Epoch[2] Complete. Time taken: 00:00:17.629
[2025-11-16 14:33:02.793][ INFO](ignite.engine.engine.SupervisedEvaluator) - Engine run complete. Time taken: 00:00:17.647
[2025-11-16 14:33:02.793][ INFO](ignite.engine.engine.SupervisedTrainer) - Epoch[2] Complete. Time taken: 00:00:29.503
2025-11-16 14:33:09,800 - INFO - Epoch: 3/5, Iter: 1/2 -- train_loss: 1.3151
2025-11-16 14:33:14,707 - INFO - Epoch: 3/5, Iter: 2/2 -- train_loss: 1.3019
[2025-11-16 14:33:14.713][ INFO](ignite.engine.engine.SupervisedEvaluator) - Engine run resuming from iteration 0, epoch 2 until 3 epochs
2025-11-16 14:33:14,713 - INFO - Epoch[3] Metrics -- train_buckle fracture_mean_dice: 0.0000 train_displaced fracture_mean_dice: 0.0000 train_mean_dice: 0.0000 train_nondisplaced fracture_mean_dice: 0.0000 train_segmental fracture_mean_dice: 0.0000
2025-11-16 14:33:14,713 - INFO - Key metric: train_mean_dice best value: 0.0 at epoch: 1
[2025-11-16 14:33:14.715][ INFO](ignite.engine.engine.SupervisedEvaluator) - Engine run resuming from iteration 0, epoch 2 until 3 epochs
2025-11-16 14:33:32,295 - INFO - Epoch[3] Metrics -- val_buckle fracture_mean_dice: 0.0000 val_displaced fracture_mean_dice: 0.0000 val_mean_dice: 0.0000 val_nondisplaced fracture_mean_dice: 0.0000 val_segmental fracture_mean_dice: 0.0000
[2025-11-16 14:33:32.295][ INFO](ignite.engine.engine.SupervisedEvaluator) - Epoch[3] Complete. Time taken: 00:00:17.581
2025-11-16 14:33:32,295 - INFO - Key metric: val_mean_dice best value: 0.0 at epoch: 1
[2025-11-16 14:33:32.295][ INFO](ignite.engine.engine.SupervisedEvaluator) - Engine run complete. Time taken: 00:00:17.582
[2025-11-16 14:33:32.295][ INFO](ignite.engine.engine.SupervisedTrainer) - Epoch[3] Complete. Time taken: 00:00:29.546
[2025-11-16 14:33:32.298][ INFO](ignite.engine.engine.SupervisedEvaluator) - Epoch[3] Complete. Time taken: 00:00:17.564
[2025-11-16 14:33:32.299][ INFO](ignite.engine.engine.SupervisedEvaluator) - Engine run complete. Time taken: 00:00:17.583
[2025-11-16 14:33:32.299][ INFO](ignite.engine.engine.SupervisedTrainer) - Epoch[3] Complete. Time taken: 00:00:29.506
2025-11-16 14:33:38,475 - INFO - Epoch: 4/5, Iter: 1/2 -- train_loss: 1.2941
2025-11-16 14:33:44,256 - INFO - Epoch: 4/5, Iter: 2/2 -- train_loss: 1.2827
2025-11-16 14:33:44,263 - INFO - Epoch[4] Metrics -- train_buckle fracture_mean_dice: 0.0000 train_displaced fracture_mean_dice: 0.0000 train_mean_dice: 0.0000 train_nondisplaced fracture_mean_dice: 0.0000 train_segmental fracture_mean_dice: 0.0000
[2025-11-16 14:33:44.263][ INFO](ignite.engine.engine.SupervisedEvaluator) - Engine run resuming from iteration 0, epoch 3 until 4 epochs
2025-11-16 14:33:44,263 - INFO - Key metric: train_mean_dice best value: 0.0 at epoch: 1
[2025-11-16 14:33:44.265][ INFO](ignite.engine.engine.SupervisedEvaluator) - Engine run resuming from iteration 0, epoch 3 until 4 epochs
2025-11-16 14:34:02,001 - INFO - Epoch[4] Metrics -- val_buckle fracture_mean_dice: 0.0000 val_displaced fracture_mean_dice: 0.0000 val_mean_dice: 0.0000 val_nondisplaced fracture_mean_dice: 0.0000 val_segmental fracture_mean_dice: 0.0000
[2025-11-16 14:34:02.002][ INFO](ignite.engine.engine.SupervisedEvaluator) - Epoch[4] Complete. Time taken: 00:00:17.739
2025-11-16 14:34:02,002 - INFO - Key metric: val_mean_dice best value: 0.0 at epoch: 1
[2025-11-16 14:34:02.002][ INFO](ignite.engine.engine.SupervisedEvaluator) - Engine run complete. Time taken: 00:00:17.739
[2025-11-16 14:34:02.002][ INFO](ignite.engine.engine.SupervisedTrainer) - Epoch[4] Complete. Time taken: 00:00:29.706
[2025-11-16 14:34:02.004][ INFO](ignite.engine.engine.SupervisedEvaluator) - Epoch[4] Complete. Time taken: 00:00:17.721
[2025-11-16 14:34:02.005][ INFO](ignite.engine.engine.SupervisedEvaluator) - Engine run complete. Time taken: 00:00:17.741
[2025-11-16 14:34:02.005][ INFO](ignite.engine.engine.SupervisedTrainer) - Epoch[4] Complete. Time taken: 00:00:29.707
2025-11-16 14:34:08,313 - INFO - Epoch: 5/5, Iter: 1/2 -- train_loss: 1.2752
2025-11-16 14:34:13,849 - INFO - Epoch: 5/5, Iter: 2/2 -- train_loss: 1.2724
[2025-11-16 14:34:13.856][ INFO](ignite.engine.engine.SupervisedEvaluator) - Engine run resuming from iteration 0, epoch 4 until 5 epochs
2025-11-16 14:34:13,856 - INFO - Epoch[5] Metrics -- train_buckle fracture_mean_dice: 0.0000 train_displaced fracture_mean_dice: 0.0000 train_mean_dice: 0.0000 train_nondisplaced fracture_mean_dice: 0.0000 train_segmental fracture_mean_dice: 0.0000
2025-11-16 14:34:13,856 - INFO - Key metric: train_mean_dice best value: 0.0 at epoch: 1
[2025-11-16 14:34:13.858][ INFO](ignite.engine.engine.SupervisedEvaluator) - Engine run resuming from iteration 0, epoch 4 until 5 epochs
2025-11-16 14:34:31,483 - INFO - Epoch[5] Metrics -- val_buckle fracture_mean_dice: 0.0000 val_displaced fracture_mean_dice: 0.0000 val_mean_dice: 0.0000 val_nondisplaced fracture_mean_dice: 0.0000 val_segmental fracture_mean_dice: 0.0000
[2025-11-16 14:34:31.483][ INFO](ignite.engine.engine.SupervisedEvaluator) - Epoch[5] Complete. Time taken: 00:00:17.627
2025-11-16 14:34:31,483 - INFO - Key metric: val_mean_dice best value: 0.0 at epoch: 1
[2025-11-16 14:34:31.483][ INFO](ignite.engine.engine.SupervisedEvaluator) - Engine run complete. Time taken: 00:00:17.627
[2025-11-16 14:34:31.483][ INFO](ignite.engine.engine.SupervisedTrainer) - Epoch[5] Complete. Time taken: 00:00:29.481
[2025-11-16 14:34:31.483][ INFO](ignite.engine.engine.SupervisedTrainer) - Engine run complete. Time taken: 00:02:28.741
[2025-11-16 14:34:31.485][ INFO](ignite.engine.engine.SupervisedEvaluator) - Epoch[5] Complete. Time taken: 00:00:17.607
[2025-11-16 14:34:31.486][ INFO](ignite.engine.engine.SupervisedEvaluator) - Engine run complete. Time taken: 00:00:17.628
[2025-11-16 14:34:31.486][ INFO](ignite.engine.engine.SupervisedTrainer) - Epoch[5] Complete. Time taken: 00:00:29.481
[2025-11-16 14:34:31.584][ INFO](ignite.engine.engine.SupervisedTrainer) - Train completed, saved final checkpoint: /home/ys2/apps/radiology/model/segmentation/train_01/checkpoint_final.pt
[2025-11-16 14:34:31.584][ INFO](ignite.engine.engine.SupervisedTrainer) - Engine run complete. Time taken: 00:02:28.836
[2025-11-16 14:34:32,277] [6540] [MainThread] [INFO] (monailabel.tasks.train.basic_train:612) - Running cleanup...
[2025-11-16 14:34:32,278] [6540] [MainThread] [INFO] (__main__:61) - Result: {"rank": 0, "current_epoch": 5, "current_iteration": 10, "total_epochs": 5, "total_iterations": 2, "epoch": 5, "start_ts": 1763274720, "total_time": "0:02:31", "best_metric": 0.0, "train": {"metrics": {"train_mean_dice": 0.0, "train_buckle fracture_mean_dice": 0.0, "train_nondisplaced fracture_mean_dice": 0.0, "train_displaced fracture_mean_dice": 0.0, "train_segmental fracture_mean_dice": 0.0}, "key_metric_name": "train_mean_dice", "best_metric": 0.0, "best_metric_epoch": 1}, "eval": {"metrics": {"val_mean_dice": 0.0, "val_buckle fracture_mean_dice": 0.0, "val_nondisplaced fracture_mean_dice": 0.0, "val_displaced fracture_mean_dice": 0.0, "val_segmental fracture_mean_dice": 0.0}, "key_metric_name": "val_mean_dice", "best_metric": 0.0, "best_metric_epoch": 1}}
[2025-11-16 14:34:32,824] [6492] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:83) - Return code: 0