[Bug] gt_instances keypoints are all zeros when fine-tuning on COCO

[calvinyeungck]

[Bug] gt_instances keypoints are all zeros when fine-tuning on COCO

Description

I encountered an issue where the keypoints in gt_instances are always an array of zeros during training (see the slurm-6618226.txt at the bottom). I have checked browse_dataset.py and PackPoseInputs, and both seem to be working correctly. However, I can't figure out where the problem might be.

Please suggest any potential errors or solutions. Thank you!

Steps to Reproduce

Attempt to fine-tune the model on the COCO dataset using the following command:

python ./mmpose/tools/train.py \
    ./mmpose/configs/body_2d_keypoint/topdown_heatmap/ap2d/td-hm_hrnet-w48_8xb32-210e_coco-256x192_testing.py

Expected Behavior

The gt_instances.keypoints should contain the correct annotated keypoint coordinates from the dataset instead of an array of zeros.

Modifications

The only modification I made was updating the settings in the configuration file and print out the datasample inside the CocoMetric.

Environment

• Python 3.8.20
• mmpose 1.3.2
• mmengine 0.10.6
• mmcv 2.2.0
• mmdet 3.3.0

The settings

_base_ = ['../../../_base_/default_runtime.py']

# runtime
train_cfg = dict(max_epochs=1, val_interval=1) #test

# optimizer
optim_wrapper = dict(optimizer=dict(
    type='Adam',
    lr=5e-4,
))

# use the pre-trained model for the whole HRNet
load_from = './mmpose/configs/body_2d_keypoint/topdown_heatmap/ap2d/td-hm_hrnet-w48_8xb32-210e_coco-384x288-c161b7de_20220915.pth'

# fintune learning policy
param_scheduler = [
    dict(
        type='LinearLR',
        begin=0,
        end=10,
        start_factor=1e-4,
        by_epoch=True
    ),  # Warm-up phase
    dict(
        type='MultiStepLR',
        begin=10,
        end=20,
        milestones=[15],
        gamma=0.1,
        by_epoch=True
    )  # Learning rate decay
]



# automatically scaling LR based on the actual training batch size
auto_scale_lr = dict(base_batch_size=512)

# hooks
# default_hooks = dict(checkpoint=dict(save_best='tPCK', rule='greater'))
default_hooks = dict(checkpoint=dict(type='CheckpointHook', interval=1))

# codec settings
codec = dict(
    type='MSRAHeatmap', input_size=(288, 384), heatmap_size=(72, 96), sigma=3)

# model settings
model = dict(
    type='TopdownPoseEstimator',
    data_preprocessor=dict(
        type='PoseDataPreprocessor',
        mean=[123.675, 116.28, 103.53],
        std=[58.395, 57.12, 57.375],
        bgr_to_rgb=True),
    backbone=dict(
        type='HRNet',
        in_channels=3,
        extra=dict(
            stage1=dict(
                num_modules=1,
                num_branches=1,
                block='BOTTLENECK',
                num_blocks=(4, ),
                num_channels=(64, )),
            stage2=dict(
                num_modules=1,
                num_branches=2,
                block='BASIC',
                num_blocks=(4, 4),
                num_channels=(48, 96)),
            stage3=dict(
                num_modules=4,
                num_branches=3,
                block='BASIC',
                num_blocks=(4, 4, 4),
                num_channels=(48, 96, 192)),
            stage4=dict(
                num_modules=3,
                num_branches=4,
                block='BASIC',
                num_blocks=(4, 4, 4, 4),
                num_channels=(48, 96, 192, 384))),
        init_cfg=dict(
            type='Pretrained',
            checkpoint='https://download.openmmlab.com/mmpose/'
            'pretrain_models/hrnet_w48-8ef0771d.pth'),
    ),
    head=dict(
        type='HeatmapHead',
        in_channels=48,
        out_channels=17,
        deconv_out_channels=None,
        loss=dict(type='KeypointMSELoss', use_target_weight=True),
        decoder=codec),
    test_cfg=dict(
        flip_test=True,
        flip_mode='heatmap',
        shift_heatmap=True,
    ))

# base dataset settings
dataset_type = 'CocoDataset'
data_mode = 'topdown'
data_root = '/fsws1/share/database/coco/'

# pipelines
train_pipeline = [
    dict(type='LoadImage'),
    dict(type='GetBBoxCenterScale'),
    dict(type='RandomFlip', direction='horizontal'),
    dict(type='RandomHalfBody'),
    dict(type='RandomBBoxTransform'),
    dict(type='TopdownAffine', input_size=codec['input_size']),
    dict(type='GenerateTarget', encoder=codec),
    dict(type='PackPoseInputs')
]
val_pipeline = [
    dict(type='LoadImage'),
    dict(type='GetBBoxCenterScale'),
    dict(type='TopdownAffine', input_size=codec['input_size']),
    dict(type='PackPoseInputs')
]

train_dataloader = dict(
    batch_size=64,
    num_workers=4,
    persistent_workers=True,
    sampler=dict(type='DefaultSampler', shuffle=True),
    dataset=dict(
        type=dataset_type,
        data_root=data_root,
        data_mode=data_mode,
        ann_file='annotations/person_keypoints_train2017.json',
        data_prefix=dict(img='images/train2017/'),
        pipeline=train_pipeline,
        metainfo = dict(from_file='configs/_base_/datasets/coco.py')
    ))

val_dataloader = dict(
    batch_size=64,
    num_workers=4,
    persistent_workers=True,
    drop_last=False,
    sampler=dict(type='DefaultSampler', shuffle=False, round_up=False),
    dataset=dict(
        type=dataset_type,
        data_root=data_root,
        data_mode=data_mode,
        ann_file='annotations/person_keypoints_val2017.json',
        bbox_file='./mmpose_train/'
        'COCO_val2017_detections_AP_H_70_person.json',
        data_prefix=dict(img='images/val2017/'),
        test_mode=True,
        pipeline=val_pipeline,
        metainfo = dict(from_file='configs/_base_/datasets/coco.py')
    ))

test_dataloader = val_dataloader

# evaluators
val_evaluator = dict(
    type='CocoMetric',
    ann_file=data_root + 'annotations/person_keypoints_val2017.json')

test_evaluator = val_evaluator