Customized dataset training with large synthesized dataset and small real dataset

[PrinceCatt]

I synthesized data using Blender and trained for 200 epochs with 5k images.
When I use this weight to do inference on some other synthesized data, it showed incomplete cuboid detection for some of the images, while it correctly detected the cuboid for some. But what is causing this problem? There was no occlusion of the object.

Based on the weights from synthesized data, I then trained with 300 real images (75 + augmentation). But when I then used it to predict the real images in the training set, it also showed incomplete cuboid detection, although some of the corner points were recognized.

At a certain epoch (800), the model performed some degree of ability to recognize its training set, yet was still unable to give a cuboid. At more or less epoch (700/900), it was nothing with a non-converging belief map.

So may I ask what is causing this problem of incomplete cuboid detection? When I take a look at the cli output there are actually 8 points. Are there any recommendations on how to improve?

[PrinceCatt]

Have you tried using a model trained with synthesized data to predict real images? I wonder if it is my poor modelling of my target that is causing the problem, or if it is not feasible at all.

[nv-jeff]

Most of the DOPE training I've done has been with synthetic images, although we have found that a mixture of real and synthetic usually provides the best results.

You could be running into one of several problems:

• DOPE converges slowly. You may need 1000s of epochs to train to train a network to the accuracy reported in the paper (about 70% w/ 1cm pose accuracy)
• You may need more data. 300 real + 5k synthetic is good, but more data in varied domains will help.
• The threshold for "detection" of a predicted cuboid point can be changed if you're willing to have more detections at the cost of lower accuracy. Look in config/config_pose.yaml. At the bottom of the file are "config params for DOPE"; varying these (especially thresh_points) can loosen the detection criteria. I suggest you dig into the code to see how these parameters are used before experimenting.