Training for custom datasets containing only LiDAR scans + annotations

[madan96]

My dataset contains pointcloud files and 3D bbox annotations in LiDAR coordinates.
I looked into converting my data into the KITTI format but I see that preparing KITTI dataset for training requires the rotation and transformation matrices from calib files and a separate images folder (which is probably optional to have). Is there a way to modify the KITTI dataset class and related classes/functions to get this to work w/o the calibration files and images?

Or perhaps, any other dataset classes that might be easier to modify to use my dataset with?

[ZwwWayne]

Hi @madan96 ,
Thanks for your kind suggestion. We will simplify the process and reduce the requirements of unnecessary data structure. For now, you might need to re-implement a dataset class that does not need these things.

[EricWiener]

@madan96 is there any chance you could please share how you prepared your dataset to use with MMDetection3D. I'm in the same situation you are in

[divyanshj16]

@madan96 I am working on something similar. I have drone lidar data and 3d bboxes. My data is in the form of las files and I have annotated 3d bounding boxes for cars. I know which boxes belong to which las files.
@madan96 Were you able to do training?
Converting to KITTI format is not very straight forward.

In addition to that, the point clouds here have a dimension of Nx4, I am not sure how is it 4, I only have XYZ.
I saw here that:

mmdetection3d/mmdet3d/datasets/pipelines/loading.py

Line 328 in 63bdad9

class LoadPointsFromFile(object):

it can take be arbitrary number of dimensions, how is that possible? @ZwwWayne

[madan96]

Hi @divyanshj16, I never tried training the network and went with another approach I was trying out. I have this branch on my fork where I committed the changes I made according to @ZwwWayne's suggestion. I remember I was able to run this code with my dataset for an epoch or 2. My suggestion would be to try converting the las files to bin or write a different LoadPointsFromFile function for your pointcloud files.

The Nx4 is for XYZI, where I stands for intensity. I think you'd be better off writing a different loading method and updating its name in the configs, along with the dimension of your cloud.
Unfortunately, I don't think I have any more information about this since it's been a while since I used this package. Good luck!

[clytze0216]

t
Hi,@madan96.sorry for my bother.
I also want to costom my own datasets which only contain anno and pointclouds.
I want to ask you if I could only replace the velodyne and label_2 to train the model?
Looking forward to your reply. Thansk again.

[curiosity654]

@madan96 I am working on something similar. I have drone lidar data and 3d bboxes. My data is in the form of las files and I have annotated 3d bounding boxes for cars. I know which boxes belong to which las files.
@madan96 Were you able to do training?
Converting to KITTI format is not very straight forward.

In addition to that, the point clouds here have a dimension of Nx4, I am not sure how is it 4, I only have XYZ.
I saw here that:

mmdetection3d/mmdet3d/datasets/pipelines/loading.py

Line 328 in 63bdad9

class LoadPointsFromFile(object):

it can take be arbitrary number of dimensions, how is that possible? @ZwwWayne

@divyanshj16
You can refer to the code for LoadPointsFromFile, where load_dim is for reshape the loaded raw data, and use_dim is for indexing the dimensions you need.

[divyanshj16]

Hi @madan96
Thanks for your help. I have been able to create custom dataset from my las files.
I wanted to know how are you setting this point cloud range and voxel size?
https://github.com/madan96/mmdetection3d/blob/c793c61c17562974ba8134ee086b95199ea33f06/configs/_base_/datasets/base-data.py#L5

[divyanshj16]

@ZwwWayne @madan96
Please help with my issue below.

I am able to train the model and the loss is also converging very nicely. I trained on custom datasets and annotations. I am training in min-max setting without yaw.
I did this by implementing a custom dataset and use the model build from build_model function. Below is the loss table and loss curve.

epoch	train_loss	valid_loss	time
0	4091.541260	4339.893066	01:12
1	3511.875244	3873.412598	01:14
2	3023.028320	3248.494141	01:15
3	2599.338623	2618.090332	01:16
4	2216.144775	2036.166260	01:15
5	1868.235840	1544.580811	01:16
6	1554.849365	1141.499268	01:16
7	1278.622559	819.518127	01:16
8	1042.989502	566.451904	01:16
9	846.021851	397.228149	01:16
10	682.200989	297.114288	01:16
11	546.985229	231.899872	01:16
12	436.301361	174.552322	01:16
13	346.867157	129.714447	01:16
14	275.539581	100.248444	01:16
15	219.054153	88.966576	01:16
16	174.229584	70.900986	01:16
17	138.545380	59.210781	01:16
18	110.305984	44.358696	01:16
19	88.070435	36.418659	01:16
20	70.549622	28.117123	01:16
21	56.747650	22.572691	01:16
22	45.944958	19.011368	01:16
23	37.335182	15.910639	01:16
24	30.486134	13.948141	01:16
25	25.047031	11.473155	01:16
26	20.718050	9.987444	01:16
27	17.267614	8.705861	01:16
28	14.527399	7.983851	01:16
29	12.329118	6.875288	01:16
30	10.539511	7.063208	01:16
31	9.089251	5.811657	01:16
32	7.899366	5.649652	01:16
33	6.933311	5.285494	01:16
34	6.134204	4.850323	01:16
35	5.471034	4.785600	01:16
36	4.917679	4.579283	01:16
37	4.439167	4.457599	01:16
38	4.053056	4.256866	01:16
39	3.714704	3.735523	01:16
40	3.434179	3.786665	01:16
41	3.190169	3.675888	01:16
42	2.981537	3.819055	01:16
43	2.785687	3.371310	01:16
44	2.618291	3.546071	01:16
45	2.475881	3.480840	01:16
46	2.339902	2.912966	01:16
47	2.233608	3.440542	01:16
48	2.134134	3.117487	01:16
49	2.050812	3.134299	01:16
50	1.956867	3.031329	01:16
51	1.870409	2.786792	01:16
52	1.788638	2.800385	01:16
53	1.736393	3.059615	01:16
54	1.686984	2.619216	01:16
55	1.642131	2.836412	01:16
56	1.597658	2.705651	01:16
57	1.553393	2.757333	01:16
58	1.518408	2.360894	01:16
59	1.477399	4.827517	01:16
60	1.455092	3.272039	01:16
61	1.423524	2.625373	01:16
62	1.382070	2.436401	01:16
63	1.343764	2.449071	01:16
64	1.298866	2.415044	01:17
65	1.247886	2.494784	01:17
66	1.214924	2.600033	01:17
67	1.187451	2.414974	01:16
68	1.156835	2.469126	01:17
69	1.122727	2.689886	01:16
70	1.079685	2.463716	01:17
71	1.036399	2.303868	01:17
72	0.989412	2.355410	01:17
73	0.947672	2.473300	01:17
74	0.904058	2.421267	01:16
75	0.865463	2.577287	01:17
76	0.827695	2.406321	01:17
77	0.794519	2.514960	01:17
78	0.767848	2.357939	01:17
79	0.743575	2.441958	01:17
80	0.708842	2.554681	01:17
81	0.675347	2.466911	01:17
82	0.642671	2.425339	01:17
83	0.612111	2.494488	01:16
84	0.574456	2.569280	01:17
85	0.545474	2.516769	01:17
86	0.518124	2.535384	01:17
87	0.492167	2.539050	01:17
88	0.463795	2.564949	01:17
89	0.435068	2.553125	01:17
90	0.409151	2.582814	01:17
91	0.386457	2.587133	01:17
92	0.367280	2.591264	01:17
93	0.351287	2.593058	01:17
94	0.338143	2.591820	01:17
95	0.327476	2.591178	01:17
96	0.318956	2.590443	01:17
97	0.312278	2.588674	01:17
98	0.307149	2.587346	01:17
99	0.303295	2.586841	01:17

However, after such good training, the results on the validation set are not good. On training I am getting reasonable results, but on validation I am getting poor results.

training set example

ground truth

prediction

validation set example

ground truth

prediction

Although, the yaw was 0 in my case, still it's predicting it, mostly ~2pi, sometimes ~pi/2

My questions:

1. What can I do to improve performance on the validation/test set and why is it happening?
   Could it be because of the fact that the training loss came down so low and validation is still stuck around 2.58?
   Some more facts: I have relatively small data, 30 point cloud patches containing approximately each containing 1.3 million points on an average with a total of ~400 labelled cars. My validation data contains ~30 labelled cars and 4 point cloud patches with 0.5 million points on an average.

2. Why is it predicting Yaw?

[Fungungun]

@ZwwWayne @madan96
Please help with my issue below.

I am able to train the model and the loss is also converging very nicely. I trained on custom datasets and annotations. I am training in min-max setting without yaw.
I did this by implementing a custom dataset and use the model build from build_model function. Below is the loss table and loss curve.

epoch train_loss valid_loss time
0 4091.541260 4339.893066 01:12
1 3511.875244 3873.412598 01:14
2 3023.028320 3248.494141 01:15
3 2599.338623 2618.090332 01:16
4 2216.144775 2036.166260 01:15
5 1868.235840 1544.580811 01:16
6 1554.849365 1141.499268 01:16
7 1278.622559 819.518127 01:16
8 1042.989502 566.451904 01:16
9 846.021851 397.228149 01:16
10 682.200989 297.114288 01:16
11 546.985229 231.899872 01:16
12 436.301361 174.552322 01:16
13 346.867157 129.714447 01:16
14 275.539581 100.248444 01:16
15 219.054153 88.966576 01:16
16 174.229584 70.900986 01:16
17 138.545380 59.210781 01:16
18 110.305984 44.358696 01:16
19 88.070435 36.418659 01:16
20 70.549622 28.117123 01:16
21 56.747650 22.572691 01:16
22 45.944958 19.011368 01:16
23 37.335182 15.910639 01:16
24 30.486134 13.948141 01:16
25 25.047031 11.473155 01:16
26 20.718050 9.987444 01:16
27 17.267614 8.705861 01:16
28 14.527399 7.983851 01:16
29 12.329118 6.875288 01:16
30 10.539511 7.063208 01:16
31 9.089251 5.811657 01:16
32 7.899366 5.649652 01:16
33 6.933311 5.285494 01:16
34 6.134204 4.850323 01:16
35 5.471034 4.785600 01:16
36 4.917679 4.579283 01:16
37 4.439167 4.457599 01:16
38 4.053056 4.256866 01:16
39 3.714704 3.735523 01:16
40 3.434179 3.786665 01:16
41 3.190169 3.675888 01:16
42 2.981537 3.819055 01:16
43 2.785687 3.371310 01:16
44 2.618291 3.546071 01:16
45 2.475881 3.480840 01:16
46 2.339902 2.912966 01:16
47 2.233608 3.440542 01:16
48 2.134134 3.117487 01:16
49 2.050812 3.134299 01:16
50 1.956867 3.031329 01:16
51 1.870409 2.786792 01:16
52 1.788638 2.800385 01:16
53 1.736393 3.059615 01:16
54 1.686984 2.619216 01:16
55 1.642131 2.836412 01:16
56 1.597658 2.705651 01:16
57 1.553393 2.757333 01:16
58 1.518408 2.360894 01:16
59 1.477399 4.827517 01:16
60 1.455092 3.272039 01:16
61 1.423524 2.625373 01:16
62 1.382070 2.436401 01:16
63 1.343764 2.449071 01:16
64 1.298866 2.415044 01:17
65 1.247886 2.494784 01:17
66 1.214924 2.600033 01:17
67 1.187451 2.414974 01:16
68 1.156835 2.469126 01:17
69 1.122727 2.689886 01:16
70 1.079685 2.463716 01:17
71 1.036399 2.303868 01:17
72 0.989412 2.355410 01:17
73 0.947672 2.473300 01:17
74 0.904058 2.421267 01:16
75 0.865463 2.577287 01:17
76 0.827695 2.406321 01:17
77 0.794519 2.514960 01:17
78 0.767848 2.357939 01:17
79 0.743575 2.441958 01:17
80 0.708842 2.554681 01:17
81 0.675347 2.466911 01:17
82 0.642671 2.425339 01:17
83 0.612111 2.494488 01:16
84 0.574456 2.569280 01:17
85 0.545474 2.516769 01:17
86 0.518124 2.535384 01:17
87 0.492167 2.539050 01:17
88 0.463795 2.564949 01:17
89 0.435068 2.553125 01:17
90 0.409151 2.582814 01:17
91 0.386457 2.587133 01:17
92 0.367280 2.591264 01:17
93 0.351287 2.593058 01:17
94 0.338143 2.591820 01:17
95 0.327476 2.591178 01:17
96 0.318956 2.590443 01:17
97 0.312278 2.588674 01:17
98 0.307149 2.587346 01:17
99 0.303295 2.586841 01:17

However, after such good training, the results on the validation set are not good. On training I am getting reasonable results, but on validation I am getting poor results.

training set example

ground truth

prediction

validation set example

ground truth

prediction

Although, the yaw was 0 in my case, still it's predicting it, mostly ~2pi, sometimes ~pi/2

My questions:

1. What can I do to improve performance on the validation/test set and why is it happening?
   Could it be because of the fact that the training loss came down so low and validation is still stuck around 2.58?
   Some more facts: I have relatively small data, 30 point cloud patches containing approximately each containing 1.3 million points on an average with a total of ~400 labelled cars. My validation data contains ~30 labelled cars and 4 point cloud patches with 0.5 million points on an average.
2. Why is it predicting Yaw?

hi did you train on point cloud data only? did you convert the coordinate system to the kitti coordinate system? I really appreciate if you could share your code on training custom dataset.

Thanks