stride_valid assert faild. non-contiguous stride can't handled #10
Description
Here is my log with a bug, how can I solve this problem?
[Exception|indice_conv|subm]feat=torch.Size([137566, 16]),w=torch.Size([7, 7, 7, 16, 16]),pair=torch.Size([2, 343, 137566]),pairnum=tensor([ 5001, 4071, 2915, 2876, 2559, 2547, 2521, 4382, 5716, 4867,
3664, 3167, 3121, 2835, 3526, 5018, 7021, 6005, 4319, 3926,
3575, 3916, 4679, 6829, 9400, 7201, 5169, 4423, 4106, 4546,
5253, 7129, 7777, 5756, 3978, 3274, 3504, 3874, 4283, 5309,
6038, 4567, 2878, 2974, 3229, 3410, 3441, 4449, 5227, 4487,
3883, 3429, 3430, 3017, 2970, 2785, 4539, 5350, 4998, 4443,
3730, 3469, 3128, 4072, 5174, 6710, 6427, 4945, 4201, 3716,
4024, 4918, 6864, 8956, 6847, 5047, 4072, 3896, 4419, 5554,
7007, 7141, 5674, 4268, 3469, 3878, 4567, 4992, 5347, 5751,
4694, 3185, 3406, 3888, 4043, 3898, 4522, 4774, 8162, 7692,
7195, 6938, 6410, 6041, 5768, 8543, 8755, 8271, 7939, 7052,
6395, 5980, 8122, 9124, 10307, 10506, 8607, 7326, 6615, 8280,
9333, 11303, 13434, 10871, 8458, 6993, 7694, 8293, 9598, 10743,
9695, 7938, 6372, 6528, 6814, 7356, 7580, 7401, 7089, 6270,
5683, 5811, 6181, 6397, 5909, 5726, 5781, 43530, 43224, 40874,
39942, 38939, 38032, 36751, 43545, 50091, 50867, 47721, 45472, 43093,
39536, 44726, 52853, 62406, 66190, 56487, 49107, 44248, 47333, 54066,
66982, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0], device='cuda:0', dtype=torch.int32),act=137566,algo=ConvAlgo.Native
no apex
No Tensorflow
SPCONV_DEBUG_SAVE_PATH not found, you can specify SPCONV_DEBUG_SAVE_PATH as debug data save path to save debug data which can be attached in a issue.
Traceback (most recent call last):
File "SpMiddleResNetFHDLargeKernel_forward.py", line 13, in <module>
ret, multi_scale_voxel_features,_ = center_net(voxel_features, batch_dict, voxel_coords, batch_size, dense_shape)
File "/home/dl-01/HDD/anaconda3/envs/openpcdet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/home/dl-01/HDD/code/FocalsConv/CenterPoint/det3d/models/backbones/scn_largekernel.py", line 290, in forward
x_conv1 = self.conv1(x)
File "/home/dl-01/HDD/anaconda3/envs/openpcdet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/home/dl-01/HDD/anaconda3/envs/openpcdet/lib/python3.8/site-packages/spconv/pytorch/modules.py", line 138, in forward
input = module(input)
File "/home/dl-01/HDD/anaconda3/envs/openpcdet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/home/dl-01/HDD/code/FocalsConv/CenterPoint/det3d/models/backbones/scn_largekernel.py", line 181, in forward
out = self.conv1(x)
File "/home/dl-01/HDD/anaconda3/envs/openpcdet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/home/dl-01/HDD/code/FocalsConv/CenterPoint/det3d/models/backbones/scn_largekernel.py", line 92, in forward
x_conv_block = self.block(x_conv)
File "/home/dl-01/HDD/anaconda3/envs/openpcdet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/home/dl-01/HDD/anaconda3/envs/openpcdet/lib/python3.8/site-packages/spconv/pytorch/conv.py", line 755, in forward
return self._conv_forward(self.training,
File "/home/dl-01/HDD/anaconda3/envs/openpcdet/lib/python3.8/site-packages/spconv/pytorch/conv.py", line 327, in _conv_forward
out_features = Fsp.indice_subm_conv(
File "/home/dl-01/HDD/anaconda3/envs/openpcdet/lib/python3.8/site-packages/torch/cuda/amp/autocast_mode.py", line 118, in decorate_fwd
return fwd(*args, **kwargs)
File "/home/dl-01/HDD/anaconda3/envs/openpcdet/lib/python3.8/site-packages/spconv/pytorch/functional.py", line 327, in forward
raise e
File "/home/dl-01/HDD/anaconda3/envs/openpcdet/lib/python3.8/site-packages/spconv/pytorch/functional.py", line 308, in forward
return ops.indice_conv(features,
File "/home/dl-01/HDD/anaconda3/envs/openpcdet/lib/python3.8/site-packages/spconv/pytorch/ops.py", line 860, in indice_conv
ConvGemmOps.indice_conv(alloc, ext_mm, GEMM_CPP, ALL_WEIGHT_IS_KRSC,
RuntimeError: /tmp/pip-build-env-h7b1kdx5/overlay/lib/python3.8/site-packages/cumm/include/tensorview/tensor.h(770)
stride_valid assert faild. non-contiguous stride can't handled.