Run error on any given input (tested custom, Free, and Tanks) #14
Description
Getting an odd error where my render script seems to fail every time. The issue almost always seems to relate to not finding the trained camera file FileNotFoundError: [Errno 2] No such file or directory: 'outputs/free/lab/baseline/2025-08-28_02:32:07/cameras_all_train.json'. There was also other errors with finding the "outputs/custom/datetime/point_cloud" folder, which I checked and had not been created yet. Here's the full error message from around the failure, which seems to include some kind of CUDA kernel failure as well? I worry this may just be a dependency issue, I was initally getting a version mismatch between CUDA and Pytorch-CUDA when I installed using the tutorial in the README.
Error log:
>> Confidence-aware Ranking... [28/08 02:35:41]
Sorted indices: [0 2 1] [28/08 02:35:41]
Sorted average confidence scores: [9.3239765 8.809109 7.406957 ] [28/08 02:35:41]
>> Calculate the co-visibility mask... [28/08 02:35:41]
/mnt/c/Users/aliha/Desktop/LongSplat/submodules/mast3r/dust3r/dust3r/inference.py:44: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
with torch.cuda.amp.autocast(enabled=bool(use_amp)):
/mnt/c/Users/aliha/Desktop/LongSplat/submodules/mast3r/dust3r/dust3r/model.py:206: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
with torch.cuda.amp.autocast(enabled=False):
/mnt/c/Users/aliha/Desktop/LongSplat/submodules/mast3r/dust3r/dust3r/inference.py:48: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
with torch.cuda.amp.autocast(enabled=False):
self.cameras_extent: 10.0 [28/08 02:35:42]
Initial voxel_size: 0.1 [28/08 02:35:42]
Number of points at initialisation : 1809 [28/08 02:35:43]
... loading model from ./submodules/mast3r/checkpoints/MASt3R_ViTLarge_BaseDecoder_512_catmlpdpt_metric.pth [28/08 02:35:43]
instantiating : AsymmetricMASt3R(enc_depth=24, dec_depth=12, enc_embed_dim=1024, dec_embed_dim=768, enc_num_heads=16, dec_num_heads=12, pos_embed='RoPE100',img_size=(512, 512), head_type='catmlp+dpt', output_mode='pts3d+desc24', depth_mode=('exp', -inf, inf), conf_mode=('exp', 1, inf), patch_embed_cls='PatchEmbedDust3R', two_confs=True, desc_conf_mode=('exp', 0, inf), landscape_only=False) [28/08 02:35:52]
<All keys matched successfully> [28/08 02:35:56]
Init Optimization: 0%| | 0/3000 [00:00<?, ?it/s]Traceback (most recent call last):
File "/mnt/c/Users/aliha/Desktop/LongSplat/train.py", line 811, in <module>
training(lp.extract(args), op.extract(args), pp.extract(args), dataset, args.debug_from, logger)
File "/mnt/c/Users/aliha/Desktop/LongSplat/train.py", line 123, in training
ssim_loss = (1 - fused_ssim(image.unsqueeze(0), gt_image.unsqueeze(0)))
File "/home/aliha/anaconda3/envs/longsplat/lib/python3.10/site-packages/fused_ssim/__init__.py", line 42, in fused_ssim
return map.mean()
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Init Optimization: 0%| | 0/3000 [00:00<?, ?it/s]
0
/mnt/c/Users/aliha/Desktop/LongSplat/submodules/mast3r/dust3r/dust3r/cloud_opt/base_opt.py:275: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
@torch.cuda.amp.autocast(enabled=False)
Looking for config file in outputs/free/road/baseline/2025-08-28_02:34:46/cfg_args
Config file found: outputs/free/road/baseline/2025-08-28_02:34:46/cfg_args
Rendering outputs/free/road/baseline/2025-08-28_02:34:46
Loading trained model at iteration None [28/08 02:36:02]
Loading Cameras with Pose [28/08 02:36:02]
Successfully loaded GT poses from /mnt/c/Users/aliha/Desktop/LongSplat/data/free/road/sparse/0/images.bin [28/08 02:36:03]
Traceback (most recent call last):
File "/mnt/c/Users/aliha/Desktop/LongSplat/render.py", line 269, in <module>
render_sets(model.extract(args), args.iteration, pipeline.extract(args), args.skip_train, args.skip_test)
File "/mnt/c/Users/aliha/Desktop/LongSplat/render.py", line 224, in render_sets
scene = Scene(dataset, gaussians, load_iteration=iteration)
File "/mnt/c/Users/aliha/Desktop/LongSplat/scene/__init__.py", line 50, in __init__
scene_info = sceneLoadTypeCallbacks["Eval"](args.source_path, args.model_path, args.images, args.eval)
File "/mnt/c/Users/aliha/Desktop/LongSplat/scene/dataset_readers.py", line 270, in readEvalSceneInfo
cam_infos_train_unsorted = readJsonCameras(json.load(open(camera_train_path)), cam_extrinsics, os.path.join(path, reading_dir))
FileNotFoundError: [Errno 2] No such file or directory: 'outputs/free/road/baseline/2025-08-28_02:34:46/cameras_all_train.json'
0