Random weight download failures after some uptime

[jamriska]

We are running a deployment of fofr/any-comfyui-workflow with the minimum instances set to 1. Everything works fine after a fresh boot, but after about a day or two, predictions start to fail with the following error:

Command '['pget', '--log-level', 'warn', '-xf', 'https://weights.replicate.delivery/default/comfy-ui/custom_nodes/comfyui_controlnet_aux/depth_anything_v2_metric_vkitti_vitl.pth.tar', 'ComfyUI/custom_nodes/comfyui_controlnet_aux/ckpts/depth-anything/Depth-Anything-V2-Metric-VKITTI-Large']' returned non-zero exit status 1.

Once this happens, all the following predictions will fail as well, and the only remedy is to reset the deployment.

Note that it does not always fail on this particular .pth. We've seen it fail on these as well:

• comfyui_controlnet_aux/depth_anything_v2_vitg.pth.tar
• comfyui_controlnet_aux/depth_anything_v2_metric_hypersim_vitl.pth.tar
• ComfyUI-Anyline/checkpoints/Anyline/MTEED.pth.tar

This is the workflow.json we use, and here's the full log:

====================================
Inputs uploaded to /tmp/inputs:
video.jpg
current.jpg
====================================
Checking inputs
✅ /tmp/inputs/video.jpg
✅ /tmp/inputs/current.jpg
====================================
Checking weights
✅ MTEED.pth exists in ComfyUI/custom_nodes/ComfyUI-Anyline/checkpoints/Anyline
✅ MTEED.pth exists in ComfyUI/custom_nodes/comfyui_controlnet_aux/ckpts/TheMistoAI/MistoLine/Anyline
✅ depth_anything_v2_vitb.pth exists in ComfyUI/custom_nodes/comfyui_controlnet_aux/ckpts/depth-anything/Depth-Anything-V2-Base
✅ depth_anything_v2_vitl.pth exists in ComfyUI/custom_nodes/comfyui_controlnet_aux/ckpts/depth-anything/Depth-Anything-V2-Large
✅ depth_anything_v2_vits.pth exists in ComfyUI/custom_nodes/comfyui_controlnet_aux/ckpts/depth-anything/Depth-Anything-V2-Small
✅ netG.pth exists in ComfyUI/custom_nodes/comfyui_controlnet_aux/ckpts/lllyasviel/Annotators
✅ depth_anything_v2_vitg.pth exists in ComfyUI/custom_nodes/comfyui_controlnet_aux/ckpts/depth-anything/Depth-Anything-V2-Giant
✅ erika.pth exists in ComfyUI/custom_nodes/comfyui_controlnet_aux/ckpts/lllyasviel/Annotators
✅ sd_xl_base_1.0.safetensors exists in ComfyUI/models/checkpoints
✅ t2i-adapter_diffusers_xl_depth_midas.safetensors exists in ComfyUI/models/controlnet
✅ depth_anything_v2_metric_hypersim_vitl.pth exists in ComfyUI/custom_nodes/comfyui_controlnet_aux/ckpts/depth-anything/Depth-Anything-V2-Metric-Hypersim-Large
⏳ Downloading depth_anything_v2_metric_vkitti_vitl.pth to ComfyUI/custom_nodes/comfyui_controlnet_aux/ckpts/depth-anything/Depth-Anything-V2-Metric-VKITTI-Large
Error: error writing file: error extracting file: open ComfyUI/custom_nodes/comfyui_controlnet_aux/ckpts/depth-anything/Depth-Anything-V2-Metric-VKITTI-Large/depth_anything_v2_metric_vkitti_vitl.pth: no such file or directory
Traceback (most recent call last):
File "/root/.pyenv/versions/3.12.6/lib/python3.12/site-packages/cog/server/worker.py", line 794, in _handle_predict_error
yield
File "/root/.pyenv/versions/3.12.6/lib/python3.12/site-packages/cog/server/worker.py", line 646, in _predict
result = predict(**payload)
^^^^^^^^^^^^^^^^^^
File "/src/predict.py", line 152, in predict
wf = self.comfyUI.load_workflow(workflow_json_content or EXAMPLE_WORKFLOW_JSON)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/src/comfyui.py", line 307, in load_workflow
self.handle_weights(wf)
File "/src/comfyui.py", line 130, in handle_weights
self.weights_downloader.download_weights(weight)
File "/src/weights_downloader.py", line 44, in download_weights
self.download_if_not_exists(
File "/src/weights_downloader.py", line 65, in download_if_not_exists
WeightsDownloader.download(weight_str, url, dest)
File "/src/weights_downloader.py", line 76, in download
subprocess.check_call(
File "/root/.pyenv/versions/3.12.6/lib/python3.12/subprocess.py", line 413, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['pget', '--log-level', 'warn', '-xf', 'https://weights.replicate.delivery/default/comfy-ui/custom_nodes/comfyui_controlnet_aux/depth_anything_v2_metric_vkitti_vitl.pth.tar', 'ComfyUI/custom_nodes/comfyui_controlnet_aux/ckpts/depth-anything/Depth-Anything-V2-Metric-VKITTI-Large']' returned non-zero exit status 1.

Is there a fix or known workaround for this issue?

Thank you!