Improve Navigation2 Launch Reliability? Frequent smoother_server timeout

[slowrunner]

How can I improve the reliability of my nav2 navigation launches?

I have to launch nav2 several times before seeing a successful bringup with global and local costmap ready for an initial pose and navigation goal.

The failed launch always contains the following warning:

[planner_server-3] [WARN] [1740264294.156083284] [planner_server.rclcpp]: failed to send response to /planner_server/change_state (timeout): client will not receive response, at ./src/rmw_response.cpp:153, at ./src/rcl/service.c:400

ROS 2: Jazzy
OS: Ubuntu 24.02
Processor: Raspberry Pi 5 8GB
Platform: Turtlebot4

I am running everything on-board. Rviz2 visualization is remote.

I have submitted an issue on the Turtlebot4 repo with logs and more information:
Frequent Nav2 smoother_server timeout (and no global costmap) on repeated navigation launches #555

Processor average load is less than 100%, with some 1 minute loads in the 4 to 6 range briefly during launch.

When the launch succeeds, navigation proceeds smoothly with no issues with average load between 2 and 3 (50-75% of processor).

[SteveMacenski]

I haven't run into any flaky bringup issues in Nav2 since the Dashing/Eloquent days, so I don't have any experience with this (nor have any filed tickets in Nav2 alluded to it either). This makes me think that perhaps there's an issue with your compute module trying to do too much given you're running on a Raspberry Pi. Most professional use-cases of Nav2 are on Jetson Orins, Intel/AMD x86 mobile CPUs, or similar.

When this happens, is the CPU or memory thrashing at > 90% usage? I don't know what 4-6 and 2-3 range mean.

What happens if you:

• don't use rviz2 remote visualization
• don't use autostart for Nav2, let the ROS 2 system come up, then after its done, then trigger the lifecycle manager to bring up nav2 (reduces discovery flood on startup)
• Tune DDS settings, for example setting up the discovery server for your respective DDS provider

[slowrunner]

This makes me think that perhaps there's an issue with your compute module trying to do too much given you're running on a Raspberry Pi. Most professional use-cases of Nav2 are on Jetson Orins, Intel/AMD x86 mobile CPUs, or similar.

Agree most often not Raspberry Pi 5. My experiments show average load of 50-70% during nav2 planning and navigation with nav2 AMCL localization using 2D LIDAR, including obstacle avoidance and replanning. So the processor is up to the task once everything is up and running.

When this happens, is the CPU or memory thrashing at > 90% usage? I don't know what 4-6 and 2-3 range mean.

Load of 4 on the 4-core Raspberry Pi 5 means all cores are busy. A 1 minute load of 6 on the 4 core RPi 5 processor means there are processes waiting, but starvation starts around 12 (twice as many processes waiting as running).

The load goes over 100% (to 150-175%) for about 60 seconds (meaning about 1/3 of the total processes are waiting for cycles, possibly as much as 1 minute. If timeouts allow for slow bringup, all processes will be getting cycles during operation mode.

What happens if you:

• don't use rviz2 remote visualization
  Interesting - will test that.

• don't use autostart for Nav2, let the ROS 2 system come up, then after its done, then trigger the lifecycle manager to bring up nav2 (reduces discovery flood on startup)

I don't know what autostart is, but I am bringing up localization fully long after the turtlebot4 service was launched, then launching navigation only when localization is fully ready.

• Tune DDS settings, for example setting up the discovery server for your respective DDS provider
  Yes this is a Discovery Server configuration already.

There are a number of configurable navigation component timeouts described in the guide, but some folks have mentioned that some are hard coded. I will have to get deeper into the navigation bringup to see why the planner-server is complaining, and that a "Warn" causes a fatal bringup.

[slowrunner]

There is some work towards exposing the timeouts here turtlebot/turtlebot4#543 (comment)

Will close this particular issue.