Workflow

The files linked are on a nchan branch under a forked NRG repository.

Data collect rosbag

Topics

• /imu/data
• /kinect_imu
• /camera/rgb/image_raw/compressed
• /odom

Preprocess rosbag

rosbag_data_recorder.py

Converts rosbag into a pickle file with data:

• Images: Processed images or bird's-eye view (BEV) images.
• IMU Data: Flattened IMU data from both Kinect and Jackal sensors.
• Odometry Data: Position and orientation data from odometry.
• Orientation Data: Orientation data from the Jackal robot.

Train model

train_naturl_representations.py

Converts pickle file into representation clusters:

• Saves the k-means clustering model.
• Saves the k-means clustering labels and sample indices.
• Saves the state dictionaries of the visual and proprioceptive encoders.

train_naturl_cost.py

Maps from representations to raw cost values:

• Saves the PyTorch neural network model
  • Combination of the visual encoder and the cost network
  • Saved as a state dictionary, which includes the parameters (weights and biases) of the neural network.

Deploy the trained model to a robot

amrl spot_autonomy

Used to deploy the learned model to run STERLING on the robot

• Start all launch file
• .jit file
• model_path
• navigation_spot_naturl.lua config file to modify for graph_navigation stack

terrain_evaluator.cc

Navigation.cc

Loads the evaluator for graph navigation:

auto terrain_evaluator = std::make_shared<TerrainEvaluator>();
terrain_evaluator->LoadModel();
evaluator_ = terrain_evaluator;