|
| 1 | +# Neural Networks |
| 2 | + |
| 3 | +<Badge type="tip" text="main (planned for: PX4 v1.17)" /> <Badge type="warning" text="Experimental" /> |
| 4 | + |
| 5 | +:::warning |
| 6 | +This is an experimental module. |
| 7 | +Use at your own risk. |
| 8 | +::: |
| 9 | + |
| 10 | +The Multicopter Neural Network (NN) module ([mc_nn_control](../modules/modules_controller.md#mc_nn_control)) is an example module that allows you to experiment with using a pre-trained neural network on PX4. |
| 11 | +It might be used, for example, to experiment with controllers for non-traditional drone morphologies, computer vision tasks, and so on. |
| 12 | + |
| 13 | +The module integrates a pre-trained neural network based on the [TensorFlow Lite Micro (TFLM)](../advanced/tflm.md) module. |
| 14 | +The module is trained for the [X500 V2](../frames_multicopter/holybro_x500v2_pixhawk6c.md) multicopter frame. |
| 15 | +While the controller is fairly robust, and might work on other platforms, we recommend [Training your own Network](#training-your-own-network) if you use a different vehicle. |
| 16 | +Note that after training the network you will need to update and rebuild PX4. |
| 17 | + |
| 18 | +TLFM is a mature inference library intended for use on embedded devices. |
| 19 | +It has support for several architectures, so there is a high likelihood that you can build it for the board you want to use. |
| 20 | +If not, there are other possible NN frameworks, such as [Eigen](https://eigen.tuxfamily.org/index.php?title=Main_Page) and [Executorch](https://pytorch.org/executorch-overview). |
| 21 | + |
| 22 | +This document explains how you can include the module in your PX4 build, and provides a broad overview of how it works. |
| 23 | +The other documents in the section provide more information about the integration, allowing you to replace the NN with a version trained on different data, or even to replace the TLFM library altogether. |
| 24 | + |
| 25 | +If you are looking for more resources to learn about the module, a website has been created with links to a youtube video and a workshop paper. A full master's thesis will be added later. [A Neural Network Mode for PX4 on Embedded Flight Controllers](https://ntnu-arl.github.io/px4-nns/). |
| 26 | + |
| 27 | +## Neural Network PX4 Firmware |
| 28 | + |
| 29 | +The module has been tested on a number of configurations, which can be build locally using the commands: |
| 30 | + |
| 31 | +```sh |
| 32 | +make px4_sitl_neural |
| 33 | +``` |
| 34 | + |
| 35 | +```sh |
| 36 | +make px4_fmu-v6c_neural |
| 37 | +``` |
| 38 | + |
| 39 | +```sh |
| 40 | +make mro_pixracerpro_neural |
| 41 | +``` |
| 42 | + |
| 43 | +You can add the module to other board configurations by modifying their `default.px4board file` configuration to include these lines: |
| 44 | + |
| 45 | +```sh |
| 46 | +CONFIG_LIB_TFLM=y |
| 47 | +CONFIG_MODULES_MC_NN_CONTROL=y |
| 48 | +``` |
| 49 | + |
| 50 | +:::tip |
| 51 | +The `mc_nn_control` module takes up roughly 50KB, and many of the `default.px4board file` are already close to filling all the flash on their boards. To make room for the neural control module you can remove the include statements for other modules, such as FW, rover, VTOL and UUV. |
| 52 | +::: |
| 53 | + |
| 54 | +## Example Module Overview |
| 55 | + |
| 56 | +The example module replaces the entire controller structure as well as the control allocator, as shown in the diagram below: |
| 57 | + |
| 58 | + |
| 59 | + |
| 60 | +In the [controller diagram](../flight_stack/controller_diagrams.md) you can see the [uORB message](../middleware/uorb.md) flow. |
| 61 | +We hook into this flow by subscribing to messages at particular points, using our neural network to calculate outputs, and then publishing them into the next point in the flow. |
| 62 | +We also need to stop the module publishing the topic to be replaced, which is covered in [Neural Network Module: System Integration](nn_module_utilities.md) |
| 63 | + |
| 64 | +### Вхід |
| 65 | + |
| 66 | +The input can be changed to whatever you want. |
| 67 | +Set up the input you want to use during training and then provide the same input in PX4. |
| 68 | +In the Neural Control module the input is an array of 15 numbers, and consists of these values in this order: |
| 69 | + |
| 70 | +- [3] Local position error. (goal position - current position) |
| 71 | +- [6] The first 2 rows of a 3 dimensional rotation matrix. |
| 72 | +- [3] Linear velocity |
| 73 | +- [3] Angular velocity |
| 74 | + |
| 75 | +All the input values are collected from uORB topics and transformed into the correct representation in the `PopulateInputTensor()` function. |
| 76 | +PX4 uses the NED frame representation, while the Aerial Gym Simulator, in which the NN was trained, uses the ENU representation. |
| 77 | +Therefore two rotation matrices are created in the function and all the inputs are transformed from the NED representation to the ENU one. |
| 78 | + |
| 79 | + |
| 80 | + |
| 81 | +ENU and NED are just rotation representations, the translational difference is only there so both can be seen in the same figure. |
| 82 | + |
| 83 | +### Output |
| 84 | + |
| 85 | +The output consists of 4 values, the motor forces, one for each motor. |
| 86 | +These are transformed in the `RescaleActions()` function. |
| 87 | +This is done because PX4 expects normalized motor commands while the Aerial Gym Simulator uses physical values. |
| 88 | +So the output from the network needs to be normalized before they can be sent to the motors in PX4. |
| 89 | + |
| 90 | +The commands are published to the [ActuatorMotors](../msg_docs/ActuatorMotors.md) topic. |
| 91 | +The publishing is handled in `PublishOutput(float* command_actions)` function. |
| 92 | + |
| 93 | +:::tip |
| 94 | +If the neural control mode is too aggressive or unresponsive the [MC_NN_THRST_COEF](../advanced_config/parameter_reference.md#MC_NN_THRST_COEF) parameter can be tuned. |
| 95 | +Decrease it for more thrust. |
| 96 | +::: |
| 97 | + |
| 98 | +## Training your own Network |
| 99 | + |
| 100 | +The network is currently trained for the [X500 V2](../frames_multicopter/holybro_x500v2_pixhawk6c.md). |
| 101 | +But the controller is somewhat robust, so it could work directly on other platforms, but performing system identification and training a new network is recommended. |
| 102 | + |
| 103 | +Since the Aerial Gym Simulator is open-source you can download it and train your own networks as long as you have access to an NVIDIA GPU. |
| 104 | +If you want to train a control network optimized for your platform you can follow the instructions in the [Aerial Gym Documentation](https://ntnu-arl.github.io/aerial_gym_simulator/9_sim2real/). |
| 105 | + |
| 106 | +You should do one system identification flight for this and get an approximate inertia matrix for your platform. |
| 107 | +On the `sys-id` flight you need ESC telemetry, you can read more about that in [DSHOT](../peripherals/dshot.md). |
| 108 | + |
| 109 | +Then do the following steps: |
| 110 | + |
| 111 | +- Do a hover flight |
| 112 | +- Read of the logs what RPM is required for the drone to hover. |
| 113 | +- Use the weight of each motor, length of the motor arms, total weight of the platform with battery to calculate an approximate inertia matrix for the platform. |
| 114 | +- Insert these values into the Aerial Gym configuration and train your network. |
| 115 | +- Convert the network as explained in [TFLM](tflm.md). |
0 commit comments