The Camera Traps application is both a simulator and IoT device software for utilizing machine learning on the edge in field research. The first implementation specializes in applying computer vision (detection and classification) to wildlife images for animal ecology studies. Two operational modes are supported: "simulation" mode and "demo" mode. When executed in simulation mode, the software serves as a test bed for studying ML models, protocols and techniques that optimize storage, execution time, power and accuracy. Simulation mode accepts two different input types: (1) an input dataset of images to act as the images that would be generated an IoT camera device or (2) an input video file that would be captured by a camera which is then processed by an image detecting plugin that saves frames with motion in them; it uses these images to drive the simulation.
Conversely, when run in "demo" mode, the application serves as software that can be deployed onto actual, Linux-based camera trap devices in the wild. In this case, the Camera Traps software relies on a digital camera accessible over a Linux device mount (the default /dev/video0 location can be re-configured), and it drives the camera directly using the Linux Motion activation software, which comes bundled with the as a plugin with Camera Traps. It includes a detection reporter plugin and MQTT component which coordinate to communicate in real time when a configurable object of interest has been detected (up to a configurable confidence threshold). As a proof of concept of the capabilities of the software, we are producing a demo integration with drone software developed by the Stewart Lab at OSU which enables the Camera Traps software to communicate over a local network to a nearby drone whenever an object of interest is detected.
- Software
- CI4AI
- Animal Ecology
The actual camera-traps software consists of a set of tightly integrated plugins running on the same IoT device as separate containers (and thus, separate OS processes).
Camera-traps uses the event-engine library to implement its plugin architecture and event-driven communication. The engine uses zmq sockets to deliver events between senders and the subscribers interested in specific events.
The event-engine supports internal and external plugins. Internal plugins are Rust plugins delivered with camera-traps and run in the camera-traps process. External plugins are configured by camera-traps to run outside the camera-traps process and use a TCP port to send and receive events. By using TCP, external plugins can be written in any language that supports the flatbuffers wire protocol.
The camera-traps application requires configuration through environment variables or configuration files. When launching the application from a releases subdirectory, the specific release's config directory will contain the default configuration files for running a short simulation test.
In general, plugins can also depend on their own environment variables and/or configuration files, and the same is true of test programs. The releases directory contains docker-compose files that use default configurations, which can serve as a template for production environment configuration.
| Target | Environment Variable | Default File | Notes |
|---|---|---|---|
| camera-traps application | TRAPS_CONFIG_FILE | ~/traps.toml | Can be 1st command line parameter |
| image_gen_plugin | /input.json | ||
| video_generating_plugin | TRAPS_VIDEO_OUTPUT_PATH | ||
| image_detecting_plugin | /etc/motion/motion.conf | ||
| detection_reporter_plugin | TRAPS_DETECTION_REPORTER_* | /traps-detection.toml | |
| image_store_plugin | TRAPS_IMAGE_STORE_FILE | ~/traps-image-store.toml | |
| power_measure_plugin | TRAPS_POWER_LOG_PATH | ~/logs | |
| oracle_monitor_plugin | TRAPS_ORACLE_OUTPUT_PATH | ~/output | |
| integration tests | TRAPS_INTEGRATION_CONFIG_FILE | ~/traps-integration.toml | |
| logger | TRAPS_LOG4RS_CONFIG_FILE | resources/log4rs.yml | Packaged with application |
The external python plugins run in their own processes and do not currently use environment variables.
The camera-traps application uses log4rs as its log manager. The log settings in resources/log4rs.yml source code will be used unless overridden by assigning a log4rs.yml configuration filepath to the TRAPS_LOG4RS_CONFIG_FILE environment variable. To maximize logging, set root level to trace in the effective log4rs.yml file. Also, include the observer_plugin in the internal plugins list in the effective traps.toml file.
Camera-traps uses a TOML file to configure the internal and external plugins it loads. Internal plugins are registered with the event-engine by simply specfying their names since their runtime characteristics are compiled into the application. External plugins, on the other hand, require more detailed information in order to be registered. Here is the example resources/traps.toml file content:
# This is the camera-traps application configuration file for versions 0.x.y of the application.
# It assumes the use of containers and docker-compose as the deployment mechanism.
title = "Camera-Traps Application Configuration v0.3.2"# The event engine's publish and subscribe port used to create the event_engine::App instance.
publish_port = 5559 subscribe_port = 5560# An absolute path to the image directory is required but a file name prefix is optional.
# If present the prefix is preprended to generated image file names. This is the directory
# into which the image_recv_plugin writes incoming images and the image_store_plugin may
# delete images or output the scores for images.
images_output_dir = "/root/camera-traps/images"
# image_file_prefix = ""# The container for both internal and external plugins. Internal plugins are written in rust
# and compiled into the camera-traps application. External plugins are usually written in
# python but can be written in any language. External plugins run in their own processes
# and communicate via tcp or ipc.
[plugins] # Uncomment the internal plugins loaded when the camera-traps application starts.
internal = [
# "image_gen_plugin",
"image_recv_plugin",
# "image_score_plugin",
"image_store_plugin",
# "observer_plugin"
]# Configure each of the active internal plugins with the image processing action they should
# take when new work is received. If no action is specified for a plugin, its no-op action
# is used by default.
internal_actions = [
"image_recv_write_file_action",
"image_store_file_action"
]# External plugins require more configuration information than internal plugins.
# Each plugin must subscribe to PluginTerminateEvent.
#
# Note that each plugin must specify the external port to use in TWO PLACES: here as well as
# in the docker-compose.yml file. If external_port changes here, it must ALSO be changed in the
# docker-compose.yml file.
[[plugins.external]]
plugin_name = "ext_image_gen_plugin"
id = "d3266646-41ec-11ed-a96f-5391348bab46"
external_port = 6000
subscriptions = [
"PluginTerminateEvent"
]
[[plugins.external]]
plugin_name = "ext_image_score_plugin"
id = "d6e8e42a-41ec-11ed-a36f-a3dcc1cc761a"
external_port = 6001
subscriptions = [
"ImageReceivedEvent",
"PluginTerminateEvent"
]
[[plugins.external]]
plugin_name = "ext_power_monitor_plugin"
id = "4a0fca25-1935-472a-8674-58f22c3a32b3"
external_port = 6010
subscriptions = [
"MonitorPowerStartEvent",
"MonitorPowerStopEvent",
"PluginTerminateEvent"
]
[[plugins.external]]
plugin_name = "ext_power_control_plugin"
id = "a59621f2-4db6-4892-bda1-59ecb7ff24ae"
external_port = 6011
subscriptions = [
"PluginTerminateEvent"
]
[[plugins.external]]
plugin_name = "ext_oracle_monitor_plugin"
id = "6e153711-9823-4ee6-b608-58e2e801db51"
external_port = 6011
subscriptions = [
"ImageScoredEvent",
"ImageStoredEvent",
"ImageDeletedEvent",
"PluginTerminateEvent"
]
Every plugin must subscribe to the PluginTerminateEvent, which upon receipt causes the plugin to stop. Subscriptions are statically defined in internal plugin code and explicitly configured for external plugins. External plugins also provide their predetermined UUIDs and external TCP ports.
Camera-traps looks for its configuration file using these methods in the order shown:
- The environment variable $TRAPS_CONFIG_FILE.
- The first command line argument.
- $HOME/traps.toml
The first file it finds it uses. If no configuration file is found the program aborts.
The names listed in the internal list are the rust plugin file names. These plugins run as separate threads in the camera-traps process. The internal_actions list contains the file names that implement the different algorithms or actions associated with each internal plugin.
A naming convention is used to associate actions with their plugins: An action name starts with its plugin name minus the trailing "plugin" part, followed by an action identifier part, and ends with "_action". Each plugin has a no-op action that causes it to take no action other than, possibly, generating the next event in the pipeline. For example, image_gen_noop_action is associated with the image_gen_plugin.
Internal plugins for which no corresponding action is specified are assigned their no-op plugin by default.
When image_recv_write_file_action is specifed, the image_recv_plugin uses the image_dir and image_file_prefix parameters to manage files. The image_dir is the directory into which image files are placed. Image file names are constructed from the information received in a NewImageEvent and have this format:
<image_file_prefix><image_uuid>.<image_format> The image_uuid and image_format are from the NewImageEvent. The image_file_prefix can be the empty string and the image_format is always lowercased when used in the file name.
To quickly start the application under Docker using docker-compose, follow these steps:
- ./installer/install.sh $PWD ./installer/example_input.yml
- cd test
- docker-compose up
- docker-compose down
The Image Scoring plugin can make use of NVIDIA GPUs to improve the performance of object detection and classification with some ML models. In order to make use of NVIDIA GPUs in the Camera Traps application, the following steps must be taken:
- Ensure the NVIDIA drivers are installed natively on the machine. For example, on Ubuntu LTS, follow the instructions in Section 3.1 here. Be sure to reboot your machine after adding the keyring and installing the drivers. You can check to see if the drivers are installed properly and communicating with the hardware by running the following command:
nvidia-smi
- Install the NVIDIA Container Toolkit and configure the Docker Runtime. See the instructions here. Make sure to restart Docker after installing and configuring the toolkit. To check if the toolkit and Docker are installed and configured correctly, run the following:
docker run --gpus=all --rm -it ubuntu nvidia-smi
The output should be similar to the output from Step 1.
- Update the Camera Traps Compose File to Use GPUs. Starting with release 0.4, the installer includes options for making NVIDIA GPUs available to both the Image Scoring and Power Monitoring plugins. See the Installer README for more details.
If running configuring a run directory using the installer with mode: demo,
the docker-compose.yml file will be configured with to send detection events
to an MQTT broker. By default, it is expected that the broker is already
running on your local machine, although that can be changed in your configure
YAML file.
On linux, you can install and run with
sudo apt-get install mosquitto mosquitto-clients -y
sudo systemctl start mosquitto
On MacOS, it can be installed and run with HomeBrew with:
brew install mosquitto
brew services run mosquitto
To verify that detection events are flowing to the MQTT broker, you should be able to run
mosquitto_sub -t cameratrap/images -t cameratrap/events
on the same machine as the broker and see the events being printed to stdout.
Note: you may need to add the following mosquitto config file in order for the docker image to able to publish to the mqtt broker:
cat /etc/mosquitto/conf.d/my.conf
listener 1883 0.0.0.0
allow_anonymous true
In-memory representations of events are translated into flatbuffer binary streams plus a leading two byte sequence that identifies the event type. These statically defined byte sequences are specified in the events.rs source file and repeated here for convenience.
Each event is assigned a binary prefix that zqm uses to route incoming binary streams to all of the event's subscribers.
pub const NEW_IMAGE_PREFIX: [u8; 2] = [0x01, 0x00];<br>
pub const IMAGE_RECEIVED_PREFIX: [u8; 2] = [0x02, 0x00];<br>
pub const IMAGE_SCORED_PREFIX: [u8; 2] = [0x03, 0x00];<br>
pub const IMAGE_STORED_PREFIX: [u8; 2] = [0x04, 0x00];<br>
pub const IMAGE_DELETED_PREFIX: [u8; 2] = [0x05, 0x00];<br>
pub const PLUGIN_STARTED_PREFIX: [u8; 2] = [0x10, 0x00];<br>
pub const PLUGIN_TERMINATING_PREFIX: [u8; 2] = [0x11, 0x00];<br>
pub const PLUGIN_TERMINATE_PREFIX: [u8; 2] = [0x12, 0x00];<br>
pub const MONITOR_POWER_START_PREFIX: [u8; 2] = [0x20, 0x00];<br>
pub const MONITOR_POWER_STOP_PREFIX: [u8; 2] = [0x21, 0x00];<br>
Each event sent or received begins with its two byte prefix followed by its serialized form as defined in the camera-traps flatbuffer definition file (events.fbs). The following section describes how to generate Rust source code from this definition file, a similar process can be used for any language supported by flatbuffers.
Flatbuffers info: https://google.github.io/flatbuffers/
The flatbuffers messages schema is defined in the resources/events.fsb file. To change the message formats do the following:
- Edit the
resources/events.fsbfile with your changes. - From the camera-traps directory, regenerate the
events_generated.rscode with the command:
$ flatc --rust -o src resources/events.fbs
- (Optional) Add the following line to the top of the
src/events_generated.rsfile so that clippy warnings are suppressed:
// this line added to keep clippy happy
#![allow(clippy::all)]
Each plugin is required to conform to the following conventions:
- Register for the PluginTerminateEvent.
- Send a PluginStartedEvent when it begins executing.
- Send a PluginTerminatingEvent when it shuts down.
The PluginStartedEvent advertises a plugin's name and uuid when it starts. When a plugin receives a PluginTerminateEvent, it checks if the event's target_plugin_name matches its name or the wildcard name (*). If either is true, then the plugin is expected to gracefully terminate. The plugin is also expected to gracefully terminate if the event's target_plugin_uuid matches the plugin's uuid. Part of plugin termination is for it to send a PluginTerminatingEvent to advertise that it's shutting down, whether in response to a PluginTerminateEvent or for any other reason.
The instructions in this section assume Docker (and docker-compose) are installed, as well as Rust, cargo and make.
From the top-level camera-traps directory, issue the following command to build the application's Docker images:
make build See Makefile for details. Use the installer install script to create a run directory. See the installer README for more details. Then, navigate to the new run directory. Issue the following command to run the application, including the external plugins for which it's configured:
docker-compose up See docker-compose.yaml for details. From the same release directory, issue the following command to stop the application:
docker-compose down
If you're just interested in building the Rust, issue cargo build from the top-level camera-traps directory. Alternatively, issue cargo run to build and run it. External plugins are not started using this approach. The internal plugins and their actions are configured using a traps.toml file, as discussed above.
The camera-traps/tests directory contains integration_tests.rs program. The integration test program runs as an external plugin configured via a traps.toml file as shown above. See the top-level comments in the source code for details.
This section addresses two questions:
- Why would I want to create a plugin?
- What kind of plugin should I create?
One would want to create their own plugin if they wanted to read or write events and perform some new action that isn't currently implemented. If an existing plugin doesn't do what you want, you have the option of modifying that plugin or creating another plugin that acts on the same events and does what you need.
For example, the image_gen_plugin injects new images into the event stream, the image_recv_plugin writes new images to file, etc. The observer_plugin is one that subscribes to all events and logs them for debugging purposes. Most of the time we don't run the observer_plugin, but if we want extended logging we just include it to run in the traps.toml file. In this case, having a separate plugin from which we can customize the logging of all events is more convenient then adding that logging capability to each existing plugin.
Another reason for introducing a new plugin would be to also service new events. As the application evolves new capabilities might require new events. This occurred as we develop support for power monitoring, which introduces 2 new events and a plugin to handle them.
When implementing a plugin the choice between internal and external is often technology driven. Do we want to write a plugin in Rust and compile it into the application (internal) or do we want to write it in some other language and start it up in its own container (external)? Considerations as to which approach to take include performance, resource usage, and availability of domain-specific libraries.
When development on a new release begins, create a new branch. If you would to test your changes, merge into the dev branch. This will trigger the building of docker images with the latest tag and a suite of tests. When development completes and the final version of the release's images are pushed to docker hub, we tag those images with the release number.
To be able to rebuild a release at anytime, we also tag the release's source code in github. The tag is the same as the release version number. Once confident that the tagged code is stable, release tags can be protected using github tag protection.
This work has been funded by grants from the National Science Foundation, including the ICICLE AI Institute (OAC 2112606) and Tapis (OAC 1931439).