IDSS Engine Commons

Coverage Report

File	Stmts	Miss	Cover	Missing
idsse/common
init.py	0	0	100%
aws_utils.py	43	2	2	95%
config.py	69	5	5	93%
http_utils.py	40	4	4	90%
json_message.py	24	1	1	96%
log_util.py	41	3	3	93%
path_builder.py	211	22	22	90%
protocol_utils.py	91	4	4	96%
rabbitmq_rpc.py	110	2	2	98%
rabbitmq_utils.py	226	31	31	86%
utils.py	177	22	22	88%
validate_schema.py	40	0	100%
idsse/common/sci
init.py	0	0	100%
bit_pack.py	85	3	3	96%
geo_image.py	236	17	17	93%
grid_proj.py	132	1	1	99%
netcdf_io.py	47	0	100%
utils.py	10	0	100%
vectaster.py	190	17	17	91%
TOTAL	1772	134	92%

Overview

The idss-engine-commons project is responsible for defining all implicit common dependencies that are used by the apps that make up the IDSS Engine Project distributed system. This repository should be used to house elements that are common across multiple projects to encourage reuse.

Twelve-Factors

The complete twelve-factors methodologies that the IDSS Engine Project adheres to can be found in the umbrella idss-engine repository. The subset of the twelve factors that follows are specifics to this app only.

Logging

To support some standardization and best practices for IDSS Engine, developers should following the logging guide found here

Contributing

Clone this repository.

Create a Python pip environment and activate it:

python3 -m venv .venv && source .venv/bin/activate

Install all Python dependencies based on what is installed in any of the GitHub Actions in .github/workflows, e.g. linter.yml.

pip install <libs_from_.github/workflows/linter.yml>

If you intend to run unit tests, you'll need to clone the idsse-testing GitHub repository, then install it as an "editable" (a.k.a. local) pip package:

pip install -e <path_to_idsse_testing_clone>/python

Formatting

Python code is formatted according to the black style guide, and any Pull Requests will be checked against this styling to ensure the new code has been auto-formatted.

To have black reformat all Python code every time you create a git commit, install the pre-commit library:

pip install pre-commit

And run this to have any Git pre-commit hooks installed for you according to .pre-commit.config.yaml:

pre-commit install

If you prefer, you can install the black library and run it yourself manually (not running on git commit):

pip install black

This will reformat all your Python files for you:

black . --line-length 99

Build, Release, and Run

The subsections below outline how to build the common images within this project. All microservices built with Docker are done within the idss-engine/build directory.

Recommended Tags development :dev stable release :release ie. :alder targeted environment :aws

Python Base Image (Basic)

---

Any python based microservice within IDSS Engine should use the following image as its base

Build

From the IDSS Engine project root directory idss-engine/build/<env>/<arch>/:

$ docker-compose build python_base

Image Name idsse/commons/python-base:<tag>

Python Base Image (Scientific)

---

Any python based microservice within IDSS Engine that requires python scientific packages (numpy, netcdf, etc) should use the following image as its base. This base image also includes common python utilities that other services can utilize. For specifics on what these utilities are, see the README within the idsse_commons directory.

Build

From the IDSS Engine project root directory idss-engine/build/<env>/<arch>/:

$ docker-compose build python_base_sci

Image Name idsse/commons/python-base-sci:<tag>

RabbitMQ Server

---

The IDSS Engine can be deployed with Docker and/or Kubernetes. If developers are deploying locally with Docker, they can create their own RabbitMQ service as follows:

Developer Note if deploying in a kubernetes cluster, it is not necessary to build this image as it will be done via a Helm Chart

Build

From the IDSS Engine project root directory idss-engine/build/<env>/<arch>/:

$ docker-compose build rabbit_mq

Development Image Name idss.engine.commons.rabbitmq.server:<tag>

Packaged/Deployed Image Name idsse/commons/rabbitmq:<tag>

Couchdb Server

---

The IDSS Engine can be deployed with Docker and/or Kubernetes. If developers are deploying locally with Docker, they can create their own RabbitMQ service as follows:

Developer Note if deploying in a kubernetes cluster, it is not necessary to build this image as it will be done via a Helm Chart

Build

From the IDSS Engine project root directory idss-engine/build/<env>/<arch>/:

$ docker-compose build couch_db

Development Image Name idss.engine.commons.couchbase.server:<tag>

Packaged/Deployed Image Name idsse/commons/couchdb:<tag>

Running IDSS Engine

See the Build, Release, Run section within the umbrella project documentation: [idss-engine](https://github.com/NOAA-GSL/idss-engine/blob/main/README.md#running-idss-engine)