Redis and rate-limiting plugin added (#6)

Author: ikethecoder

.github/workflows/gh-pages.yml

@@ -0,0 +1,18 @@
+name: Build and Deploy
+on:
+  push:
+    branches: [ dev ]
+jobs:
+  build-and-deploy:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout 🛎️
+        uses: actions/checkout@v2.3.1 # If you're using actions/checkout@v2 you must set persist-credentials to false in most cases for the deployment to work correctly.
+        with:
+          persist-credentials: false
+
+      - name: Deploy
+        uses: peaceiris/actions-gh-pages@v3
+        with:
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          publish_dir: ./docs/static

README.md

@@ -1,16 +1,15 @@
 # GWA APIs
 
-For self-service of the APIs, a set of microservices are used to coordinate updates by the API Owners.
+For self-service of APIs, a set of microservices are used to coordinate updates by the providers of APIs.
 
-* Gateway API : Provides a way for API Owners to update their Kong configuration (and internally the OCP Edge Router)
-* Authz API : Provides a way for API Owners to update Keycloak for access to the API Services Portal
-* Catalog API : Providers a way for API Owners to update the API details in the BC Data Catalog
+* `Gateway` : Provides a way for API Owners to update their Kong configuration (and internally the OCP Edge Router)
+* `Authz` : Provides a way for API Owners to update Keycloak for access to functionality on the API Services Portal
+* `Catalog` : Provides a way for API Owners to update the API details in the BC Data Catalog
 
 All APIs are protected by an OIDC JWT Token with the following claims:
 
-* `aud` : https://gwa-qwzrwc-dev.pathfinder.gov.bc.ca/
-* `namespace` : Identifies the namespace that the APIs belong to, used to scope what changes are synced with Kong
-* `scope` : `manage:config`
+* `aud` : `gwa`
+* `namespace` : Identifies the namespace that the APIs belong to, used to scope what changes are allowed.
 
 **Configuration:**
 
@@ -33,30 +32,9 @@ All APIs are protected by an OIDC JWT Token with the following claims:
 | `KC_PASSWORD`     | Keycloak access for administrative rights to manage groups for namespaces | `xxx`
 | `HOST_TRANSFORM_ENABLED` | For Dev and Test a way to transform the host for working in these environments | `false`
 | `HOST_TRANSFORM_BASE_URL` | For Dev and Test a way to transform the host for working in these environments |
+| `PLUGINS_RATELIMITING_REDIS_PASSWORD` | The Redis credential added to the rate-limiting Kong plugin during publish |
 
-
-## Gateway API
-
-The `Gateway API` has a `dry-run` and `sync` of Kong and OCP configuration.
-
-The token must have a valid scope for managing the config.
-
-
-# Access
-
-```
-scope := permission/resource.access
-
-permission, resource, access := any string (without space, period, slash, or asterisk) | asterisk
-```
-
-permission type is based on single or bulk records.
-
-resource types: GatewayConfig, Catalog
-
-access: read, write
-
-# Flow
+# API Provider Flow
 
 ## 1. Register a new namespace
 
@@ -119,9 +97,23 @@ services:
 
 Run: `gwa new` and follow the prompts.
 
+Example:
+
+```
+gwa new -o sample.yaml https://bcgov.github.io/gwa-api/openapi/simple.yaml
+```
+
+> The current beta version of `gwa new` results in Kong configuration that needs to be edited before it is ready to be applied.
+
+> Make the following edits:
+> * Add a `hosts` list under each `route` with the external URL of your service on the gateway (i.e./ a value that is: `$NAME.api.gov.bc.ca`)
+> * The `service` `url` might need to be edited to equal your upstream URL
+> * Optionally: Add a qualifier to the namespace tags if you are separating your configuration into different pipelines
+
+
 ## 4. Apply gateway configuration
 
-The Swagger console for the `gwa-api` can be used to publish Kong Gateway configuration, or the `gwa-cli` can be used.
+The Swagger console for the `gwa-api` can be used to publish Kong Gateway configuration, or the `gwa Command Line` can be used.
 
 ### Swagger Console
 
@@ -139,7 +131,7 @@ Select a `configFile` file.
 
 Send the request.
 
-### Command Line
+### gwa Command Line
 
 **Install**
 
@@ -175,6 +167,12 @@ gwa init -T --namespace=$NS --client-id=<YOUR SERVICE ACCOUNT ID> --client-secre
 gwa pg sample.yaml 
 ```
 
+If you want to see the expected changes but not actually apply them, you can run:
+
+```
+gwa pg --dry-run sample.yaml
+```
+
 ## 5. Verify routes
 
 In our test environment, the hosts that you defined in the routes get altered; to see the actual hosts, log into the <a href="https://gwa-qwzrwc-test.pathfinder.gov.bc.ca/int" target="_blank">API Services Portal</a> and view the hosts under `Services`.
@@ -188,24 +186,34 @@ ab -n 20 -c 2 https://${NAME}-api-gov-bc-ca.test.189768.xyz/headers
 
 ## 6. View metrics
 
-Go to <a href="https://grafana-qwzrwc-test.pathfinder.gov.bc.ca/" target="_blank">Grafana</a> to view metrics for your configured services.
+The following metrics can be viewed in real-time for the Services that you configure on the Gateway:
+
+* Request Rate : Requests / Second (by Service/Route, by HTTP Status)
+* Latency : Standard deviations measured for latency inside Kong and on the Upstream Service (by Service/Route)
+* Bandwidth : Ingress/egress bandwidth (by Service/Route)
+* Total Requests : In 5 minute windows (by Consumer, by User Agent, by Service, by HTTP Status)
 
+All metrics can be viewed by an arbitrary time window - defaults to `Last 24 Hours`.
+
+Go to <a href="https://grafana-qwzrwc-test.pathfinder.gov.bc.ca/" target="_blank">Grafana</a> to view metrics for your configured services.
 
 ## 7. Grant access to others
 
-The `acl` command is an all-inclusive membership list, so the `--users` should have the full list of members.  Any user that is a member but not in the `--users` list will be removed from the namespace.
+The `acl` command provides a way to update the access for the namespace.  It expects an all-inclusive membership list, so the `--users` should have the full list of members.  Any user that is a member but not in the `--users` list will be removed from the namespace.
 
-For administrative privileges (such as managing Service Accounts), add the usernames to the `--managers` argument.
+For elevated privileges (such as managing Service Accounts), add the usernames to the `--managers` argument.
 
 ```
-gwa acl --managers acope@idir --users acope@idir jjones@idir
+gwa acl --users acope@idir jjones@idir --managers acope@idir
 ```
 
+The result will show the ACL changes.  The Add/Delete counts represent the membership changes of registered users.  The Missing count represents the users that will automatically be added to the namespace once they have logged into the `APS Services Portal`.
+
 ## 8. Add to your CI/CD Pipeline
 
 Update your CI/CD pipelines to run the `gwa-cli` to keep your services updated on the gateway.
 
-### Github Actions
+### Github Actions Example
 
 In the repository that you maintain your CI/CD Pipeline configuration, use the Service Account details from `Step 2` to set up two `Secrets`:
 

docs/static/openapi/simple.yaml

@@ -0,0 +1,20 @@
+openapi: 3.0.0
+info:
+  title: Sample API
+  description: Optional multiline or single-line description in [CommonMark](http://commonmark.org/help/) or HTML.
+  version: 0.0.1
+servers:
+  - url: https://httpbin.org
+    description: The upstream service
+paths:
+  "/headers":
+    get:
+      summary: Returns the request headers that the upstream receives.
+      description: Optional extended description in CommonMark or HTML.
+      responses:
+        '200':    # status code
+          description: A JSON object of request header name and values
+          content:
+            application/json:
+              schema: 
+                type: object

microservices/gatewayApi/README.md

@@ -26,16 +26,19 @@ hostip=$(ifconfig en0 | awk '$1 == "inet" {print $2}')
 
 docker run -ti --rm \
  -e CONFIG_PATH=/tmp/production.json -e ENVIRONMENT=production \
- -e OIDC_BASE_URL=https://auth-qwzrwc-dev.pathfinder.gov.bc.ca/auth/realms/aps \
+ -e OIDC_BASE_URL=https://auth.cloud/auth/realms/aps \
  -e TOKEN_MATCH_AUD=account \
  -e WORKING_FOLDER=/tmp \
- -e KONG_ADMIN_URL=https://adminapi-qwzrwc-dev.pathfinder.gov.bc.ca  \
- -e KC_SERVER_URL=https://auth-qwzrwc-dev.pathfinder.gov.bc.ca/auth/ \
+ -e KONG_ADMIN_URL=https://adminapi.cloud  \
+ -e KC_SERVER_URL=https://auth.cloud/auth/ \
  -e KC_REALM=aps \
  -e KC_USERNAME=kcadmin \
- -e KC_PASSWORD="SdufuSYnFAANnluWrAH0waHavE9YWdCu" \
+ -e KC_PASSWORD="" \
  -e KC_USER_REALM=master \
  -e KC_CLIENT_ID=admin-cli \
+ -e HOST_TRANSFORM_ENABLED=true \
+ -e HOST_TRANSFORM_BASE_URL=api.cloud \
+ -e PLUGINS_RATELIMITING_REDIS_PASSWORD="" \
  -v `pwd`/_tmp:/ssl \
  -v ~/.kube/config:/root/.kube/config \
  --add-host=docker:$hostip -p 2000:2000 gwa_kong_api

microservices/gatewayApi/entrypoint.sh

@@ -24,6 +24,16 @@ cat > "${CONFIG_PATH:-./config/default.json}" <<EOF
     "hostTransformation": {
         "enabled": ${HOST_TRANSFORM_ENABLED:-false},
         "baseUrl": "${HOST_TRANSFORM_BASE_URL}"
+    },
+    "plugins": {
+        "rate_limiting": {
+            "redis_database": 0,
+            "redis_host": "redis-master",
+            "redis_port": 6379,
+            "redis_password": "${PLUGINS_RATELIMITING_REDIS_PASSWORD}",
+            "redis_timeout": 2000,
+            "policy": "redis"
+        }
     }
 }
 EOF

microservices/gatewayApi/utils/masking.py

@@ -0,0 +1,12 @@
+import config
+
+conf = config.Config().data
+
+def mask (payload):
+    # Need to mask the redis details
+    redis_pswd = conf['plugins']['rate_limiting']['redis_password']
+    redis_host = conf['plugins']['rate_limiting']['redis_host']
+    returned = payload
+    for val in [redis_pswd, redis_host]:
+        returned = returned.replace(val, '****')
+    return returned

microservices/gatewayApi/utils/transforms.py

@@ -0,0 +1,49 @@
+import config
+import logging
+from flask import current_app as app
+
+conf = config.Config().data
+
+# Some plugins will need to be automatically transformed, such as the
+# rate-limiting plugin
+def plugins_transformations (namespace, yaml):
+    traverse_plugins (yaml)
+
+def rate_limiting (plugin):
+    log = app.logger
+    override_config = conf['plugins']['rate_limiting']
+
+    if 'config' not in plugin:
+        plugin['config'] = {}
+
+    plugin_config = plugin['config']
+
+    for k, v in override_config.items():
+        plugin_config[k] = v
+
+    # Add null values to the following if they are not specified
+    for nval in ['second', 'minute', 'hour', 'day', 'month', 'year', 'header_name']:
+        if nval not in plugin_config:
+            plugin_config[nval] = None
+
+    # Add these defaults if not defined
+    defaults = {
+        "enabled": True,
+        "protocols": [
+            "http",
+            "https"
+        ]
+    }
+    for k, v in defaults.items():
+        if k not in plugin:
+            plugin[k] = v
+
+def traverse_plugins (yaml):
+    traversables = ['services', 'routes', 'plugins', 'upstreams', 'consumers', 'certificates']
+    for k in yaml:
+        if k in traversables:
+            for item in yaml[k]:
+                if k == 'plugins':
+                    if item['name'] == 'rate-limiting':
+                        rate_limiting(item)
+                traverse_plugins (item)

microservices/gatewayApi/utils/validators.py

@@ -15,3 +15,4 @@ def host_valid(input_string):
     # regex = re.compile(host_validation_rule)
     # match = regex.match(str(input_string))
     # return bool(match is not None)
+

microservices/gatewayApi/v1/routes/gateway.py

@@ -13,6 +13,9 @@
 from clients.openshift import prepare_apply_routes, prepare_delete_routes, apply_routes, delete_routes
 
 from utils.validators import host_valid
+from utils.transforms import plugins_transformations
+from utils.masking import mask
+
 
 gw = Blueprint('gwa', 'gateway')
 
@@ -60,6 +63,7 @@ def write_config(namespace: str) -> object:
 
             #
             # Enrich the rate-limiting plugin with the appropriate Redis details
+            plugins_transformations (namespace, gw_config)
 
             with open("%s/%s" % (tempFolder, 'config-%02d.yaml' % index), 'w') as file:
                 yaml.dump(gw_config, file)
@@ -109,7 +113,7 @@ def write_config(namespace: str) -> object:
         if deck_run.returncode != 0:
             cleanup (tempFolder)
             log.warn("%s - %s" % (namespace, out.decode('utf-8')))
-            abort(make_response(jsonify(error="Sync Failed.", results=out.decode('utf-8')), 400))
+            abort(make_response(jsonify(error="Sync Failed.", results=mask(out.decode('utf-8'))), 400))
 
         elif cmd == "sync":
             route_count = prepare_apply_routes (namespace, selectTag, tempFolder)
@@ -127,7 +131,7 @@ def write_config(namespace: str) -> object:
         if cmd == 'diff':
             message = "Dry-run.  No changes applied."
 
-        return make_response(jsonify(message=message, results=out.decode('utf-8')))
+        return make_response(jsonify(message=message, results=mask(out.decode('utf-8'))))
     else:
         log.error("Missing input")
         log.error(request.get_data())
@@ -183,7 +187,7 @@ def host_transformation (namespace, yaml):
     transforms = 0
     conf = app.config['hostTransformation']
     if conf['enabled'] is True:
-        if 'service' in yaml:
+        if 'services' in yaml:
             for service in yaml['services']:
                 if 'routes' in service:
                     for route in service['routes']:
@@ -199,7 +203,7 @@ def validate_hosts (yaml):
     log = app.logger
     errors = []
 
-    if 'service' in yaml:
+    if 'services' in yaml:
         for service in yaml['services']:
             if 'routes' in service:
                 for route in service['routes']: