Various changes (#9)

Author: ikethecoder

README.md

@@ -38,32 +38,30 @@ All APIs are protected by an OIDC JWT Token with the following claims:
 
 ## 1. Register a new namespace
 
-A `namespace` represents a collections of Kong Services and Routes that are managed independently.
+A `namespace` represents a collection of Kong Services and Routes that are managed independently.
 
 To create a new namespace, go to the <a href="https://gwa-qwzrwc-test.pathfinder.gov.bc.ca/int" target="_blank">API Services Portal</a>.
 
-After login and selection of an existing namespace, go to the `New Namespace` tab and click the `Create Namespace` button.
+After login (and selection of an existing namespace if you have one already assigned), go to the `New Namespace` tab and click the `Create Namespace` button.
 
-The namespace must be an alphanumeric string between 5 and 10 characters.
+The namespace must be an alphanumeric string between 5 and 15 characters (RegExp reference: `^[a-z][a-z0-9]{4,14}$`).
 
 Logout by clicking your username at the top right of the page.  When you login again, you should be able to select the new namespace from the `API Programme Services` project selector.
 
 ## 2. Generate a Service Account
 
 Go to the `Service Accounts` tab and click the `Create Service Account`.  A new credential will be created - make a note of the `ID` and `Secret`.
 
-With access:
+The credential has the following access:
 * `admin:gateway` : Permission to publish gateway configuration to Kong
 * `admin:acl`     : Permission to update the Access Control List for controlling access to viewing metrics, service configuration and service account management
 * `admin:catalog` : Permission to update BC Data Catalog datasets for describing APIs available for consumption
 
 ## 3. Prepare configuration
 
-The gateway configuration can be hand-crafted or you can use the `gwa` `new` command to walk you through the creation of the config.
+The gateway configuration can be hand-crafted or you can use a command line interface that we developed called `gwa` to convert your Openapi v3 spec to a Kong configuration.
 
-To view a list of available plugins, you can run: `gwa plugins`.
-
-To view examples go [here](/docs/samples/service-plugins).
+### Hand-crafted (recommended if you don't have an Openapi spec)
 
 **Simple Example**
 
@@ -91,9 +89,11 @@ services:
 " > sample.yaml
 ```
 
+To view optional plugin examples go [here](/docs/samples/service-plugins).
+
 > NOTE: If you have separate pipelines for your environments (i.e./ dev, test and prod), you can split your configuration and update the `tags`.  So for example, you can use a tag `ns.$NS.dev` to sync Kong configuration for `dev` Service and Routes only.
 
-**gwa CLI Example**
+### gwa Command Line
 
 Run: `gwa new` and follow the prompts.
 
@@ -115,23 +115,27 @@ gwa new -o sample.yaml https://bcgov.github.io/gwa-api/openapi/simple.yaml
 
 The Swagger console for the `gwa-api` can be used to publish Kong Gateway configuration, or the `gwa Command Line` can be used.
 
-### Swagger Console
+```
+gwa new -o sample.yaml https://bcgov.github.io/gwa-api/openapi/simple.yaml
+```
 
-Go to <a href="https://gwa-api-qwzrwc-test.pathfinder.gov.bc.ca/api/doc" target="_blank">gwa-api Swagger Console</a>.
+> See below for the `gwa` CLI install instructions.
 
-Select the `PUT` `/namespaces/{namespace}/gateway` API.
+> The current beta version of `gwa new` results in Kong configuration that needs to be edited before it is ready to be applied.
 
-The Service Account uses the OAuth2 Client Credentials Grant Flow.  Click the `lock` link on the right and enter in the Service Account credentials that were generated in step #2.
+> Make the following edits:
+> * Add a `hosts` list under each `route` with the external URL of your service on the gateway (i.e./ a value that is: `$NAME.api.gov.bc.ca`)
+> * The `service` `url` might need to be edited to equal your upstream URL
+> * Optionally: Add a qualifier to the namespace tags if you are separating your configuration into different pipelines
 
-For the `Parameter namespace`, enter the namespace that you created in step #1.
 
-Select `dryRun` to `true`.
+## 4. Apply gateway configuration
 
-Select a `configFile` file.
+The Swagger console for the `gwa-api` can be used to publish Kong Gateway configuration, or the `gwa Command Line` can be used.
 
+### gwa Command Line (recommended)
 Send the request.
 
-### gwa Command Line
 
 **Install**
 
@@ -173,6 +177,23 @@ If you want to see the expected changes but not actually apply them, you can run
 gwa pg --dry-run sample.yaml
 ```
 
+### Swagger Console
+
+Go to <a href="https://gwa-api-qwzrwc-test.pathfinder.gov.bc.ca/api/doc" target="_blank">gwa-api Swagger Console</a>.
+
+Select the `PUT` `/namespaces/{namespace}/gateway` API.
+
+The Service Account uses the OAuth2 Client Credentials Grant Flow.  Click the `lock` link on the right and enter in the Service Account credentials that were generated in step #2.
+
+For the `Parameter namespace`, enter the namespace that you created in step #1.
+
+Select `dryRun` to `true`.
+
+Select a `configFile` file.
+
+Send the request.
+
+
 ## 5. Verify routes
 
 In our test environment, the hosts that you defined in the routes get altered; to see the actual hosts, log into the <a href="https://gwa-qwzrwc-test.pathfinder.gov.bc.ca/int" target="_blank">API Services Portal</a> and view the hosts under `Services`.

docs/samples/service-plugins/service-rate-limit.yaml

@@ -11,18 +11,12 @@ services:
       hide_client_headers: false
       limit_by: consumer
       minute: 10
-      policy: cluster
       header_name: null
       second: null
       hour: null
       day: null
       month: null
       year: null
-      redis_database: 0
-      redis_host: null
-      redis_password: null
-      redis_port: 6379
-      redis_timeout: 2000
     enabled: true
     protocols:
     - http

microservices/gatewayApi/clients/kong.py

@@ -1,4 +1,23 @@
+from flask import current_app as app
+import requests
+import urllib.parse
 
 # Access the Kong Admin API for details about the Kong configuration
 #
-# Use the Route Hosts found in Kong to ensure there are no conflicts
+# Use the Route Hosts found in Kong to ensure there are no conflicts
+def get_routes ():
+    return recurse_get_routes ([], "/routes")
+
+def recurse_get_routes (result, url):
+    log = app.logger
+    admin_url = app.config['kongAdminUrl']
+
+    log.debug("%s%s" % (admin_url, url))
+    r = requests.get("%s%s" % (admin_url, url))
+    json =  r.json()
+    data = json['data']
+    result.extend(data)
+
+    if json['next'] is not None:
+        recurse_get_routes (result, json['next'])
+    return result

microservices/gatewayApi/clients/ocp_networksecuritypolicy.py

@@ -0,0 +1,150 @@
+import time
+import os
+import yaml
+from datetime import datetime
+from flask import current_app as app
+from string import Template
+from subprocess import Popen, PIPE, STDOUT
+
+from clients.openshift import kubectl_apply, kubectl_delete
+
+def check_nsp (ns, ocp_ns):
+    log = app.logger
+    b = check_exists ("nsp", ns, ocp_ns)
+    log.debug("[%s] Check? aps-upstream-%s-%s : %s" % (ns, ns, ocp_ns, b))
+    return b
+
+    
+def check_exists (_type, ns, ocp_ns):
+    log = app.logger
+    args = [
+        "kubectl", "get", _type, "aps-upstream-%s-%s" % (ns, ocp_ns)
+    ]
+    run = Popen(args, stdout=PIPE, stderr=STDOUT)
+    out, err = run.communicate()
+    if run.returncode != 0:
+        return False
+    else:
+        return True
+
+def apply_nsp (ns, ocp_ns, rootPath):
+    log = app.logger
+
+    template = Template("""
+kind: NetworkSecurityPolicy
+apiVersion: security.devops.gov.bc.ca/v1alpha1
+metadata:
+  name: aps-upstream-${ns}-${ocp_ns}
+  labels:
+    aps-generated-by: "gwa-cli"
+    aps-published-on: "${fmt_time}"
+    aps-namespace: "${ns}"
+    aps-published-ts: "${timestamp}"
+spec:
+  description: |
+    allow namespace to access the internet
+  source:
+    - - app.kubernetes.io/instance=kong
+  destination:
+    - - $$namespace=${ocp_ns}
+""")
+
+    ts = int(time.time())
+    fmt_time = datetime.now().strftime("%Y.%m-%b.%d")
+
+    out_filename = "%s/nsp.yaml" % rootPath
+
+    with open(out_filename, 'w') as out_file:
+        index = 1
+        log.debug("[%s] NSP aps-upstream-%s-%s" % (ns, ns, ocp_ns))
+        out_file.write(template.substitute(ns=ns, ocp_ns=ocp_ns, timestamp=ts, fmt_time=fmt_time))
+        out_file.write('\n---\n')
+        index = index + 1
+
+    kubectl_apply (out_filename)
+
+def delete_nsp (ns, ocp_ns):
+    log = app.logger
+
+    name = "aps-upstream-%s-%s" % (ns, ocp_ns)
+    
+    kubectl_delete ("nsp", name)
+
+def get_ocp_service_namespaces(rootPath):
+    files_to_ignore = ["deck.yaml", "routes-current.yaml", "routes-deletions.yaml"]
+
+    service_ns_list = []
+
+    for x in os.walk(rootPath):
+        for file in x[2]:
+            if file not in files_to_ignore:
+                full_path = "%s/%s" % (x[0],file)
+
+                stream = open(full_path, 'r')
+                data = yaml.load(stream, Loader=yaml.SafeLoader)
+
+                if 'services' in data:
+                    for service in data['services']:
+                        if 'host' in service and service['host'].endswith('.svc'):
+                            h = service['host']
+                            parts = h.split('.')
+                            ns = parts[len(parts) - 2]
+                            service_ns_list.append(ns)
+
+    service_ns_list = list(set(service_ns_list))
+    return service_ns_list
+
+# def prepare_deletions (ns, ocp_ns, rootPath):
+#     log = app.logger
+
+#     args = [
+#         "kubectl", "get", "nsp", "-l", "aps-namespace=%s" % select_tag, "-o", "json"
+#     ]
+#     run = Popen(args, stdout=PIPE, stderr=PIPE)
+#     out, err = run.communicate()
+#     if run.returncode != 0:
+#         log.error("Failed to get existing routes", out, err)
+#         raise Exception("Failed to get existing routes")
+
+#     current_routes = []
+
+#     existing = json.loads(out)
+#     for route in existing['items']:
+#         current_routes.append(route['metadata']['name'])
+
+#     host_list = get_host_list(rootPath)
+
+#     delete_list = []
+#     for route_name in current_routes:
+#         match = False
+#         for host in host_list:
+#             if route_name == "wild-%s-%s" % (select_tag.replace('.','-'), host):
+#                 match = True
+#         if match == False:
+#             delete_list.append(route_name)
+
+#     template = Template("""
+# apiVersion: route.openshift.io/v1
+# kind: Route
+# metadata:
+#   name: ${name}
+
+# """)
+
+#     ts = int(time.time())
+#     fmt_time = datetime.now().strftime("%Y.%m-%b.%d")
+
+#     out_filename = "%s/routes-deletions.yaml" % rootPath
+
+#     with open(out_filename, 'w') as out_file:
+#         index = 1
+#         for route_name in delete_list:
+#             log.debug("[%s] Route D %03d %s" % (select_tag, index, route_name))
+#             out_file.write(template.substitute(name=route_name))
+#             out_file.write('\n---\n')
+#             index = index + 1
+
+#     if len(delete_list) == 0:
+#         log.debug("[%s] Route D No Deletions Needed" % select_tag)
+
+#     return len(delete_list)

microservices/gatewayApi/clients/openshift.py

@@ -19,14 +19,28 @@ def read_and_indent(full_path, indent):
     return result
 
 def apply_routes (rootPath):
+    kubectl_apply ("%s/routes-current.yaml" % rootPath)
+
+def kubectl_apply (fileName):
+    log = app.logger
+    args = [
+        "kubectl", "apply", "-f", fileName
+    ]
+    run = Popen(args, stdout=PIPE, stderr=STDOUT)
+    out, err = run.communicate()
+    if run.returncode != 0:
+        log.error("Failed to apply", out, err)
+        raise Exception("Failed to apply routes")
+
+def kubectl_delete (type, name):
     log = app.logger
     args = [
-        "kubectl", "apply", "-f", "%s/routes-current.yaml" % rootPath
+        "kubectl", "delete", type, name
     ]
     run = Popen(args, stdout=PIPE, stderr=STDOUT)
     out, err = run.communicate()
     if run.returncode != 0:
-        log.error("Failed to apply routes", out, err)
+        log.error("Failed to delete", out, err)
         raise Exception("Failed to apply routes")
 
 def delete_routes (rootPath):

microservices/gatewayApi/utils/validators.py

@@ -1,7 +1,7 @@
 
 import re
 
-namespace_validation_rule='^[a-z][a-z0-9]{4,14}$'
+namespace_validation_rule='^[a-z][a-z0-9-]{4,14}$'
 
 host_validation_rule='[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'