Merge pull request #51 from aws-solutions/develop

Update to version v1.7.0

Author: aassadza

CHANGELOG.md

@@ -5,96 +5,116 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.7.0] - 2022-02-14
+
+### Added
+
+- The capability to ingest custom data by uploading files as JSON, XLSX, or CSV files
+
+### Updated
+
+- Use Amazon Kinesis Data Firehose partition projection to store and partition the data by source date (instead of system processing date)
+- Use Amazon Athena dynamic partitioning features to run SQL queries on data stored in S3 bucket
+- AWS CDK version 1.137.0
+- AWS SDK version 2.1067.0
+
+### Removed
+
+- Creating of AWS Glue partitions (replaced with Amazon Athena dynamic partitions)
+
 ## [1.6.1] - 2021-10-26
+
 ### Fixed
--   GitHub [issue #42](https://github.com/aws-solutions/discovering-hot-topics-using-machine-learning/issues/42). To fix the issue, RSS feed ingestion lambda function and SQLs related to the Amazon QuickSight dashboard were updated.
+
+- GitHub [issue #42](https://github.com/aws-solutions/discovering-hot-topics-using-machine-learning/issues/42). To fix the issue, RSS feed ingestion lambda function and SQLs related to the Amazon QuickSight dashboard were updated.
 
 ### Updated
--   AWS CDK version to 1.125.0
--   AWS SDK version to 2.1008.0
+
+- AWS CDK version to 1.125.0
+- AWS SDK version to 2.1008.0
 
 ## [1.6.0] - 2021-09-27
 
 ### Added
 
--   Capability to ingest YouTube comments
+- Capability to ingest YouTube comments
 
 ### Updated
 
--   AWS CDK version to 1.121.0
--   AWS SDK version to 2.991.0
--   Updated Amazon QuickSight analysis and dashboard to reflect the new ingestion source
+- AWS CDK version to 1.121.0
+- AWS SDK version to 2.991.0
+- Updated Amazon QuickSight analysis and dashboard to reflect the new ingestion source
 
 ## [1.5.0] - 2021-07-22
 
 ### Added
 
--   Ingest RSS feeds from over ~3000+ news websites across the world
+- Ingest RSS feeds from over ~3000+ news websites across the world
 
 ### Updated
 
--   AWS CDK version to 1.110.1
--   AWS SDK version to 2.945.0
--   Updated Nodejs Lambda runtimes to use Nodejs 14.x
--   Updated Amazon QuickSight analysis and dashboard to reflect the new ingestion source
--   Updated AWS StepFunction workflows to handle parallel ingestion (tweets from Twitter and RSS feeds from News websites)
+- AWS CDK version to 1.110.1
+- AWS SDK version to 2.945.0
+- Updated Nodejs Lambda runtimes to use Nodejs 14.x
+- Updated Amazon QuickSight analysis and dashboard to reflect the new ingestion source
+- Updated AWS StepFunction workflows to handle parallel ingestion (tweets from Twitter and RSS feeds from News websites)
 
 ### Fixed
 
--   Truncated tweets through merging [GitHub pull request #26](https://github.com/awslabs/discovering-hot-topics-using-machine-learning/pull/26)
+- Truncated tweets through merging [GitHub pull request #26](https://github.com/awslabs/discovering-hot-topics-using-machine-learning/pull/26)
 
 ## [1.4.0] - 2021-02-04
 
 ### Added
 
--   Capability to use geo coordinates when invoking the Twitter API to filter tweets returned by its Search API
--   New visuals and sheets (tabs) on Amazon QuickSight to perform analysis using geo coordinates (when available with tweets)
--   Additional remediation to handle throttling conditions from Twitter v1.1 API calls and push additional information to Amazon CloudWatch Logs that can be used to create alarms or notifications using CloudWatch Metric Filters
+- Capability to use geo coordinates when invoking the Twitter API to filter tweets returned by its Search API
+- New visuals and sheets (tabs) on Amazon QuickSight to perform analysis using geo coordinates (when available with tweets)
+- Additional remediation to handle throttling conditions from Twitter v1.1 API calls and push additional information to Amazon CloudWatch Logs that can be used to create alarms or notifications using CloudWatch Metric Filters
 
 ### Updated
 
--   Switched to AWS Managed KMS keys for AWS Glue Security Configuration
--   AWS CDK version to 1.83.0
--   AWS SDK version to 2.828.0
+- Switched to AWS Managed KMS keys for AWS Glue Security Configuration
+- AWS CDK version to 1.83.0
+- AWS SDK version to 2.828.0
 
 ## [1.3.0] - 2020-11-24
 
 ### Changed
 
--   Implementation to refactor and to reuse the following architecture patterns from [AWS Solutions Constructs](https://aws.amazon.com/solutions/constructs/)
-    -   aws-kinesisfirehose-s3
-    -   aws-kinesisstreams-lambda
-    -   aws-lambda-step-function
+- Implementation to refactor and to reuse the following architecture patterns from [AWS Solutions Constructs](https://aws.amazon.com/solutions/constructs/)
+  - aws-kinesisfirehose-s3
+  - aws-kinesisstreams-lambda
+  - aws-lambda-step-function
 
 ### Updated
 
--   The join condition for Topic Modeling in Amazon QuickSight dataset to provide accurate topic identification for a specific run
--   ID and name generation for Amazon QuickSigh resource to use dynamic value based on the stack name
--   AWS CDK version to 1.73.0
--   AWS SDK version to 2.790.0
+- The join condition for Topic Modeling in Amazon QuickSight dataset to provide accurate topic identification for a specific run
+- ID and name generation for Amazon QuickSigh resource to use dynamic value based on the stack name
+- AWS CDK version to 1.73.0
+- AWS SDK version to 2.790.0
 
 ## [1.2.0] - 2020-10-29
 
 ### Added
 
--   New and simplified interactive Amazon QuickSight dashboard that is now automatically generated through an AWS CloudFormation deployment and that customers can extend to suit their business case
+- New and simplified interactive Amazon QuickSight dashboard that is now automatically generated through an AWS CloudFormation deployment and that customers can extend to suit their business case
 
 ### Updated
 
--   Updated to AWS CDK v1.69.0
--   Consolidate Amazon S3 access Log bucket across the solution. All access log files have a prefix that corresponds to the bucket for which they are generated
+- Updated to AWS CDK v1.69.0
+- Consolidate Amazon S3 access Log bucket across the solution. All access log files have a prefix that corresponds to the bucket for which they are generated
 
 ## [1.1.0] - 2020-09-29
 
 ### Updated
 
--   S3 storage for inference outputs to use Apache Parquet
--   Add partitioning to AWS Glue tables
--   Update to AWS CDK v1.63.0
--   Update to AWS SDK v2.755.0
+- S3 storage for inference outputs to use Apache Parquet
+- Add partitioning to AWS Glue tables
+- Update to AWS CDK v1.63.0
+- Update to AWS SDK v2.755.0
 
 ## [1.0.0] - 2020-08-28
 
 ### Added
 
--   Initial release
+- Initial release

NOTICE.txt

@@ -25,12 +25,15 @@ boto3 - Apache-2.0
 botocore - Apache-2.0
 chai - MIT license
 crhelper - Apache-2.0
+googleapis - Apache-2.0
 jest - MIT license
+jmespath - MIT License
 momentjs - MIT license
 moto - Apache-2.0
 newscatcher - MIT license
 nock - MIT license
 node - MIT license
+openpyx - MIT license
 pytest-cov - MIT license
 pytest - MIT license
 requests - Apache-2.0

README.md

@@ -6,21 +6,21 @@ The solution automates digital asset (text and image) ingestion from twitter, RS
 
 The solution performs the following key features:
 
--   **Performs topic modeling to detect dominant topics**: identifies the terms that collectively form a topic from within customer feedback
--   **Identifies the sentiment of what customers are saying**: uses contextual semantic search to understand the nature of online discussions
--   **Determines if images associated with your brand contain unsafe content**: detects unsafe and negative imagery in content
--   **Helps customers identify insights in near real-time**: you can use a visualization dashboard to better understand context, threats, and opportunities almost instantly
+- **Performs topic modeling to detect dominant topics**: identifies the terms that collectively form a topic from within customer feedback
+- **Identifies the sentiment of what customers are saying**: uses contextual semantic search to understand the nature of online discussions
+- **Determines if images associated with your brand contain unsafe content**: detects unsafe and negative imagery in content
+- **Helps customers identify insights in near real-time**: you can use a visualization dashboard to better understand context, threats, and opportunities almost instantly
 
 This solution deploys an AWS CloudFormation template that supports Twitter, RSS feeds, and YouTube comments as data source options for ingestion, but the solution can be customized to aggregate other social media platforms and internal enterprise systems.
 
 For a detailed solution deployment guide, refer to [Discovering Hot Topics using Machine Learning](https://aws.amazon.com/solutions/implementations/discovering-hot-topics-using-machine-learning)
 
 ## On this Page
 
--   [Architecture Overview](#architecture-overview)
--   [Deployment](#deployment)
--   [Source Code](#source-code)
--   [Creating a custom build](#creating-a-custom-build)
+- [Architecture Overview](#architecture-overview)
+- [Deployment](#deployment)
+- [Source Code](#source-code)
+- [Creating a custom build](#creating-a-custom-build)
 
 ## Architecture Overview
 
@@ -54,13 +54,13 @@ After you deploy the solution, use the included Amazon QuickSight dashboard to v
 
 [AWS CDK Solutions Constructs](https://aws.amazon.com/solutions/constructs/) make it easier to consistently create well-architected applications. All AWS Solutions Constructs are reviewed by AWS and use best practices established by the AWS Well-Architected Framework. This solution uses the following AWS CDK Constructs:
 
--   aws-events-rule-lambda
--   aws-kinesisfirehose-s3
--   aws-kinesisstreams-lambda
--   aws-lambda-dynamodb
--   aws-lambda-s3
--   aws-lambda-step-function
--   aws-sqs-lambda
+- aws-events-rule-lambda
+- aws-kinesisfirehose-s3
+- aws-kinesisstreams-lambda
+- aws-lambda-dynamodb
+- aws-lambda-s3
+- aws-lambda-step-function
+- aws-sqs-lambda
 
 ## Deployment
 
@@ -78,12 +78,12 @@ The solution is deployed using a CloudFormation template with a lambda backed cu
     ├── bin                             [entrypoint of the CDK application]
     ├── lambda                          [folder containing source code the lambda functions]
     │   ├── capture_news_feed           [lambda function to ingest news feeds]
-    │   ├── create-partition            [lambda function to create glue partitions]
     │   ├── firehose_topic_proxy        [lambda function to write topic analysis output to Amazon Kinesis Firehose]
     │   ├── firehose-text-proxy         [lambda function to write text analysis output to Amazon Kinesis Firehose]
-    │   ├── ingestion-consumer          [lambda function that consumes messages from Amazon Kinesis Data Stream]
+    │   ├── ingestion-consumer          [lambda function that consumes messages from Amazon Kinesis Data Streams]
+    │   ├── ingestion-custom            [lambda function that reads files from Amazon S3 bucket and pushes data to Amazon Kinesis Data Streams]
     │   ├── ingestion-producer          [lambda function that makes Twitter API call and pushes data to Amazon Kinesis Data Stream]
-    │   ├── ingestion-youtube           [lambda function that ingests comments from YouTube videos and pushes data to Amazon Kinesis Data Stream]
+    │   ├── ingestion-youtube           [lambda function that ingests comments from YouTube videos and pushes data to Amazon Kinesis Data Streams]
     │   ├── integration                 [lambda function that publishes inference outputs to Amazon Events Bridge]
     │   ├── layers                      [lambda layer function library for Node and Python layers]
     │   │   ├── aws-nodesdk-custom-config
@@ -106,10 +106,12 @@ The solution is deployed using a CloudFormation template with a lambda backed cu
     │   ├── ingestion                   [CDK constructs for data ingestion]
     │   ├── integration                 [CDK constructs for Amazon Events Bridge]
     │   ├── quicksight-custom-resources [CDK construct that invokes custom resources to create Amazon QuickSight resources]
+    │   ├── s3-event-notification       [CDK construct that configures S3 events to be pushed to Amazon EventBridge]
     │   ├── storage                     [CDK constructs that define storage of the inference events]
     │   ├── text-analysis-workflow      [CDK constructs for text analysis of ingested data]
     │   ├── topic-analysis-workflow     [CDK constructs for topic visualization of ingested data]
     │   └── visualization               [CDK constructs to build a relational database model for visualization]
+    ├── discovering-hot-topics.ts
 ```
 
 ## Creating a custom build
@@ -124,22 +126,22 @@ Clone this git repository
 
 ### 2. Build the solution for deployment
 
--   To run the unit tests
+- To run the unit tests
 
 ```
 cd <rootDir>/source
 chmod +x ./run-all-tests.sh
 ./run-all-tests.sh
 ```
 
--   Configure the bucket name of your target Amazon S3 distribution bucket
+- Configure the bucket name of your target Amazon S3 distribution bucket
 
 ```
 export DIST_OUTPUT_BUCKET=my-bucket-name
 export VERSION=my-version
 ```
 
--   Now build the distributable:
+- Now build the distributable:
 
 ```
 cd <rootDir>/deployment
@@ -148,7 +150,7 @@ chmod +x ./build-s3-dist.sh
 
 ```
 
--   Parameter details
+- Parameter details
 
 ```
 $DIST_OUTPUT_BUCKET - This is the global name of the distribution. For the bucket name, the AWS Region is added to the global name (example: 'my-bucket-name-us-east-1') to create a regional bucket. The lambda artifact should be uploaded to the regional buckets for the CloudFormation template to pick it up for deployment.
@@ -158,13 +160,13 @@ $CF_TEMPLATE_BUCKET_NAME - The name of the S3 bucket where the CloudFormation te
 $QS_TEMPLATE_ACCOUNT - The account from which the Amazon QuickSight templates should be sourced for Amazon QuickSight Analysis and Dashboard creation
 ```
 
--   When creating and using buckets it is recommeded to:
+- When creating and using buckets it is recommeded to:
 
-    -   Use randomized names or uuid as part of your bucket naming strategy.
-    -   Ensure buckets are not public.
-    -   Verify bucket ownership prior to uploading templates or code artifacts.
+  - Use randomized names or uuid as part of your bucket naming strategy.
+  - Ensure buckets are not public.
+  - Verify bucket ownership prior to uploading templates or code artifacts.
 
--   Deploy the distributable to an Amazon S3 bucket in your account. _Note:_ you must have the AWS Command Line Interface installed.
+- Deploy the distributable to an Amazon S3 bucket in your account. _Note:_ you must have the AWS Command Line Interface installed.
 
 ```
 aws s3 cp ./global-s3-assets/ s3://my-bucket-name-<aws_region>/discovering-hot-topics-using-machine-learning/<my-version>/ --recursive --acl bucket-owner-full-control --profile aws-cred-profile-name

source/.eslintignore

@@ -0,0 +1,12 @@
+# don't ever lint node_modules
+node_modules
+cdk.out
+# don't lint build output (make sure it's set to your correct build folder name)
+dist
+# don't lint test folders
+test
+# don't lint coverage output
+coverage
+*.config.js
+# dont lint eslint config
+*.eslint*

source/.gitignore

@@ -0,0 +1,18 @@
+*.js
+!jest.config.js
+*.d.ts
+node_modules
+
+# CDK asset staging directory
+.cdk.staging
+cdk.out
+
+# Parcel build directories
+.cache
+.build
+
+# JS files in the lambda folder should not be ignored
+!lambda/**/*.js
+
+# Coverage reports
+test/coverage-reports

source/.prettierignore

@@ -0,0 +1 @@
+node_modules

source/.prettierrc.yml

@@ -0,0 +1,8 @@
+# .prettierrc or .prettierrc.yaml
+proseWrap: 'preserve'
+trailingComma: 'none'
+tabWidth: 4
+semi: true
+singleQuote: true
+quoteProps: 'preserve'
+printWidth: 120

source/images/architecture.png

@@ -0,0 +1,14 @@
+module.exports = {
+    root: true,
+    parserOptions: {
+        ecmaVersion: 2021
+    },
+    env: {
+        node: true
+    },
+    extends: ['eslint:recommended'],
+    rules: {
+        indent: ['error', 4],
+        quotes: ['warn', 'single']
+    }
+};

source/lambda/.eslintrc.js

@@ -1,5 +1,6 @@
-moto==2.2.2
-pytest==6.2.4
-pytest-cov==2.12.1
+moto==2.3.1
+pytest==6.2.5
+pytest-cov==3.0.0
 botocore
-mock==4.0.3
+mock==4.0.3
+responses==0.16.0

source/lambda/capture_news_feed/requirements-dev.txt

@@ -15,11 +15,11 @@
 import json
 import os
 import unittest
-from datetime import datetime
+from datetime import datetime, timezone
 
 from moto import mock_kinesis
 from shared_util.service_helper import get_service_client
-from util.stream_helper import buffer_data_into_stream
+from shared_util.stream_helper import buffer_data_into_stream
 
 
 @mock_kinesis
@@ -48,7 +48,7 @@ def test_buffer_data_into_stream(self):
             "platform": "fakeplatform",
             "search_query": "query_str",
             "feed": {
-                "created_at": datetime.now().timestamp(),
+                "created_at": datetime.now(timezone.utc).timestamp(),
                 "id": "fakeid",
                 "id_str": "fakeid",
                 "text": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.",