Release 0.4.4

📢 General announcements:

• 🎉 Dynatrace AWS S3 log forwarder is now officially supported. (blog post)

🌟 Features:

• Allows detection of gzip logs based on the Content-Encoding metadata of S3 object. (#33)

🐛 Bugs:

• Fixes Dockerfile to make sure that both pip and dependencies are properly installed. (#37)
• Fixes a bug in recognition of ALB log files. (#44)

🔧 Maintenance:

• Updates base lambda container image from lambda/python:3.9.2023.04.17.20 to lambda/python:3.9.2023.06.28.13.
• Bumps requests from 2.28.2 to 2.31.0.
• Bumps regex from 2023.3.23 to 2023.6.3.
• Bumps aws-lambda-powertools from 2.14.0 to 2.18.0.
• Pins urllib3 version below 2.x.x to avoid introductions of breaking changes for botocore#2926
• Bumps moto from 4.1.7 to 4.1.12.

Release 0.4.3

🔧 Maintenance:

• Update base lambda container image from lambda/python:3.9.2023.04.04.12-x86_64 to lambda/python:3.9.2023.04.17.20-x86_64
• Bump requests from 2.28.1 to 2.28.2
• Bump jsonslicer from 0.1.7 to 0.1.8
• Bump regex from 2022.6.2 to 2023.3.23
• Bump aws-lambda-powertools from 2.6.0 to 2.14.0
• Bump moto from 4.0.3 to 4.1.7
• Bump responses from to 0.21.0 to 0.23.1

Release 0.4.2

Features:

• Unifies the eventbridge-cross-region and eventbridge-cross-account CloudFormation templates in a unique eventbridge-cross-region-or-account-forward-rules template for easier maintenance and remove potential confusion since differences were minimal.

Bugs:

• Fixes a bug on the CrossRegionCrossAccount Event bus policy where the SourceArn was incorrect
• Updates the EventBridge name prefix from dt-fwd-s3-object-created-to-{account_id}-{aws_region}-{hash} to dt-s3-log-fwd-to-{account_id}-{aws_region}-{hash} to avoid hitting the 64 chars name limit when using larger region code names. Now the rule name won't exceed 64 chars independently of the AWS region code.

Maintenance:

• Bumps the base lambda container images to lambda/python:3.9.2023.04.04.12 including security updates.

Release 0.4.1

• Updates base container image to lambda/python:3.9.2023.03.15.15 with multiple security updates

Release 0.4.0

Features

Adds log processing rules for:

• Amazon CloudFront access logs
• AWS Global Accelerator flow logs
• Amazon Managed Streaming for Kafka
• AWS Network Firewall
• Amazon Redshift audit logs
• Amazon S3 access logs
• Route 53 VPC resolver logs
• Amazon VPC Flow logs (default logs)
• AWS WAF logs

Also, adds support for attribute extraction of CloudWatch Logs -> Kinesis Firehose -> S3 for:

• AWS Lambda
• Amazon EKS control plane logs
• Amazon Route 53 public zone query logs

Bug fixes

• Fix ReceivedUncompressedLogFileSize metric calculation error.

Release 0.3.1

Bugs:

• Adds graceful handling of HTTP 400 errors from Dynatrace triggered when log timestamp of a log entry is older than 24 hours.
• AWS AppConfig deployment CloudFormation resource name is shortened to StackName-AllAtOnce as it's limited to 64 characters and deployment fails with long stack names. Add a warning on the docs to limit stack name to 54 characters.

Features:

• Public container images for AWS Lambda are now created as part of CI/CD pipeline

Release 0.3.0

This new release adds several enhancements and changes to the dynatrace-aws-s3-log-forwarder:

• Decoupling of the configuration of the log forwarder from code using AWS AppConfig. (Log forwarding and log processing rules format has changed, if you're upgrading read the notes below).
• Make cross-account deployments easier to manage, EventBridge policies are now managed via SAM template.
• Harmonize log attributes names with Dynatrace semantic attribute dictionary
• Optimizations to Dockerfile
• NotificationEmail parameter is now optional
• Default to x86_64 architecture container images. Users can still chose to build arm64 images.
• Documentation enhancements

Important notes for users upgrading:

• The log forwarding Lambda function now defaults to load configuration from AWS AppConfig. You'll need to deploy the new SAM template.yaml version to create required resources. To keep loading configuration from local files set the parameter LogForwarderConfigurationLocation to local.
• Log forwarding rules formatting has changed from a list of local files config/log_forwarding_rules/<bucket_name>.yaml format requiring a file per S3 bucket, to an array of log forwarding rules in YAML on an AWS AppConfig configuration profile named log-forwarding-rules (or optionally a local file config/log-forwarding-rules.yaml). You can find an example log-forwarding-rules configuration here. The previous format is still supported, but will be deprecated on upcoming releases.
• Log processing rules formatting has changed from a list of local files config/log_forwarding_rules/<rule_name>.yaml format requiring a file per rule, to an array of log processing rules in YAML on an AWS AppConfig configuration profile named log-processing-rules (or optionally a local file config/log-forwarding-rules.yaml). You can find an example log-processing-rules configuration here.
• Context attributes names added to logs have changed to be consistent with the sematric attributes dictionary:
  • log.source.bucket_name is now log.source.aws.s3.bucket.name
  • log.source.s3_key_name is now log.source.aws.s3.key.name
  • log.source.forwarder is now cloud.log_forwarder
• Build instructions have changed and there's a previous step to download dependent AWS Lambda extensions.

Release 0.2.0

• A CloudWatch monitoring dashboard is created to monitor the log forwarder
• Cross-account and cross-region use cases enhancements: better docs and use of a dedicated event bus.
• Simpler parameter customization: Lambda and SQS settings are now available as SAM template parameters with recommended defaults.
• Set up maximum Lambda concurrency
• Logging enhancements
• Fixes a bug in the ReceivedUncompressedLogFileSize CloudWatch metric

Release 0.1.0

Initial release of the dynatrace-aws-s3-log-forwarder