Support for Cloudfront realtime logs

[bgardner-noggin]

Cloudfront does not support delivering standard access logs to an S3 bucket in all regions. I am utilising cloudfront realtime logs to work around this limitation

This feature request is for these templates to support an optional parameter "Existing cloudfront realtime logs table". This would also require another optional parameter for an "Existing Glue Database"

Realtime logs have a configurable set of fields to record, some of which are sensitive so we are not logging them. This is why the stack should support an existing table, rather than trying to create a new one

I've already started a fork for these changes. Extra changes I am making at the same time is removal of the custom lambdas / events to manage the partitions, instead using partition projection.

Are these changes something that this project would be interested in? I'm currently attempting to make these changes with the intention of submitting a pull request, but if this project is not interested it will make my changes simpler as I can simply neuter parts of the code base in my fork.

[WillAWS]

Hi bgardner-noggin, thank you for your suggestion. Due to the limitation of CloudFront, WAF and S3 lambda event, the solution must be deployed in us-east-1 and S3 bucket must exist in the same region if CloudFront endpoint is chosen. Is your approach using CloudFront realtime logs a workaround for this limitation so that the solution can support CloudFront logs in S3 in other regions? Currently CloudFront realtime logs is not on our roadmap yet and we can't promise anything, but feel free to share your PR or simply your repo if you want to make simpler changes as PR may take too long.

[bgardner-noggin]

Yes, we are using realtime logs so that we log into an S3 bucket in the same region as our application. This is required by our clients as they have concerns about data sovereignty (real or imagined)

Manually deploying a related stack in us-east-1 is not feasible for us as as we initiate this stack as a substack, and the outputs are also referenced elsewhere in our stacks.

I'd already worked around the WAFv2::WebACL and WAFv2::IPSet limitations of us-east-1 by using a custom resource to build these (which also eliminates the Custom::Timer as boto3 has native retries on throttling)

S3 lambda event should not be a problem as my buckets are in the same region as the lambda

Once I am finished I will generate a pull request from my forked repo

[bios6]

Thanks for raising this issue. We will add this to our backlog and look into this issue