This was confirmed to be a bug in Lambda Function URLs. The fix was deployed around or before June 1, 2022 and provisioned concurrency for Function URLs no longer causes 400 status code / ThrottlingException responses if the number of concurrent requests sent is higher than the provisioned concurrency of the Function URL associated with a Lambda version or alias.
This repository contains an example of a possible bug with Lambda Provisioned Concurrency where the function will not scale above the Provisioned Concurrency value, even with Reserved Concurrency set far higher (to account-level unreserved concurrency).
This behavior does not match what is expected and it is demonstrated that an ALB pointing to a Provisioned Concurrency Lambda Alias does not reject requests with a 400 response code beyond the Provisioned Concurrency value. This seems to confirm that there may be a bug with Function URLs that have Provisioned Concurrency set.
Indications of the error:
- 400 Status Code on response
x-amzn-ErrorType: ThrottlingExceptionheader on response{"message":"Rate Exceeded."}response body
The CDK stack deploys 2 different examples both with an ALB and a Function URL:
- Docker-based Lambda built for ARM64
- Zip-based Lambda built for ARM64
The AWS Lambda URLs are configured with security set to lambda.FunctionUrlAuthType.NONE so they are open to any caller on the Internet. If this violates your security policies, change the type from lambda.FunctionUrlAuthType.NONE to lambda.FunctionUrlAuthType.AWS_IAM, then use awscurl instead of curl to test the function invocation.
nvm use
npm i
npm run build
npm run build:rollup
docker-compose build
docker-compose up
# At this point the service should be up but will not be initialized until first request
# To get a shell, if needed, when `docker-compose up` is running:
docker-compose exec app /bin/sh
# To send a test request to the local instance:
npm run test-local-request
# To deploy to an AWS account
npx cdk deploy
# Invoke the Lambda Function URL
curl -v [paste the URL here]
# Invoke the Lambda ALB URL
curl -v [paste the URL here]
Test concurrency limits using ab (apache bench) or hey (arguments are the same for both).
hey - Can be installed as a precompiled binary or via brew install hey on Mac.
The default provisioned concurrency of the Lambda Aliases is 5.
In general, test the following:
- Run with
-c(concurrent requests) equal to provisioned concurrency- In this case there should be no
400status code responses from any URL
- In this case there should be no
- Run with
-cmuch larger than provisioned concurrency (e.g.-c 100)- In this case the
$LATESTALB URL, Provisioned Alias ALB URL, and$LATESTFunction URL should return no400status code responses - The Provisioned Alias Function URL will return a large portion of
400status code responses and will never improve (it's not an issue of needing to start more instances... it simply will not start more instances or send more thanprovisioned concurrencyrequests to the function)
- In this case the
hey -c 100 -n 1000 https://[function-url-id].lambda-url.us-east-1.on.aws/
Result: OK
Result: OK
Result: OK
Result: OK
Result: FAILS - 98% of requests return a 400 status code
curl -v "https://[lambda-provisioned-alias-url].lambda-url.us-east-1.on.aws/"
> GET / HTTP/1.1
> Host: [lambda-provisioned-alias-url].lambda-url.us-east-1.on.aws
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 400 Bad Request
< Date: Wed, 11 May 2022 01:08:24 GMT
< Content-Type: application/json
< Content-Length: 28
< Connection: keep-alive
< x-amzn-RequestId: 1db11618-724e-4940-83ce-f5e4069c7593
< x-amzn-ErrorType: ThrottlingException
<
* Connection #0 to host [lambda-provisioned-alias-url].lambda-url.us-east-1.on.aws left intact
{"message":"Rate Exceeded."}
