No. of SQS Listener Endpoint Container Threads are Breaching the Maximum Limit

[vishnumurthy-nd]

Below is my sqs configuration:

Spring SQS Configuration

spring.cloud.aws.sqs.enabled=true
spring.cloud.aws.sqs.listener.max-concurrent-messages=10
spring.cloud.aws.sqs.listener.max-messages-per-poll=10

Service is connecting to only one Queue. Ideally the threads count in logs should have been spanned b/w 1 to 10, but I have observed count is reaching maximum of 75 in some cases. Below the sample thread count. I am using the default SQS configuration nothing is overridden from code side. Hence, I assume in some cases number of message processing is going to 75.

Could you please help why its happening and how to ensure thread count won't breaches the value 10. Additional thing is my consumer is taking more than 2 seconds to process the message and its not creating this issue.

io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-10]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-11]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-12]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-13]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-14]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-15]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-16]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-17]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-18]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-19]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-1]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-20]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-21]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-22]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-23]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-24]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-25]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-26]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-27]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-28]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-29]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-2]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-30]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-31]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-32]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-33]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-34]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-35]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-36]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-37]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-38]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-39]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-3]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-40]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-41]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-42]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-43]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-44]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-4]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-5]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-6]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-7]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-8]
io.awspring.cloud.sqs.sqsListenerEndpointContainer#0-9]

[tomazfernandes]

Hi @vishnumurthy-nd, this is not the expected behavior.

Are you listening to many queues in the same Listener?

Here's where we create the executor - notice we have a fixed pool size:

spring-cloud-aws/spring-cloud-aws-sqs/src/main/java/io/awspring/cloud/sqs/listener/AbstractPipelineMessageListenerContainer.java

Lines 245 to 256 in 02e997c

	protected TaskExecutor createTaskExecutor() {
	ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
	int poolSize = getContainerOptions().getMaxConcurrentMessages() * this.messageSources.size();
	executor.setMaxPoolSize(poolSize);
	executor.setCorePoolSize(poolSize);
	// Necessary due to a small racing condition between releasing the permit and releasing the thread.
	executor.setQueueCapacity(poolSize);
	executor.setAllowCoreThreadTimeOut(true);
	executor.setThreadFactory(createThreadFactory());
	executor.afterPropertiesSet();
	return executor;
	}

If you can't share more information about your configuration, can you provide a Minimal Reproducible Example that illustrates the issue?

Thanks.

[vishnumurthy-nd]

@tomazfernandes Thanks for responding, here my concern was why thread count reached upto 45, when we have created the thread pool with limited size.

I was able to identify the issue here, its due to the thread life cycle management by thread pool keep-alive max time 60sec.

[tomazfernandes]

Yup, that's the expected behavior if your threads are usually stale for more than the default 60 seconds.

Thanks for sharing the result of your investigation.