When to use the Worker Service?

[fade2black]

I've read the docs about Services and in particular Worker Service. I am confused about when to use the Worker Service.

Say my worker consumes messages from an SNS+SQS which arrive once in three days, but once arrived it may take 15-20 minutes to process all of them (1 message takes no more than 60 seconds to process). So, my worker is kind of idle for most time. Does it mean I need to pay for my worker service even when it is idle? Lambdas seem to be more suitable in this case but the worker service does not use lambdas.

Note: Spots do not work since the jobs are NOT fault-tolerant.

[paragbhingre]

As far as I can tell, there isn't any extra cost for ECS, which is worker service in this case. But AWS resources do cost money, such as SNS, SQS, and so on. So the resources that you are paying money for on a per-use basis are SNS and SQS and not ECS. So even if your service is sitting idle for a certain time, you only pay for those SNS and SQS based on their usage. 

[paragbhingre]

I think I understood what you were asking when you mentioned "Does it mean I need to pay for my worker service even when it is idle?" --> So there is this feature request from another customer, feel free to add your scenario here and upvote so that we can prioritize this feature.

[fade2black]

@paragbhingre yeah, exactly. Actually I've already integrated SAM as bkg jobs (which needs an access to the Inet and to a DB instance), but it brings in a lot of manual setup such as creating NAT gateway, Secret Manager, RDS Proxy, VPC Endpoint (for accessing Secret Manger), configuring Subnets and Sec groups. These things are not configurable out-of-box via Copilot or SAM
Actually in Copilot's Pub/Sub workers are never idle since they constantly must poll the queue, and hence extra charge.

[paragbhingre]

Yes, your request totally makes sense to me.

[fade2black]

@paragbhingre Anyway, I'm curious what the AWS Copilot will offer as a solution for background jobs. Waiting for updates. :-)

[paragbhingre]

Do the scheduled jobs meet your needs? 

[fade2black]

@paragbhingre

Do the scheduled jobs meet your needs?

Not at all. They are "Scheduled Jobs", mine are Triggered (by some event) Jobs. For example by SQS. The scheduled jobs are only triggered either on a fixed schedule (in some point in time) or periodically by providing a rate.

More info: https://aws.github.io/copilot-cli/docs/concepts/jobs/

[bvtujo]

Hey @fade2black, your worker services will actually scale to zero if you allow them to and there are no messages in the queue for a long time.

Say I have a Worker Service manifest with the following config:

count:
  range: 0-2
  queue_delay:
    acceptable_latency: 1m
    message_processing_time: 250ms

When messages arrive in the queue burstily, you can see ECS scale up to consume them, then wait for the scale-in cooldown, then gradually scale down to zero.

So you'll only pay for resources used when processing messages + the scale-in cooldown time, which is much better than just running a Fargate service all the time in the background.

Note that there's also some scaleup time involved depending on how you set your acceptable_latency and message_processing_time fields. ECS will attempt to keep the oldest message at or below the acceptable latency by scaling in or out as needed.