Pipeline deploy causing temporal 503 on services?

[tyoc213]

Hi there, been deploying with a new pipeline my 5 services now on testing, but I have seen that while they are being deployed I got a 503 service temporarily unavailable, in particular in the 3rd stage of the pipeline (1. source, 2. build, 3. deploy).

Is this intended? I thought the purpose of the pipelines was to have zero downtime? or Im wrong in this?

By the way, I havent seen this if I use from command line the copilot svc deploy -n x --e staging only seen this downtime while executing stage 3 of the pipeline.

[Lou1415926]

Hello @tyoc213 👋🏼! This is not intended nor expected. It is also peculiar that it behaves differently in pipeline and from the cli. Both he pipeline and the cli send update request to CloudFormation, which then triggers update to the ECS / App Runner service. By the time CloudFormation accepts the update request, everything should be the same whether the request originates from the pipeline or from the cli 🤔 .

A few questions:

1. Did you observe this on all 5 services when deploying to "deploy"?
2. Have you configured the deployment field? If so, what is the configuration?
3. Can you take a look at the CloudFormation stack, go to the "Events" tab, and see if anything is unexpected there (e.g. any red XXX_FAILED_YYY events)

[tyoc213]

Edit to add more context:

After seeing that I could not change to spot instance from a normal instance, deleted the service and deployed it again until OK. Then ran it from the pipeline and it worked OK!, but still got the 503 on my services, which I could get this responses from the api and the 2 webapps.

one of the webapps answers

But the deploy looks like this

webapp-1 is already deployed

[tyoc213]

On new deploys I don't see an error, but still seeing the behavior, any hints?

[KollaAdithya]

Hi @tyoc213

just to have some clarifications on this. webapp-1 is deployed successfully and webapp-2 and api services are deployment in progress. Is this corrrect?

[KollaAdithya]

As @Lou1415926 already asked about this whether the deployment is default or recreate in your service manifest. Can you confirm on this.
Because if the deployment is recreate stop all the old tasks and launch new tasks.

[tyoc213]

yeah, missed that as what I had on deploy, indeed it was, thx

[tyoc213]

@KollaAdithya @Lou1415926 just one extra question are deployments with recreated faster than rolling??? (without consider the 503 temporal error and which will have first the new version out)?

[KollaAdithya]

@tyoc213 👋
The deployment will be faster in case of deployment.rolling: recreate because under the hood sets minimumHealthPercent is set to 0 and maximumHealthPercent is set to 100.

Here are few examples mentioned in the doc below on how different percentages of minimumHealth and maximum health takes effect on time taken for deployment of the service.

https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/service-options.html