Skip to content

[FR] x-vpc: Allow Lookup.Identifier usage #798

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
jakub-qg opened this issue Mar 25, 2025 · 7 comments
Open

[FR] x-vpc: Allow Lookup.Identifier usage #798

jakub-qg opened this issue Mar 25, 2025 · 7 comments
Assignees
@JohnPreston
Labels
enhancement New feature or request

Comments

@jakub-qg
Copy link

jakub-qg commented Mar 25, 2025 •
edited
Loading

Is your feature request related to a problem? Please describe.

Make the syntax work:

x-vpc:
  Lookup:
    VpcId:
      Identifier: vpc-xxx

Output (if there are no VPC with tags in whole AWS acct):

2025-03-25 11:39:47 [    INFO] Processing x-vpc
Traceback (most recent call last):
  File "/Users/user/.local/bin/compose-x", line 8, in <module>
    sys.exit(main())
             ~~~~^^
  File "/Users/user/.local/pipx/venvs/ecs-composex/lib/python3.13/site-packages/ecs_composex/cli.py", line 213, in main
    root_stack = generate_full_template(settings)
  File "/Users/user/.local/pipx/venvs/ecs-composex/lib/python3.13/site-packages/ecs_composex/ecs_composex.py", line 257, in generate_full_template
    add_x_resources(settings)
    ~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/Users/user/.local/pipx/venvs/ecs-composex/lib/python3.13/site-packages/ecs_composex/ecs_composex.py", line 181, in add_x_resources
    x_stack = module.stack_class(
        module.mapping_key,
    ...<2 lines>...
        Parameters={ROOT_STACK_NAME.title: Ref(AWS_STACK_NAME)},
    )
  File "/Users/user/.local/pipx/venvs/ecs-composex/lib/python3.13/site-packages/ecs_composex/vpc/vpc_stack.py", line 371, in __init__
    self.vpc_resource.lookup_vpc()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/user/.local/pipx/venvs/ecs-composex/lib/python3.13/site-packages/ecs_composex/vpc/vpc_stack.py", line 173, in lookup_vpc
    vpc_settings = lookup_x_vpc_settings(self)
  File "/Users/user/.local/pipx/venvs/ecs-composex/lib/python3.13/site-packages/ecs_composex/vpc/vpc_aws.py", line 106, in lookup_x_vpc_settings
    vpc_arn = find_aws_resource_arn_from_tags_api(
        vpc_resource.lookup[VPC_ID.title],
    ...<2 lines>...
        allow_multi=False,
    )
  File "/Users/user/.local/pipx/venvs/ecs-composex/lib/python3.13/site-packages/ecs_composex/common/aws.py", line 237, in find_aws_resource_arn_from_tags_api
    return handle_search_results(
        arns, name, res_types, aws_resource_search, allow_multi=allow_multi
    )
  File "/Users/user/.local/pipx/venvs/ecs-composex/lib/python3.13/site-packages/ecs_composex/common/aws.py", line 176, in handle_search_results
    raise LookupError(
    ...<3 lines>...
    )
LookupError: ('No resources were found with the provided tags and information', None, 'ec2:vpc')

Output (if there are two tagged VPCs but using "Identifier"):

LookupError: ('More than one resource None:ec2:vpc was found with the current tags.Found', ['arn:aws:ec2:us-east-2:123:vpc/vpc-xxx', 'arn:aws:ec2:us-east-2:123:vpc/vpc-yyy'])

Not sure if this is a bug or feature. Docs are unclear.

Lookups docs suggest this should be possible:

Identifier | The Service resource ID (i.e. vpc-abcd1234)

x-vpc docs nor test cases ever mention this field

Reasoning:

  • Network resources shared via RAM shares do not have Tags by default (unless some effort is made), ID/Arn lookup would be great.

Describe the solution you'd like
Allow ID/ARN lookup

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.
@jakub-qg jakub-qg added the enhancement New feature or request label Mar 25, 2025
@jakub-qg jakub-qg changed the title [FR] x-vpc: Allow Lookup.Identifier usage [FR] x-vpc: Allow Lookup.Identifier usage for untagged resources Mar 25, 2025
@jakub-qg jakub-qg changed the title [FR] x-vpc: Allow Lookup.Identifier usage for untagged resources [FR] x-vpc: Allow Lookup.Identifier usage Mar 25, 2025
@JohnPreston
Copy link
Member

Hello :)
So this would use the Identifier as the tag key and : vpc-xxx as the tag value to use. So if that pair is not set, you won't find it. Lookup works using Tags as the mean to find the resources.

x-vpc:
  Lookup:
    VpcId:
      Tags:
        Identifier: vpc-xxx

Should work and is the recommended way to use Lookup (finds items via tags api). The tags you select here for the VPC in particular though must resolve to a single resource. For Subnets, the tags can match up to multiple subnets of course.

There is a way to give the ARN indeed of the resource you are looking for but that's not the recommended way. Let me dig it though.

@jakub-qg
Copy link
Author

Thank you, however that syntax is not working.

x-vpc:
  Lookup:
    VpcId:
      Tags:
        Identifier: vpc-030xxxxxxxxxx

Also tried:

      Tags:
        - Identifier: vpc-030xxxxxxxxxx

LookupError: ('No resources were found with the provided tags and information', None, 'ec2:vpc')

Well, at least I finally tagged the network resources!

@JohnPreston
Copy link
Member

JohnPreston commented Mar 25, 2025 via email

@JohnPreston
Copy link
Member

Thank you, however that syntax is not working.

x-vpc:
  Lookup:
    VpcId:
      Tags:
        Identifier: vpc-030xxxxxxxxxx

Also tried:

      Tags:
        - Identifier: vpc-030xxxxxxxxxx

LookupError: ('No resources were found with the provided tags and information', None, 'ec2:vpc')

Well, at least I finally tagged the network resources!

Well so, if your network resources are not tagged, this explains this error message, there is nothing to find :D

@jakub-qg
Copy link
Author

To clarify, the Identifier tag didn't work for looking up the vpc by name. I'd expect such functionality to exist, this is a common pattern we use with e.g. Terraform (we pass the network res names/ids via variables)

@JohnPreston
Copy link
Member

Lookup uses the Tags as the primary focal point to find resources. It uses the AWS tagging API rather than, i.e. go to EC2, describe all VPCs, and find the one that matches all the Tags. The excellent Tagging Resource API does that better and faster. Then yes, once compose-x has the ARN of the resource, doing all sorts of further describes on the resource itself, becomes much easier.

Resources in AWS have an ID that is regional, but the real identifier that works all the time is the ARN. Therefore, in the code you can pass the ARN directly. But not the shorthand ID which is regional. I haven't yet had the case of multi-region use-cases but ARNs just remove the guessing altogether.

"Name" is not a thing (not consistently for all resources anyways, and I always recommend not to give things a name, let CFN generate it), there is a massively common pattern used by AWS customers to use the tag key Name to identify resources.

So, in Lookup, if you did

x-vpc:
  Lookup:
    VpcId:
      Arn: arn:aws:ec2:region::account/vpc/vpc-id # on top of my head

Then that would work, as you are saying "Use the ARN, don't bother looking up (See https://github.com/compose-x/ecs_composex/blob/main/ecs_composex/compose/x_resources/__init__.py#L249)

Otherwise, yes, your resources, network or otherwise, must be tagged appropriately for this to work. I will check in the docs, as this has had a few evolution and however the docs contain the JSON schemas of what is expected to be set by the user, this does not yet render into the most human friendly docs to go through.

I hope this sheds some light on this. I will spin a VPC up etc. and give you a working example of what works with the latest version of the code.

@mikejgray
Copy link

I wasn't able to get the lookup syntax working with just giving it an Arn, unfortunately, since that would be my preference. But I did get this tag syntax working:

x-vpc:
  Lookup:
    VpcId:
      Tags:
        Name: "Customer Dashboard"
        env: prod
    PublicSubnets:
      Tags:
        "aws-cdk:subnet-type": Public
        env: prod
    AppSubnets:
      Tags:
        "aws-cdk:subnet-type": Private
        env: prod
    StorageSubnets:
      Tags:
        "aws-cdk:subnet-type": Private
        env: prod

Some quirk in the tag API required that I include both tags for the VPC even though only one VPC has that particular Name tag. Not a bug in this tool, but apparently in AWS.

Throwing in my +1 for being able to reference explicitly by a VPC ID!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@JohnPreston JohnPreston

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@JohnPreston @mikejgray @jakub-qg