Skip to content

support list application inference profile #156

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from

Conversation

heisenbergye
Copy link
Contributor

@heisenbergye heisenbergye commented Jun 20, 2025

Issue #, if available:

Description of changes: support for application inference profiles created by users

such as users who applied for Bedrock MAP, they need to create Application Inference Profiles with MAP tag based on foundation models or system Inference Profiles, as docs https://github.com/aws-samples/sample-bedrock-inference-profile-mgmt-tool

after tagged, list Application Inference Profiles ARN in bedrock access gateway

'anthropic.claude-3-5-sonnet-20240620-v1:0': 'arn:aws:bedrock:us-west-2:808577411626:application-inference-profile/v8gtbe8pryd4'
'arn:aws:bedrock:us-west-2:808577411626:application-inference-profile/v8gtbe8pryd4': {'modalities': ['TEXT', 'IMAGE']},

users need to invoke model with Application Inference Profiles ARN

curl http://127.0.0.1:8000/api/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer bedrock" \
-d '{
"model": "arn:aws:bedrock:us-west-2:xxxxxx:application-inference-profile/v8gtbe8pryd4",
"max_tokens": 4096,
"messages": [{
"role": "user",
"content": "the price of iphone16"
}]
}'

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

for users create AWS Bedrock Application Inference Profiles.

such as users who applied for Bedrock MAP,  they need to create Application Inference Profiles with MAP tag, as docs  https://github.com/aws-samples/sample-bedrock-inference-profile-mgmt-tool
after tagged, users need to invoke model with Application Inference Profiles ARN
@zxkane
Copy link
Member

zxkane commented Jun 23, 2025

Thanks for your contribution. It's implemented by #131

@zxkane zxkane closed this Jun 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants