azure.azcollection.azure_rm_adgroup fails for existing group members who are Managed Identities

[howardjones]

SUMMARY

Using azure.azcollection.azure_rm_adgroup to ensure group membership results in an error if a listed object_id is
already a member:

One or more added object references already exist for the following modified properties: 'members'.

There is an existing, but closed as fixed/not-an-issue Issue in #1519

ISSUE TYPE

• Bug Report

COMPONENT NAME

azure_rm_adgroup

ANSIBLE VERSION

bash-5.1$ ansible --version
ansible [core 2.18.5]
  config file = None
  configured module search path = ['/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.11/site-packages/ansible
  ansible collection location = /runner/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.11.11 (main, Feb 10 2025, 00:00:00) [GCC 11.5.0 20240719 (Red Hat 11.5.0-5)] (/usr/bin/python3.11)
  jinja version = 3.1.6
  libyaml = True

COLLECTION VERSION

bash-5.1$ ansible-galaxy collection list azure.azcollection

# /usr/share/ansible/collections/ansible_collections
Collection         Version
------------------ -------
azure.azcollection 3.3.1 

CONFIGURATION

OS / ENVIRONMENT

Azure public cloud. Running playbooks inside a recently-built AWX EE using ansible-runner. Python 3.11 in the EE.

STEPS TO REPRODUCE

application_managed_identity_object_id is the principalId of a Managed Identity created with:

az identity create -g {{ resource_group }} -n {{ application_name }}-mi -o json

- name: Ensure Users are Members of a Group using object_id
  azure.azcollection.azure_rm_adgroup:
    raw_membership: true
    object_id: "{{ directory_readers_object_id}}"
    state: 'present'
    present_members:
      - "{{ application_managed_identity_object_id }}"

EXPECTED RESULTS

1st time - add managed identity to the AD group (correct)
2nd time - do nothing (instead it fails)

ACTUAL RESULTS

        error: MainError(additional_data={}, code='Request_BadRequest', details=None, inner_error=InnerError(additional_data={}, client_request_id='d5faa2ef-7449-4886-a7ad-f77f4369ca56', date=datetime.datetime(2025, 5, 15, 12, 41, 4), odata_type=None, request_id='3d0ee01d-23b2-4c44-9154-7e3bf46291b0'), message="One or more added object references already exist for the following modified properties: 'members'.", target=None)

The complete trace is:

The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_6p9t_yzf/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py", line 356, in exec_module
  File "/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_6p9t_yzf/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py", line 378, in update_members
  File "/usr/lib64/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_6p9t_yzf/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py", line 550, in add_group_member
  File "/usr/local/lib/python3.11/site-packages/msgraph/generated/groups/item/members/ref/ref_request_builder.py", line 93, in post
    return await self.request_adapter.send_no_response_content_async(request_info, error_mapping)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/kiota_http/httpx_request_adapter.py", line 389, in send_no_response_content_async
    await self.throw_failed_responses(response, error_map, parent_span, parent_span)
  File "/usr/local/lib/python3.11/site-packages/kiota_http/httpx_request_adapter.py", line 575, in throw_failed_responses
    raise exc
msgraph.generated.models.o_data_errors.o_data_error.ODataError: 
        APIError
        Code: 400
        message: None
        error: MainError(additional_data={}, code='Request_BadRequest', details=None, inner_error=InnerError(additional_data={}, client_request_id='d5faa2ef-7449-4886-a7ad-f77f4369ca56', date=datetime.datetime(2025, 5, 15, 12, 41, 4), odata_type=None, request_id='3d0ee01d-23b2-4c44-9154-7e3bf46291b0'), message="One or more added object references already exist for the following modified properties: 'members'.", target=None)
        

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/.ansible-/tmp/ansible-tmp-1747312860.9912302-878-45544755762225/AnsiballZ_azure_rm_adgroup.py", line 107, in <module>
    _ansiballz_main()
  File "/tmp/.ansible-/tmp/ansible-tmp-1747312860.9912302-878-45544755762225/AnsiballZ_azure_rm_adgroup.py", line 99, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/tmp/.ansible-/tmp/ansible-tmp-1747312860.9912302-878-45544755762225/AnsiballZ_azure_rm_adgroup.py", line 47, in invoke_module
    runpy.run_module(mod_name='ansible_collections.azure.azcollection.plugins.modules.azure_rm_adgroup', init_globals=dict(_module_fqn='ansible_collections.azure.azcollection.plugins.modules.azure_rm_adgroup', _modlib_path=modlib_path),
  File "<frozen runpy>", line 226, in run_module
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_6p9t_yzf/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py", line 578, in <module>
  File "/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_6p9t_yzf/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py", line 574, in main
  File "/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_6p9t_yzf/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py", line 279, in __init__
  File "/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_6p9t_yzf/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py", line 484, in __init__
  File "/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_6p9t_yzf/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py", line 361, in exec_module
  File "/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_6p9t_yzf/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py", line 517, in fail
  File "/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_6p9t_yzf/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible/module_utils/basic.py", line 1468, in fail_json
  File "/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_6p9t_yzf/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible/module_utils/basic.py", line 1442, in _return_formatted
  File "/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_6p9t_yzf/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible/module_utils/common/parameters.py", line 927, in remove_values
  File "/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_6p9t_yzf/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible/module_utils/common/parameters.py", line 470, in _remove_values_conditions
TypeError: Value of unknown type: <class 'msgraph.generated.models.o_data_errors.o_data_error.ODataError'>, 
        APIError
        Code: 400
        message: None
        error: MainError(additional_data={}, code='Request_BadRequest', details=None, inner_error=InnerError(additional_data={}, client_request_id='d5faa2ef-7449-4886-a7ad-f77f4369ca56', date=datetime.datetime(2025, 5, 15, 12, 41, 4), odata_type=None, request_id='3d0ee01d-23b2-4c44-9154-7e3bf46291b0'), message="One or more added object references already exist for the following modified properties: 'members'.", target=None)
        
fatal: [localhost]: FAILED! => {
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_6p9t_yzf/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 356, in exec_module\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_6p9t_yzf/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 378, in update_members\n  File \"/usr/lib64/python3.11/asyncio/base_events.py\", line 654, in run_until_complete\n    return future.result()\n           ^^^^^^^^^^^^^^^\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_6p9t_yzf/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 550, in add_group_member\n  File \"/usr/local/lib/python3.11/site-packages/msgraph/generated/groups/item/members/ref/ref_request_builder.py\", line 93, in post\n    return await self.request_adapter.send_no_response_content_async(request_info, error_mapping)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/kiota_http/httpx_request_adapter.py\", line 389, in send_no_response_content_async\n    await self.throw_failed_responses(response, error_map, parent_span, parent_span)\n  File \"/usr/local/lib/python3.11/site-packages/kiota_http/httpx_request_adapter.py\", line 575, in throw_failed_responses\n    raise exc\nmsgraph.generated.models.o_data_errors.o_data_error.ODataError: \n        APIError\n        Code: 400\n        message: None\n        error: MainError(additional_data={}, code='Request_BadRequest', details=None, inner_error=InnerError(additional_data={}, client_request_id='d5faa2ef-7449-4886-a7ad-f77f4369ca56', date=datetime.datetime(2025, 5, 15, 12, 41, 4), odata_type=None, request_id='3d0ee01d-23b2-4c44-9154-7e3bf46291b0'), message=\"One or more added object references already exist for the following modified properties: 'members'.\", target=None)\n        \n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/tmp/.ansible-/tmp/ansible-tmp-1747312860.9912302-878-45544755762225/AnsiballZ_azure_rm_adgroup.py\", line 107, in <module>\n    _ansiballz_main()\n  File \"/tmp/.ansible-/tmp/ansible-tmp-1747312860.9912302-878-45544755762225/AnsiballZ_azure_rm_adgroup.py\", line 99, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/tmp/.ansible-/tmp/ansible-tmp-1747312860.9912302-878-45544755762225/AnsiballZ_azure_rm_adgroup.py\", line 47, in invoke_module\n    runpy.run_module(mod_name='ansible_collections.azure.azcollection.plugins.modules.azure_rm_adgroup', init_globals=dict(_module_fqn='ansible_collections.azure.azcollection.plugins.modules.azure_rm_adgroup', _modlib_path=modlib_path),\n  File \"<frozen runpy>\", line 226, in run_module\n  File \"<frozen runpy>\", line 98, in _run_module_code\n  File \"<frozen runpy>\", line 88, in _run_code\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_6p9t_yzf/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 578, in <module>\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_6p9t_yzf/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 574, in main\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_6p9t_yzf/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 279, in __init__\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_6p9t_yzf/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py\", line 484, in __init__\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_6p9t_yzf/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_adgroup.py\", line 361, in exec_module\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_6p9t_yzf/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py\", line 517, in fail\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_6p9t_yzf/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible/module_utils/basic.py\", line 1468, in fail_json\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_6p9t_yzf/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible/module_utils/basic.py\", line 1442, in _return_formatted\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_6p9t_yzf/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible/module_utils/common/parameters.py\", line 927, in remove_values\n  File \"/tmp/ansible_azure.azcollection.azure_rm_adgroup_payload_6p9t_yzf/ansible_azure.azcollection.azure_rm_adgroup_payload.zip/ansible/module_utils/common/parameters.py\", line 470, in _remove_values_conditions\nTypeError: Value of unknown type: <class 'msgraph.generated.models.o_data_errors.o_data_error.ODataError'>, \n        APIError\n        Code: 400\n        message: None\n        error: MainError(additional_data={}, code='Request_BadRequest', details=None, inner_error=InnerError(additional_data={}, client_request_id='d5faa2ef-7449-4886-a7ad-f77f4369ca56', date=datetime.datetime(2025, 5, 15, 12, 41, 4), odata_type=None, request_id='3d0ee01d-23b2-4c44-9154-7e3bf46291b0'), message=\"One or more added object references already exist for the following modified properties: 'members'.\", target=None)\n        \n",
    "module_stdout": "",
    "msg": "MODULE FAILURE: No start of json char found\nSee stdout/stderr for the exact error",
    "rc": 1
}

[howardjones]

Added some basic debugging to azure_rm_adgroup update_members() function

            with open("/tmp/mylog","a") as log_f:
                log_f.write(f"CURRENT MEMBERS: {current_members}\n\n")
            members_to_add = list(set(present_members_by_object_id.keys()) - set(current_members))
            with open("/tmp/mylog","a") as log_f:
                log_f.write(f"MEMBERS TO ADD: {members_to_add}\n\n")

And I have found that despite this Managed Identity being listed in the Azure Entra ID portal as a member of the group, it does not appear in the CURRENT MEMBERS list above, either eith raw_membership set to true or false (longer list with raw_membership: true, but still not containing my ID)

So Ansible incorrectly has the ID in the members_to_add list.

[howardjones]

I seperated out the get_raw_group_members() method, and confirmed that while I get some members with odata_type='#microsoft.graph.servicePrincipal' I don't get all the ones listed as direct members in the Entra portal for this group. I have also confirmed that I can use the GraphServiceClient.service_principals endpoints to fetch this object_id by ID - that is, the SP does exist, with odata_type='#microsoft.graph.servicePrincipal'

(I get 20 members instead of the 35 listed in the azure portal - it's not just this one missing!)

import asyncio
from msgraph_beta.generated.groups.groups_request_builder import GroupsRequestBuilder
from msgraph_beta.generated.models.group import Group
from msgraph_beta.generated.groups.item.transitive_members.transitive_members_request_builder import \
    TransitiveMembersRequestBuilder
from msgraph_beta.generated.groups.item.group_item_request_builder import GroupItemRequestBuilder
from msgraph_beta.generated.models.reference_create import ReferenceCreate
from msgraph_beta.generated.users.item.user_item_request_builder import UserItemRequestBuilder
from msgraph_beta.generated.service_principals.service_principals_request_builder import ServicePrincipalsRequestBuilder

from msgraph_beta import GraphServiceClient
from azure.identity import DefaultAzureCredential




class Blah:
    def __init__(self):
        self.azure_credential_track2 = DefaultAzureCredential()
        self._client = self.get_msgraph_client()
    
    def get_msgraph_client(self):
        return GraphServiceClient(self.azure_credential_track2)
    
    async def get_raw_group_members(self, group_id, filters=None):
        request_configuration = GroupItemRequestBuilder.GroupItemRequestBuilderGetRequestConfiguration(
            query_parameters=GroupItemRequestBuilder.GroupItemRequestBuilderGetQueryParameters(
                # this ensures service principals are returned
                # see https://learn.microsoft.com/en-us/graph/api/group-list-members?view=graph-rest-1.0&tabs=http
                expand=["members"]
            ),
        )
        if filters:
            request_configuration.query_parameters.filter = filters
        group = await self._client.groups.by_group_id(group_id).get(request_configuration=request_configuration)
        return group.members


    async def get_user(self, user_id):
        req = ServicePrincipalsRequestBuilder.ServicePrincipalsRequestBuilderGetRequestConfiguration(
            query_parameters=ServicePrincipalsRequestBuilder.ServicePrincipalsRequestBuilderGetRequestConfiguration()
        )
        user = await self._client.service_principals.by_service_principal_id(user_id).get(request_configuration=req)
        return user

    def test(self, group_id):
        ret = asyncio.get_event_loop().run_until_complete(self.get_raw_group_members(group_id))
        current_members = [(object.id, object.display_name, object.odata_type) for object in ret]
                
        return current_members
    
    def test2(self, user_id):
        ret = asyncio.get_event_loop().run_until_complete(self.get_user(user_id=user_id))
        print(ret)
    


def main():
    
    group_id = "XXXXXX-YYYY-4e07-a3ab-6de8cbd4e560"
    user_id = "XXXXXX-YYYY-4b39-9842-0a1e617f6497"
    
    t = Blah()  
    cm = t.test(group_id)    
    print("========================")
    for c in cm:
        print(f"  {c[0]} {c[2]} {c[1]}")
        
    t.test2(user_id)

if __name__ == "__main__":
    main()

[Fred-sun]

@howardjones Add a member to a security or Microsoft 365 group. When using the API to add multiple members in one request, you can add up to only 20 members.

[howardjones]

This is ONE member in the request to add, and they are already a member.

But the request to LIST used by Ansible only returns 20 of the 35, which don't include the one, so it tries again.

[Fred-sun]

@howardjones Perhaps when it obtaining the member information, due to multiple pages for members, it only got the 20 members on the first page. I'm verifying this.
Also, If set raw_members=True, The members can be added or not?

[Fred-sun]

@howardjones I see that the "{{application_managed_identity_object_id}}" above you is the managed identity of the object_id. The members here should be group's object_id. Therefore, when adding the group object_id you provided, the error you mentioned will be reported because it cannot be found. Thank you!

- name: Create Root Group
  azure_rm_adgroup:
    display_name: "{{ resource_prefix }}-Group-Root"
    mail_nickname: "{{ resource_prefix }}-Group-Root"
    description: 'for test'
    state: 'present'

- name: Create Group Member 1
  azure_rm_adgroup:
    display_name: "{{ resource_prefix }}-Group-Member-1"
    mail_nickname: "{{ resource_prefix }}-Group-Member-1"
    state: 'present'
  register: create_pass_first

- name: Create Group Member 2
  azure_rm_adgroup:
    display_name: "{{ resource_prefix }}-Group-Member-2"
    mail_nickname: "{{ resource_prefix }}-Group-Member-2"
    state: 'present'
  register: create_pass_second

- name: Add the member to root group
  azure_rm_adgroup:
    display_name: "{{ resource_prefix }}-Group-Root"
    mail_nickname: "{{ resource_prefix }}-Group-Root"
    raw_membership: false
    state: 'present'
    present_members:
      - "{{ create_pass_first.object_id }}"
      - "{{ create_pass_second.object_id }}"

[howardjones]

@Fred-sun huh? The object_id field is the group you are adding members to and the present_members list (of one object_id in my case) are the members to verify.

The members should ALSO be groups?? How do you make a group of users if all you can add is other groups? That doesn't make any sense.

raw_members makes no difference to the outcome ("One or more added object references already exist"), but it does give a different (longer) incomplete list of current_members.

I could see that paging might be part of the answer, but Ansible doesn't use the Group List Members endpoint, it uses the Get Group By Id endpoint, so technically, the API is returning one object. If it used Group List Members endpoint, then there would be paging, but then there seem to be issues with Service Principals except in the Beta API.

[Fred-sun]

@howardjones Yes, I'm looking at the function of the parameter raw_membership and the errors caused by setting it to true. I will conduct an in-depth investigation into the other issues you mentioned. Thanks!

[howardjones]

I have two other issues open for this (one in msgraph-sdk for the maybe 20 limit, and one in msgraph-beta-sdk for something where even if you do use the beta sdk to get a list of group members, it actually calls the non-beta API endpoint).

It almost seems like you can't get a correct list of group members from any SDK currently :-(

I was trying to use the group list members API in the beta SDK, with paging:

    
    async def get_group_members(self, group_id):
        
        current_members = []
        
        # get group members
        members = await self._client.groups.by_group_id(group_id).members.get()
        if members:
            current_members.extend([m.id for m in members.value])

        # iterate over result batches > 100 rows
        while members is not None and members.odata_next_link is not None:
            members = await self._client.groups.by_group_id(id).members.with_url(members.odata_next_link).get()
            if members:
                current_members.extend([m.id for m in members.value])
                    
        return current_members

[howardjones]

With a different workaround for the msgraph_beta issue, I can now get a list of current_members that DOES contain my user. The page size is 100 users, not 20. Using the function in the previous comment, msgraph-beta-sdk and this:

   def get_msgraph_client(self):
        c =  GraphServiceClient(self.azure_credential_track2)
        # force the beta SDK to use the beta endpoint (issue 743 in their github)
        c.path_parameters['base_url'] = "https://graph.microsoft.com/beta/"
        c.request_adapter.base_url = "https://graph.microsoft.com/beta"

        return c

[Fred-sun]

@howardjones Idempotence failure is likely due to the failure to obtain the paging information, resulting in the return value not existing and an idempotence error. I added "Get pagination Information" in #2050. This issue should have been fixed. Could you give it a try? Thank you!