Error during creation of credential definition with revocation enabled #3624
Comments
|
@esune — heads up on this one that might be related to what you are doing. |
|
Status Update: After days of debugging, I’m still stuck. The error persists when revocation is enabled in multitenant setups and What Happens
What I’ve Tried
Where It BreaksThe error happens here: Why? The code suddenly tries to find the tenant’s DID in the admin wallet instead of the tenant’s wallet. The Mystery
Request for Help: Simplest Reproduction:
|
|
Thank you for the troubleshooting @MonolithicMonk, really appreciated. I will try to take a look asap and see if I can help, haven't yet had a chance to dig into this specific issue. Will report back if/when I do, if anyone else has input please let's continue this conversation - we'll get to a resolution. |
|
Full context of the error logs before failure: DEBUG /acapy_agent/anoncreds/revocation.py:396 ENTER: create_and_register_revocation_list() - RevRegDef ID: RANDOM2025DIDRANDOM101:4:RANDOM2025DIDRANDOM101:3:CL:2738835:random1:CL_ACCUM:1, Options: {'revocation_registry_size': 100, 'support_revocation': True}
DEBUG /acapy_agent/anoncreds/revocation.py:402 Create and register revocation list profile: <Settings('trimmed', transport.max_outbound_retry=4, transport.ws.heartbeat_interval=3, transport.ws.timeout_interval=15, default_label=Testest Test, wallet.key=9c232979-23eb-4b4a-88ca-9d8f8961eede, wallet.name=Testest Test, wallet.storage_type=postgres_storage, wallet.type=askar-anoncreds)>
DEBUG /acapy_agent/anoncreds/revocation.py:405 Normalized options: {'revocation_registry_size': 100, 'support_revocation': True}
DEBUG /acapy_agent/anoncreds/revocation.py:408 Attempting to fetch revocation registry data from storage - RevRegDef ID: RANDOM2025DIDRANDOM101:4:RANDOM2025DIDRANDOM101:3:CL:2738835:random1:CL_ACCUM:1
DEBUG /acapy_agent/anoncreds/revocation.py:420 Storage fetch results - RevRegDef Entry: Found, Private Entry: Found
DEBUG /acapy_agent/anoncreds/revocation.py:455 Fetching associated credential definition: RANDOM2025DIDRANDOM101:3:CL:2738835:random1
INFO /acapy_agent/anoncreds/revocation_setup.py:134 Successfully uploaded tails file for RevRegDef ID: RANDOM2025DIDRANDOM101:4:RANDOM2025DIDRANDOM101:3:CL:2738835:random1:CL_ACCUM:0
DEBUG /acapy_agent/anoncreds/revocation_setup.py:158 Proceeding to create/register revocation list for RevRegDef ID: RANDOM2025DIDRANDOM101:4:RANDOM2025DIDRANDOM101:3:CL:2738835:random1:CL_ACCUM:0
DEBUG /acapy_agent/anoncreds/revocation.py:396 ENTER: create_and_register_revocation_list() - RevRegDef ID: RANDOM2025DIDRANDOM101:4:RANDOM2025DIDRANDOM101:3:CL:2738835:random1:CL_ACCUM:0, Options: {'revocation_registry_size': 100, 'support_revocation': True}
DEBUG /acapy_agent/anoncreds/revocation.py:402 Create and register revocation list profile: <Settings('trimmed', transport.max_outbound_retry=4, transport.ws.heartbeat_interval=3, transport.ws.timeout_interval=15, default_label=Testest Test, wallet.key=9c232979-23eb-4b4a-88ca-9d8f8961eede, wallet.name=Testest Test, wallet.storage_type=postgres_storage, wallet.type=askar-anoncreds)>
DEBUG /acapy_agent/anoncreds/revocation.py:405 Normalized options: {'revocation_registry_size': 100, 'support_revocation': True}
DEBUG /acapy_agent/anoncreds/revocation.py:408 Attempting to fetch revocation registry data from storage - RevRegDef ID: RANDOM2025DIDRANDOM101:4:RANDOM2025DIDRANDOM101:3:CL:2738835:random1:CL_ACCUM:0
DEBUG /acapy_agent/anoncreds/revocation.py:471 Successfully retrieved credential definition
DEBUG /acapy_agent/anoncreds/revocation.py:485 Deserializing revocation registry definition
DEBUG /acapy_agent/anoncreds/revocation.py:488 Successfully deserialized RevRegDef
DEBUG /acapy_agent/anoncreds/revocation.py:497 Deserializing credential definition
DEBUG /acapy_agent/anoncreds/revocation.py:500 Successfully deserialized CredDef
DEBUG /acapy_agent/anoncreds/revocation.py:509 Loading revocation registry private definition
DEBUG /acapy_agent/anoncreds/revocation.py:514 Successfully loaded private definition
DEBUG /acapy_agent/anoncreds/revocation.py:524 Updating tails location to local path for RevRegDef ID: RANDOM2025DIDRANDOM101:4:RANDOM2025DIDRANDOM101:3:CL:2738835:random1:CL_ACCUM:1
DEBUG /acapy_agent/anoncreds/revocation.py:530 Tails location updated - Original: https://tails-test.vonx.io/hash/6aKH42FKW4bAEMcMBCvdEREh6teYzBZNCHa34XQF5igx, New: /home/.indy_client/tails/6aKH42FKW4bAEMcMBCvdEREh6teYzBZNCHa34XQF5igx
DEBUG /acapy_agent/anoncreds/revocation.py:537 Creating revocation status list for RevRegDef ID: RANDOM2025DIDRANDOM101:4:RANDOM2025DIDRANDOM101:3:CL:2738835:random1:CL_ACCUM:1
INFO /acapy_agent/anoncreds/revocation.py:549 Successfully created revocation status list for RevRegDef ID: RANDOM2025DIDRANDOM101:4:RANDOM2025DIDRANDOM101:3:CL:2738835:random1:CL_ACCUM:1
DEBUG /acapy_agent/anoncreds/revocation.py:562 Retrieving AnonCreds registry implementation
DEBUG /acapy_agent/anoncreds/revocation.py:564 Registry implementation: AnonCredsRegistry
DEBUG /acapy_agent/anoncreds/revocation.py:569 Registering revocation list for RevRegDef ID: RANDOM2025DIDRANDOM101:4:RANDOM2025DIDRANDOM101:3:CL:2738835:random1:CL_ACCUM:1
DEBUG /acapy_agent/anoncreds/revocation.py:420 Storage fetch results - RevRegDef Entry: Found, Private Entry: Found
DEBUG /acapy_agent/anoncreds/revocation.py:455 Fetching associated credential definition: RANDOM2025DIDRANDOM101:3:CL:2738835:random1
DEBUG /acapy_agent/anoncreds/revocation.py:471 Successfully retrieved credential definition
DEBUG /acapy_agent/anoncreds/revocation.py:485 Deserializing revocation registry definition
DEBUG /acapy_agent/anoncreds/revocation.py:488 Successfully deserialized RevRegDef
DEBUG /acapy_agent/anoncreds/revocation.py:497 Deserializing credential definition
DEBUG /acapy_agent/anoncreds/revocation.py:500 Successfully deserialized CredDef
DEBUG /acapy_agent/anoncreds/revocation.py:509 Loading revocation registry private definition
DEBUG /acapy_agent/anoncreds/revocation.py:514 Successfully loaded private definition
DEBUG /acapy_agent/anoncreds/revocation.py:524 Updating tails location to local path for RevRegDef ID: RANDOM2025DIDRANDOM101:4:RANDOM2025DIDRANDOM101:3:CL:2738835:random1:CL_ACCUM:0
DEBUG /acapy_agent/anoncreds/revocation.py:530 Tails location updated - Original: https://tails-test.vonx.io/hash/BJZD1uTuBsWeehBGJvzp1XZYjz8mkcdn9PLQAXzgkQRq, New: /home/.indy_client/tails/BJZD1uTuBsWeehBGJvzp1XZYjz8mkcdn9PLQAXzgkQRq
DEBUG /acapy_agent/anoncreds/revocation.py:537 Creating revocation status list for RevRegDef ID: RANDOM2025DIDRANDOM101:4:RANDOM2025DIDRANDOM101:3:CL:2738835:random1:CL_ACCUM:0
INFO /acapy_agent/anoncreds/revocation.py:549 Successfully created revocation status list for RevRegDef ID: RANDOM2025DIDRANDOM101:4:RANDOM2025DIDRANDOM101:3:CL:2738835:random1:CL_ACCUM:0
DEBUG /acapy_agent/anoncreds/revocation.py:562 Retrieving AnonCreds registry implementation
DEBUG /acapy_agent/anoncreds/revocation.py:564 Registry implementation: AnonCredsRegistry
DEBUG /acapy_agent/anoncreds/revocation.py:569 Registering revocation list for RevRegDef ID: RANDOM2025DIDRANDOM101:4:RANDOM2025DIDRANDOM101:3:CL:2738835:random1:CL_ACCUM:0
DEBUG /acapy_agent/ledger/indy_vdr.py:222 Opening the pool ledger
DEBUG /acapy_agent/ledger/indy_vdr.py:222 Opening the pool ledger
DEBUG /acapy_agent/ledger/indy_vdr.py:222 Opening the pool ledger
WARNING /acapy_agent/ledger/multiple_ledger/indy_vdr_manager.py:135 Did RANDOM2025DIDRANDOM101 not posted to ledger indicio-demo
DEBUG /acapy_agent/ledger/indy_vdr.py:1245 send_revoc_reg_entry.Wallet instance: <AskarWallet>
DEBUG /acapy_agent/ledger/indy_vdr.py:1246 send_revoc_reg_entry.Profile context: <Settings('trimmed', transport.max_outbound_retry=4, transport.ws.heartbeat_interval=3, transport.ws.timeout_interval=15, default_label=multitenant-admin, wallet.key=multitenant-admin-wallet-key, wallet.name=multitenant-admin-wallet, wallet.storage_type=postgres_storage, wallet.type=askar-anoncreds)>
DEBUG /acapy_agent/ledger/indy_vdr.py:1249 send_revoc_reg_entry.Public DID retrieved from wallet: None
ERROR /acapy_agent/anoncreds/revocation.py:584 Failed to register revocation list for ID RANDOM2025DIDRANDOM101:4:RANDOM2025DIDRANDOM101:3:CL:2738835:random1:CL_ACCUM:1: Unknown DID: RANDOM2025DIDRANDOM101
Traceback (most recent call last):
File "/acapy_agent/anoncreds/revocation.py", line 575, in create_and_register_revocation_list
result = await anoncreds_registry.register_revocation_list(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/acapy_agent/anoncreds/registry.py", line 171, in register_revocation_list
return await registrar.register_revocation_list(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/acapy_agent/anoncreds/default/legacy_indy/registry.py", line 890, in register_revocation_list
result = await self._revoc_reg_entry_with_fix(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/acapy_agent/anoncreds/default/legacy_indy/registry.py", line 824, in _revoc_reg_entry_with_fix
rev_entry_res = await ledger.send_revoc_reg_entry(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/acapy_agent/ledger/indy_vdr.py", line 1252, in send_revoc_reg_entry
did_info = await wallet.get_local_did(issuer_did)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/acapy_agent/wallet/askar.py", line 398, in get_local_did
raise WalletNotFoundError("Unknown DID: {}".format(did))
acapy_agent.wallet.error.WalletNotFoundError: Unknown DID: RANDOM2025DIDRANDOM101 |
|
Edits to generate logs: # acapy_agent/anoncreds/revocation_setup.py
async def on_rev_reg_def(self, profile: Profile, event: RevRegDefFinishedEvent):
"""Handle rev reg def finished."""
payload = event.payload
# --- Detailed Logging START ---
LOGGER.debug("=" * 80)
LOGGER.debug(f"Entered 'on_rev_reg_def' handler.")
LOGGER.debug(
"ENTER: on_rev_reg_def() - Received RevRegDefFinishedEvent. "
"Event ID: %s, RevRegDef ID: %s, Tag: %s",
event.payload,
payload.rev_reg_def_id,
payload.rev_reg_def.tag
)
LOGGER.debug("Full event payload: %s", payload)
LOGGER.debug("Profile settings at entry: %s", profile.context.settings)
auto_create_revocation = True
if is_author_role(profile):
LOGGER.debug("Agent is in author role, checking auto-create settings")
auto_create_revocation = profile.settings.get(
"endorser.auto_create_rev_reg", False
)
LOGGER.info(
"Author role auto-create revocation registry setting: %s",
auto_create_revocation
)
else:
LOGGER.debug("Agent is NOT in author role, proceeding with default auto-create")
if auto_create_revocation:
LOGGER.info(
"Auto-creation of revocation registry enabled. "
"Proceeding with revocation setup for RevRegDef ID: %s",
payload.rev_reg_def_id
)
revoc = AnonCredsRevocation(profile)
failed_to_upload_tails = False
# Tails file upload sequence
LOGGER.debug(
"Attempting tails file upload for RevRegDef ID: %s",
payload.rev_reg_def_id
)
try:
LOGGER.debug("Calling upload_tails_file() with RevRegDef: %s", payload.rev_reg_def)
await revoc.upload_tails_file(payload.rev_reg_def)
LOGGER.info(
"Successfully uploaded tails file for RevRegDef ID: %s",
payload.rev_reg_def_id
)
except AnonCredsRevocationError as err:
LOGGER.warning(
"FAILED to upload tails file for RevRegDef ID: %s. Error: %s",
payload.rev_reg_def_id,
str(err),
exc_info=True
)
failed_to_upload_tails = True
LOGGER.debug("Set failed_to_upload_tails flag to True")
if failed_to_upload_tails:
LOGGER.warning(
"Tails file upload failed. Adding 'failed_to_upload' flag to options. "
"Original options: %s",
payload.options
)
payload.options["failed_to_upload"] = True
LOGGER.debug("Updated options: %s", payload.options)
# Revocation list creation
LOGGER.debug(
"Proceeding to create/register revocation list for RevRegDef ID: %s",
payload.rev_reg_def_id
)
try:
await revoc.create_and_register_revocation_list(
payload.rev_reg_def_id,
payload.options
)
LOGGER.info(
"Successfully created and registered revocation list for RevRegDef ID: %s",
payload.rev_reg_def_id
)
except Exception as e:
LOGGER.error(
"Failed to create/register revocation list for RevRegDef ID: %s. Error: %s",
payload.rev_reg_def_id,
str(e),
exc_info=True
)
raise
# Active registry management
if payload.rev_reg_def.tag == str(0):
LOGGER.debug(
"First registry detected (tag=0). Attempting to set active registry. "
"RevRegDef ID: %s",
payload.rev_reg_def_id
)
try:
await revoc.set_active_registry(payload.rev_reg_def_id)
LOGGER.info(
"Successfully set active registry to: %s",
payload.rev_reg_def_id
)
LOGGER.debug("Current profile settings after activation: %s", profile.context.settings)
except Exception as e:
LOGGER.error(
"Failed to set active registry for ID: %s. Error: %s",
payload.rev_reg_def_id,
str(e),
exc_info=True
)
raise
else:
LOGGER.debug(
"Not setting active registry - tag %s is not the initial registry (0)",
payload.rev_reg_def.tag
)
else:
LOGGER.info(
"Auto-creation of revocation registry DISABLED. "
"Skipping further processing for RevRegDef ID: %s",
payload.rev_reg_def_id
)
LOGGER.debug(
"EXIT: on_rev_reg_def() - Completed processing for RevRegDef ID: %s",
payload.rev_reg_def_id
)# acapy_agent/anoncreds/revocation.py
async def create_and_register_revocation_list(
self, rev_reg_def_id: str, options: Optional[dict] = None
):
"""Create and register a revocation list."""
LOGGER.debug(
"ENTER: create_and_register_revocation_list() - RevRegDef ID: %s, Options: %s",
rev_reg_def_id,
options
)
LOGGER.debug("Create and register revocation list profile: %s", self.profile.context.settings)
options = options or {}
LOGGER.debug("Normalized options: %s", options)
try:
LOGGER.debug(
"Attempting to fetch revocation registry data from storage - "
"RevRegDef ID: %s", rev_reg_def_id
)
async with self.profile.session() as session:
rev_reg_def_entry = await session.handle.fetch(
CATEGORY_REV_REG_DEF, rev_reg_def_id
)
rev_reg_def_private_entry = await session.handle.fetch(
CATEGORY_REV_REG_DEF_PRIVATE, rev_reg_def_id
)
LOGGER.debug(
"Storage fetch results - RevRegDef Entry: %s, Private Entry: %s",
"Found" if rev_reg_def_entry else "Missing",
"Found" if rev_reg_def_private_entry else "Missing"
)
except AskarError as err:
LOGGER.error(
"Storage error fetching revocation registry data for ID %s: %s",
rev_reg_def_id,
str(err),
exc_info=True
)
raise AnonCredsRevocationError(
"Error retrieving required revocation registry definition data"
) from err
# Validate required entries
missing = []
if not rev_reg_def_entry:
missing.append("revocation registry definition")
if not rev_reg_def_private_entry:
missing.append("revocation registry private definition")
if missing:
LOGGER.error(
"Missing required revocation registry components for ID %s: %s",
rev_reg_def_id,
", ".join(missing)
)
raise AnonCredsRevocationError(
f"Missing required revocation registry data: {', '.join(missing)}"
)
try:
cred_def_id = rev_reg_def_entry.value_json["credDefId"]
LOGGER.debug(
"Fetching associated credential definition: %s", cred_def_id
)
async with self.profile.session() as session:
cred_def_entry = await session.handle.fetch(
CATEGORY_CRED_DEF, cred_def_id
)
if not cred_def_entry:
LOGGER.error(
"Credential definition not found: %s", cred_def_id
)
raise AnonCredsRevocationError(
f"Credential definition {cred_def_id} not found"
)
LOGGER.debug("Successfully retrieved credential definition")
except AskarError as err:
LOGGER.error(
"Storage error fetching credential definition %s: %s",
cred_def_id,
str(err),
exc_info=True
)
raise AnonCredsRevocationError(
f"Error retrieving cred def {cred_def_id}"
) from err
# Deserialization phase
LOGGER.debug("Deserializing revocation registry definition")
try:
rev_reg_def = RevRegDef.deserialize(rev_reg_def_entry.value_json)
LOGGER.debug("Successfully deserialized RevRegDef")
except Exception as e:
LOGGER.error(
"Failed to deserialize revocation registry definition: %s",
str(e),
exc_info=True
)
raise
LOGGER.debug("Deserializing credential definition")
try:
cred_def = CredDef.deserialize(cred_def_entry.value_json)
LOGGER.debug("Successfully deserialized CredDef")
except Exception as e:
LOGGER.error(
"Failed to deserialize credential definition: %s",
str(e),
exc_info=True
)
raise
LOGGER.debug("Loading revocation registry private definition")
try:
rev_reg_def_private = RevocationRegistryDefinitionPrivate.load(
rev_reg_def_private_entry.value_json
)
LOGGER.debug("Successfully loaded private definition")
except Exception as e:
LOGGER.error(
"Failed to load revocation registry private definition: %s",
str(e),
exc_info=True
)
raise
# Tails location handling
LOGGER.debug(
"Updating tails location to local path for RevRegDef ID: %s",
rev_reg_def_id
)
original_tails = rev_reg_def.value.tails_location
rev_reg_def.value.tails_location = self.get_local_tails_path(rev_reg_def)
LOGGER.debug(
"Tails location updated - Original: %s, New: %s",
original_tails,
rev_reg_def.value.tails_location
)
# Revocation list creation
LOGGER.debug(
"Creating revocation status list for RevRegDef ID: %s",
rev_reg_def_id
)
try:
rev_list = RevocationStatusList.create(
cred_def.to_native(),
rev_reg_def_id,
rev_reg_def.to_native(),
rev_reg_def_private,
rev_reg_def.issuer_id,
)
LOGGER.info(
"Successfully created revocation status list for RevRegDef ID: %s",
rev_reg_def_id
)
except Exception as e:
LOGGER.error(
"Failed to create revocation status list: %s",
str(e),
exc_info=True
)
raise
# Registry interaction
LOGGER.debug("Retrieving AnonCreds registry implementation")
anoncreds_registry = self.profile.inject(AnonCredsRegistry)
LOGGER.debug(
"Registry implementation: %s",
type(anoncreds_registry).__name__
)
LOGGER.debug(
"Registering revocation list for RevRegDef ID: %s",
rev_reg_def_id
)
try:
native_rev_list = RevList.from_native(rev_list)
result = await anoncreds_registry.register_revocation_list(
self.profile, rev_reg_def, native_rev_list, options
)
LOGGER.info(
"Revocation list registration result for ID %s: %s",
rev_reg_def_id,
result.revocation_list_state.state
)
except Exception as e:
LOGGER.error(
"Failed to register revocation list for ID %s: %s",
rev_reg_def_id,
str(e),
exc_info=True
)
raise
# Handle failed upload state
if options.get("failed_to_upload", False):
LOGGER.warning(
"Tails file upload failed detected, setting revocation list state to FAILED"
)
original_state = result.revocation_list_state.state
result.revocation_list_state.state = RevListState.STATE_FAILED
LOGGER.debug(
"State transition - From: %s, To: %s",
original_state,
result.revocation_list_state.state
)
# Storage operation
LOGGER.debug("Storing revocation list result")
try:
await self.store_revocation_registry_list(result)
LOGGER.info(
"Successfully stored revocation list for RevRegDef ID: %s",
rev_reg_def_id
)
except Exception as e:
LOGGER.error(
"Failed to store revocation list result: %s",
str(e),
exc_info=True
)
raise
LOGGER.debug(
"EXIT: create_and_register_revocation_list() - RevRegDef ID: %s",
rev_reg_def_id
)
return result |
|
I should also add that this issue only occurs after |
|
@MonolithicMonk are the edits you made to generate logs something that would be useful to include in ACA-Py in general, or just for this troubleshooting? If you push changes to a branch might be easier to take them into account for further testing. |
|
@esune sorry for late reply, I was busy outdoors today. The edits I made are strictly to debug the issue. They may be useful going forward but as is they are far too verbose |
|
I've been able to track the issue down to # acapy_agent/ledger/base.py
@abstractmethod
async def send_revoc_reg_entry(
self,
revoc_reg_id: str,
revoc_def_type: str,
revoc_reg_entry: dict,
issuer_did: Optional[str] = None,
write_ledger: bool = True,
endorser_did: Optional[str] = None,
profile: Optional[Profile] = None, # <----- adding optional parameter
) -> dict:
"""Publish a revocation registry entry to the ledger."""# acapy_agent/ledger/indy_vdr.py
async def send_revoc_reg_entry(
self,
revoc_reg_id: str,
revoc_def_type: str,
revoc_reg_entry: dict,
issuer_did: Optional[str] = None,
write_ledger: bool = True,
endorser_did: Optional[str] = None,
profile: Optional[Profile] = None, # <---- adding optional parameter
) -> dict:
"""Publish a revocation registry entry to the ledger."""
current_profile = profile or self.profile
async with current_profile.session() as session:
# rest of code ...
if current_profile.context.settings.get("wallet.type") == "askar-anoncreds":
# rest of code ...
|
Thank you! |
|
@esune The following is the logs that arises after resolving the profile context switching issue. It is very similar fix however, it is leading me down a rabbit hole of injecting a profile parameter into several functions to get the proper context. I'm not comfortable going down that path because it spans several files that I'm not familiar with. Not to mention the unintended consequences. The code base obviously knew how to determine tenant context before without the need to explicitly pass in a profile to every function. Here is the error logs for the new issue that arose: DEBUG /acapy_agent/core/event_bus.py:101 Notifying subscribers: <Event topic=acapy::record::endorse_transaction::transaction_acked, payload={"trimmed"}>
DEBUG /acapy_agent/core/event_bus.py:101 Notifying subscribers: <Event topic=acapy::record::endorse_transaction::transaction_acked, payload={"trimmed"}>
DEBUG /acapy_agent/core/event_bus.py:101 Notifying subscribers: <Event topic=anoncreds::revocation-registry-definition::finished, ("trimmed")>
DEBUG /acapy_agent/core/event_bus.py:101 Notifying subscribers: <Event topic=anoncreds::revocation-registry-definition::finished, ("trimmed")>
DEBUG /acapy_agent/ledger/indy_vdr.py:222 Opening the pool ledger
DEBUG /acapy_agent/ledger/indy_vdr.py:222 Opening the pool ledger
DEBUG /acapy_agent/ledger/indy_vdr.py:222 Opening the pool ledger
ERROR /acapy_agent/core/event_bus.py:123 Error occurred while processing event
Traceback (most recent call last):
File "/acapy_agent/core/event_bus.py", line 121, in notify
await processor()
File "/acapy_agent/anoncreds/revocation_setup.py", line 105, in on_rev_reg_def
await revoc.create_and_register_revocation_list(
File "/acapy_agent/anoncreds/revocation.py", line 452, in create_and_register_revocation_list
result = await anoncreds_registry.register_revocation_list(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/acapy_agent/anoncreds/registry.py", line 171, in register_revocation_list
return await registrar.register_revocation_list(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/acapy_agent/anoncreds/default/legacy_indy/registry.py", line 891, in register_revocation_list
result = await self._revoc_reg_entry_with_fix(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/acapy_agent/anoncreds/default/legacy_indy/registry.py", line 824, in _revoc_reg_entry_with_fix
rev_entry_res = await ledger.send_revoc_reg_entry(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/acapy_agent/ledger/indy_vdr.py", line 1271, in send_revoc_reg_entry
resp = await legacy_indy_registry.txn_submit(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/acapy_agent/anoncreds/default/legacy_indy/registry.py", line 1224, in txn_submit
return await shield(
^^^^^^^^^^^^^
File "/home/dgateman/.pyenv/versions/3.12.9/lib/python3.12/asyncio/futures.py", line 289, in __await__
yield self # This tells Task to wait for completion.
^^^^^^^^^^
File "/home/dgateman/.pyenv/versions/3.12.9/lib/python3.12/asyncio/tasks.py", line 385, in __wakeup
future.result()
File "/home/dgateman/.pyenv/versions/3.12.9/lib/python3.12/asyncio/futures.py", line 202, in result
raise self._exception.with_traceback(self._exception_tb)
File "/home/dgateman/.pyenv/versions/3.12.9/lib/python3.12/asyncio/tasks.py", line 314, in __step_run_and_handle_result
result = coro.send(None)
^^^^^^^^^^^^^^^
File "/acapy_agent/ledger/indy_vdr.py", line 1350, in txn_submit
resp = await self._submit(
^^^^^^^^^^^^^^^^^^^
File "/acapy_agent/ledger/indy_vdr.py", line 360, in _submit
await wallet.sign_message(request.signature_input, sign_did.verkey)
File "/acapy_agent/wallet/askar.py", line 729, in sign_message
raise WalletNotFoundError("Missing key for sign operation")
acapy_agent.wallet.error.WalletNotFoundError: Missing key for sign operation |
|
@MonolithicMonk I attempted to create a minimal example of the issue you're seeing here: https://github.com/Indicio-tech/acapy-minimal-example/blob/test/rev-reg-entry-failure/examples/multitenancy/example.py This example succeeds (it can fail occasionally but only from timeouts when a ledger is being particularly slow). You can run that example with |
|
@dbluhm See your logs, you will likely find |
|
Actually, no, there aren't any |
|
You're running v1.2.4 |
|
I have observed this error in version 1.3*. |
As a side note, I actually love your repro repo . It shows a lot about how acapy works internally |
|
Re:
@dbluhm — perhaps we should add that repo to OWF as “acapy-debug” or something like that. @MonolithicMonk ’s comment is about the 10th I’ve seen about how useful that repo is. Thoughts? We could add some docs for it on https://aca-py.org and encourage it’s use. |
I think we'd be interested in that but double checking with the powers that be! |
I just tested with the py3.12-nightly-2025-04-16 image and still don't see |
After further testing, I have now discovered that the trigger for the error is setting the configuration value The test was conducted using my testing environment however I tested both your configuration and mine and discovered the difference are those values - ps: I used my testing environment because I think your test repo hard codes a check for genesis_url and I didn't want to dig through it. |
|
By the way, could this have been related to this bug? #3646 The fix would have been included in the @MonolithicMonk Is it possible to replicate the issue from the latest main branch? Does it still persist? |
|
I just tested with latest nightly and the error persists for |
|
Hmm, I see. That's a very strange issue. AFAICT, it should be something to do with how I recently made some logging improvements to what's going on there. (#3332) Could you possible share the startup debug logs? Mainly the ledger config stage, to see if anything stands out. I was looking at the argparse and ledger config code just now, and figured it's worth it to make some of that more readable... kind of like a baby step to try debug this issue: #3664 |
This comment has been minimized.
This comment has been minimized.
|
Thank you - I'm not 100% clued up on exactly how things should be, but that does help Can you share how you are creating and configuring the connection between the issuer-tenant and the endorser? And do you configure any metadata on the connection after it's complete - like setting endorser/author roles or endorser info? Edit: I may be on to a red herring, because the issue seems related to multi-ledger, but I just want to be sure |
|
There does seem to be something amiss with the multi-ledger config. And it's being elucidated with this refactoring: #3664 Hoping that we can get that refactored PR merged, with improved logging for multi-ledger case, and re-test what's going on here |
|
I connect tenants to endorser using |
|
While the focus of #3664 doesn't directly address this issue, my suspicion is that the solution lies somewhere in one of the files edited - |
|
Yeah, the PR is just to try make it a bit more clear what's going on. I was also looking at those 2 files, and thought it's to do with how There are unfortunately quite a lot of changes between 1.2.4 and the 1.3.0rc's, because 1.2.x has just been patched with cherry-picks. So the release date of Mar 13 for 1.2.4 is not reflective of the main branch at Mar 13. As for One way to possibly test if that's the breaking change, is to reset on the commit right before that (f30c77a) and then reset to the one merging that PR (ef1ab19). See if the problem isn't there before, and pops up after. Only if you're up for testing on Good Friday :-) |
|
Hi @MonolithicMonk - I hope all's well. Just wanted to check in if there's any news on this front? |
|
I was hoping that the PR @ff137 did would help isolate the issue but have not look into this myself. It's not a configuration I use. Would still like to avoid merge the anti-pattern if possible. May be able to look into it this week. |
@MonolithicMonk That's all good. My last comment pointed out that I think the issue was introduced by #3470 - and it's over my head what changes were going on there. Maybe @dbluhm or @jamshale has insight into whether/how that PR would have caused this issue (since it's the last change to touch the file, and it's relevant to profile scoping). I also added a suggestion on a way to test if that PR is in fact the cause of the breaking change:
Only cuz I can't replicate the issue, I'd need the feedback to know if that does help prove if the PR caused the issue or not. Anyway I'd like to help, but it's in your guys' hands. |
|
I'll try and look at it but since there's a way to do the same configuration that doesn't have the problem I'd like to not block 1.3.0 any longer. If the configuration needs to be done this way they will need to stay on 1.2.4 and we can look at this for a new patch release. There's enough testing, and enough people have looked at this issue, I'm pretty confident it's isolated enough. I agree that perhaps this PR #3470 that actually fixes multitenancy unique profiles could potentially be related if there was an issue with how this configuration was expecting to use the root profile incorrectly. I can try and setup this configuration and look into it asap. |
and others
Problem Description
I get the error an error during the creation of a credential definition with revocation enabled using anoncreds. The credential definition is actually created however, the error occurs during the revocation operation. I still haven't been able to determine what the cause of the error is. Here is a partial log:
I can create the credential definition in a multitenant - endorsement setup. However, after the successful creation and endorsement of the Revocation Registry Definition (RevRegDef), the automatic process to create the corresponding initial Revocation Registry Entry (the list state) fails with the WalletNotFoundError shown above.
To clarify the sequence based on my logs:
A tenant agent (using askar-anoncreds wallet type, configured with an endorser) successfully creates a CredDef via endorsement.
The same tenant agent successfully creates the associated RevRegDef via endorsement. The anoncreds::revocation-registry-definition::finished event is emitted.
The event handler revocation_setup.py::on_rev_reg_def triggers automatically upon receiving this event.
This handler successfully uploads the tails file and then calls revocation.py::create_and_register_revocation_list.
This leads down a call stack eventually calling indy_vdr.py::send_revoc_reg_entry, which needs to look up the tenant's issuer DID using askar.py::get_local_did to prepare the transaction for endorsement.
Failure: get_local_did raises WalletNotFoundError for the tenant's own DID at this point in the execution flow.
It seems that while the initial request context (handled via API and endorsement flow) correctly resolves the tenant's wallet and DID, the execution context available within the on_rev_reg_def event handler (running asynchronously after the RevRegDef write is acknowledged) is somehow unable to access the same DID within the tenant's wallet.
Possibly related to #3586
The text was updated successfully, but these errors were encountered: