Coop close doesn't work

[JssDWt]

node id: 03532f66199c2b2c6f3b61319e32c6bddcba78288ef03157f2cef3b7e3d90f72e8
short_channel_id: 822435x1817x0

The Close request hangs indefinitely (it's currently hanging)

These are the relevant logs:

[2025-07-03 10:30:58.532 INFO breez_sdk_core::breez_services:1761] node-logs: DEBUG   plugin-gl-plugin-internal: Received event Ok(RpcCall(\"/cln.Node/ListPeerChannels\")) from gl_plugin
[2025-07-03 10:30:58.557 DEBUG gl_client::node::service:153] Sending request Request { method: POST, uri: /cln.Node/Close, version: HTTP/2.0, headers: {"te": "trailers", "content-type": "application/grpc", "glauthpubkey": "xx", "glauthsig": "xx", "glts": "AAABl89pD/w", "glrune": "xx"}, body: UnsyncBoxBody }
[2025-07-03 10:30:58.745 INFO breez_sdk_core::breez_services:1761] node-logs: DEBUG   plugin-gl-plugin-internal: Received event Ok(RpcCall(\"/cln.Node/Close\")) from gl_plugin
[2025-07-03 10:30:58.785 INFO breez_sdk_core::breez_services:1761] node-logs: INFO    02c811e575be2df47d8b48dab3d3f1c9b0f6e16d0d40b5ed78253308fc2bd7170d-chan#1: State changed from CHANNELD_NORMAL to CHANNELD_SHUTTING_DOWN
[2025-07-03 10:30:58.785 INFO breez_sdk_core::breez_services:1761] node-logs: DEBUG   lightningd: close_command: timeout = 172800

According to the peer the channel is still in CHANNELD_NORMAL. It never receives any close request. I see no signer issues in the logs.

This happens to many nodes at the moment, let's take this one as the base case.

Instances

@cdecker collects the node_ids here so we can check them off as we address them

• 034374e3575943c0a23f67e1170d4480e4da2da8704067fd870404ba6af69c6f8d
• 032090dae69100f3da3d604a892fd7d7838820d3f106b8223a0d6c29448f94ec46
• 03c03161a9111fdf965c6c43b89ea47a6aec5e9dacb7fd8997a7a4b33bd51f8707
• 024610da907b94695b208cef6afb0460f5967f227f31cad4b2d46ecf04793c7780
• 03532f66199c2b2c6f3b61319e32c6bddcba78288ef03157f2cef3b7e3d90f72e8
• 03c03161a9111fdf965c6c43b89ea47a6aec5e9dacb7fd8997a7a4b33bd51f8707
• 0390aae21cd07794c3b15fb2db390cb8a44a9aff16a0317457fa689394777ebc46
• 02556c825fb9914892525e62c7d15c265bd796ae6b029e0fdf75061caaad231f37

[cdecker]

We have identified this to be caused by a signer <> node desync issue. Specifically we're seeing two different symptoms on this, both returned to the node as rejections by the signer:

• State number disagreement:

Here are two examples:

**BROKEN** plugin-gl-plugin-internal: Signer reports an error: Other(processing request: Signing(Status { code: FailedPrecondition, message: \"policy failure: get_per_commitment_point: commitment_number 549 invalid when next_holder_commit_num is 539\" }))

and

**BROKEN** plugin-gl-plugin-internal: Signer reports an error: Other(processing request: Signing(Status { code: FailedPrecondition, message: \"policy failure: validate_mutual_close_tx: to_holder_value 67458 is larger than holder_info.broadcaster_value_sat 57398\" }))

I am working on a script to check and fix up any mismatches between the signer state and the node state, which may take a couple of days to complete and then test. In the meantime having a list of presumed stuck nodes would be good, allowing us to check them one by one as soon as we have the tooling.

[kingonly]

Fixing this would also allow users to continue and manually close their channels, correct?

[cdecker]

Correct, the moment we fix up the signer state to match the node state, the signer will just pick up the new state, and start signing off on operations again. From there, any pending close command will be executed, and finished. That's why I am hoping Relai can keep the signer running in the background, and remove UI elements pertaining to LN while still keeping the signer working.

[elgamal7]

I’ll add similar cases here in the comments to test whether the fix works as expected:

1. 034374e3575943c0a23f67e1170d4480e4da2da8704067fd870404ba6af69c6f8d - diagnostic_data_b1c75263-0fbd-4f99-9e60-f7333e561717.log
2. 03c03161a9111fdf965c6c43b89ea47a6aec5e9dacb7fd8997a7a4b33bd51f8707

[elgamal7]

Hi, do you already have an update on this issue?

[matteorelai]

I'll add a similar case here, where the user has an active channel but cannot close it:

032090dae69100f3da3d604a892fd7d7838820d3f106b8223a0d6c29448f94ec46
diagnostic_data_ae2c705d-dcc0-46e4-9546-fb1e05508b3d.log

[elgamal7]

Hi cdecker.

"That's why I am hoping Relai can keep the signer running in the background, and remove UI elements pertaining to LN while still keeping the signer working."

What exactly should I tell our engineers not to disable so the signer keeps running in the background?

When do you expect a fix for the signer <> node desync issue?

[cdecker]

I think the sdk starts the signer as soon as you initialize the sdk. So that's all that is required. You can remove all UI components related to LN, but initializing and running the sdk gives us the ability to debug and unstick the nodes as soon as we have a fix.