[bug]: Possible Memory Leak Gossip context.AfterFunc() / v0.19.0-beta

[MegalithicBTC]

BACKGROUND:
running this image:

 lightninglabs/lnd:v0.19.0-beta

using these (new for 0.19.x) config settings:

gossip.msg-rate-bytes=18750000
gossip.msg-burst-bytes=3000000 
gossip.sub-batch-delay=1s
no-disconnect-on-pong-failure=true
num-restricted-slots=500

NOTES ABOUT THIS NODE:
This node has about 200 open channels. Many of these channels are with LDK peers with intermittant connections. These peers with intermittant connections may be repeatedly requesting a lot of gossip -- typically when they come back online, they'll request gossip again, so nodes that are going online and offline many times per day might be requesting gossip many times per day.

OBSERVED MEMORY BEHAVIOR:
Over 2 to 3 days, memory consumption grew to 17GB for the LND container. In contrast, I have a different LND node, ALSO running 0.19, but with NO flakey LDK peers, where memory consumption is stable at around 2.5 GB. This points me again at the hypothesis that this memory leak is happening here because this problem node has a lot of intermittent channel peers who are frequently requesting gossip.

HOW I AM REMEDIATING THIS ISSUE FOR NOW:
Restarting the node returns memory usage to normal. It seems if I restart this node every 24 hours, I contain the issue so it is manageable.

HYPOTHESIS THAT GOSSIP IS THE SOURCE OF THE PROBLEM:

Discovery/Gossip dominates the call stack:

github.com/lightningnetwork/lnd/discovery.(*GossipSyncer).replyHandler - 6.16GB (69.74%)
github.com/lightningnetwork/lnd/discovery.(*GossipSyncer).replyPeerQueries - 6.16GB (69.74%)
github.com/lightningnetwork/lnd/discovery.(*GossipSyncer).replyShortChanIDs - 6.02GB (68.19%)

Specific gossip operations consuming memory:

FilterGossipMsgs - 988MB
ApplyGossipFilter - 1.42GB

WHAT IS HAPPENING?

The gossip syncer is creating thousands of contexts via ContextGuard.Create() when:

Replying to peer queries about channel ranges
Handling short channel ID requests
Filtering gossip messages
Processing gossip batches
Each context creation triggers the ContextGuard.ctxQuitUnsafe() method which calls context.AfterFunc(), creating goroutines that accumulate and hold memory.

DETAILED PROFILE DATA:
I've enabled profiling on this node, and here provide its output:

go tool pprof heap.pb.gz
File: lnd
Type: inuse_space
Time: 2025-06-02 05:49:41 PDT
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top -cum
Showing nodes accounting for 6843.71MB, 77.52% of 8827.97MB total
Dropped 396 nodes (cum <= 44.14MB)
Showing top 10 nodes out of 37
      flat  flat%   sum%        cum   cum%
         0     0%     0%  8427.50MB 95.46%  github.com/lightningnetwork/lnd/fn/v2.(*ContextGuard).Create
     250MB  2.83%  2.83%  6975.90MB 79.02%  github.com/lightningnetwork/lnd/fn/v2.(*ContextGuard).ctxQuitUnsafe
 1697.68MB 19.23% 22.06%  6376.70MB 72.23%  context.AfterFunc
         0     0% 22.06%  6156.50MB 69.74%  github.com/lightningnetwork/lnd/discovery.(*GossipSyncer).replyHandler
         0     0% 22.06%  6156.50MB 69.74%  github.com/lightningnetwork/lnd/discovery.(*GossipSyncer).replyPeerQueries
         0     0% 22.06%  6019.57MB 68.19%  github.com/lightningnetwork/lnd/discovery.(*GossipSyncer).replyShortChanIDs
 2963.84MB 33.57% 55.64%  4679.02MB 53.00%  context.(*cancelCtx).propagateCancel
 1715.18MB 19.43% 75.06%  1715.18MB 19.43%  context.(*cancelCtx).Done
     217MB  2.46% 77.52%  1451.60MB 16.44%  context.WithCancel
         0     0% 77.52%  1415.90MB 16.04%  github.com/lightningnetwork/lnd/discovery.(*GossipSyncer).ApplyGossipFilter.func2
(pprof) list context.withCancel
Total: 8.62GB
ROUTINE ======================== context.withCancel in context/context.go
    1.21GB     1.21GB (flat, cum) 13.99% of Total
 Error: could not find file context/context.go on path /mypool/workspace/LND-With-Docker
(pprof) tree
Showing nodes accounting for 8507.27MB, 96.37% of 8827.97MB total
Dropped 396 nodes (cum <= 44.14MB)
----------------------------------------------------------+-------------
      flat  flat%   sum%        cum   cum%   calls calls% + context              
----------------------------------------------------------+-------------
                                         4679.02MB   100% |   context.AfterFunc
 2963.84MB 33.57% 33.57%  4679.02MB 53.00%                | context.(*cancelCtx).propagateCancel
                                         1715.18MB 36.66% |   context.(*cancelCtx).Done
----------------------------------------------------------+-------------
                                         1715.18MB   100% |   context.(*cancelCtx).propagateCancel
 1715.18MB 19.43% 53.00%  1715.18MB 19.43%                | context.(*cancelCtx).Done
----------------------------------------------------------+-------------
                                         6376.70MB   100% |   github.com/lightningnetwork/lnd/fn/v2.(*ContextGuard).ctxQuitUnsafe
 1697.68MB 19.23% 72.23%  6376.70MB 72.23%                | context.AfterFunc
                                         4679.02MB 73.38% |   context.(*cancelCtx).propagateCancel
----------------------------------------------------------+-------------
                                         1234.59MB   100% |   context.WithCancel (inline)
 1234.59MB 13.99% 86.22%  1234.59MB 13.99%                | context.withCancel
----------------------------------------------------------+-------------
                                          349.20MB   100% |   github.com/lightningnetwork/lnd/fn/v2.(*ContextGuard).ctxQuitUnsafe (inline)
  349.20MB  3.96% 90.17%   349.20MB  3.96%                | github.com/lightningnetwork/lnd/fn/v2.(*ContextGuard).addCancelFnUnsafe
----------------------------------------------------------+-------------
                                         6975.90MB   100% |   github.com/lightningnetwork/lnd/fn/v2.(*ContextGuard).Create
     250MB  2.83% 93.01%  6975.90MB 79.02%                | github.com/lightningnetwork/lnd/fn/v2.(*ContextGuard).ctxQuitUnsafe
                                         6376.70MB 91.41% |   context.AfterFunc
                                          349.20MB  5.01% |   github.com/lightningnetwork/lnd/fn/v2.(*ContextGuard).addCancelFnUnsafe (inline)
----------------------------------------------------------+-------------
                                         1451.60MB   100% |   github.com/lightningnetwork/lnd/fn/v2.(*ContextGuard).Create
     217MB  2.46% 95.46%  1451.60MB 16.44%                | context.WithCancel
                                         1234.59MB 85.05% |   context.withCancel (inline)
----------------------------------------------------------+-------------
                                           41.51MB 81.37% |   github.com/lightningnetwork/lnd/graph/db.(*KVStore).FilterChannelRange.func1
                                            9.50MB 18.63% |   github.com/btcsuite/btcwallet/walletdb/bdb.(*db).View
   28.51MB  0.32% 95.79%    51.01MB  0.58%                | github.com/lightningnetwork/lnd/graph/db.deserializeChanEdgeInfo
----------------------------------------------------------+-------------
                                          136.93MB   100% |   github.com/lightningnetwork/lnd/discovery.(*GossipSyncer).replyPeerQueries
   14.95MB  0.17% 95.96%   136.93MB  1.55%                | github.com/lightningnetwork/lnd/discovery.(*GossipSyncer).replyChanRangeQuery
                                             112MB 81.79% |   github.com/lightningnetwork/lnd/discovery.(*ChanSeries).FilterChannelRange
----------------------------------------------------------+-------------
                                           90.62MB   100% |   github.com/btcsuite/btclog/v2.(*sLogger).Tracef
   14.23MB  0.16% 96.12%    90.62MB  1.03%                | fmt.Sprintf
                                           77.39MB 85.40% |   fmt.(*pp).doPrintf
----------------------------------------------------------+-------------
                                          110.74MB   100% |   github.com/btcsuite/btcwallet/walletdb/bdb.(*db).View
    9.23MB   0.1% 96.22%   110.74MB  1.25%                | github.com/lightningnetwork/lnd/graph/db.(*KVStore).FilterChannelRange.func1
                                           41.51MB 37.48% |   github.com/lightningnetwork/lnd/graph/db.deserializeChanEdgeInfo
----------------------------------------------------------+-------------
                                           54.68MB   100% |   github.com/davecgh/go-spew/spew.(*dumpState).dump
    6.61MB 0.075% 96.30%    54.68MB  0.62%                | github.com/davecgh/go-spew/spew.(*dumpState).dumpSlice
                                           36.60MB 66.94% |   github.com/davecgh/go-spew/spew.(*dumpState).dump
----------------------------------------------------------+-------------
                                           56.18MB   100% |   github.com/davecgh/go-spew/spew.(*dumpState).dumpPtr
                                           56.18MB   100% |   github.com/davecgh/go-spew/spew.fdump
                                           36.60MB 65.15% |   github.com/davecgh/go-spew/spew.(*dumpState).dumpSlice
       5MB 0.057% 96.35%    56.18MB  0.64%                | github.com/davecgh/go-spew/spew.(*dumpState).dump
                                           56.18MB   100% |   github.com/davecgh/go-spew/spew.(*dumpState).dumpPtr
                                           54.68MB 97.33% |   github.com/davecgh/go-spew/spew.(*dumpState).dumpSlice
----------------------------------------------------------+-------------
                                             112MB   100% |   github.com/lightningnetwork/lnd/discovery.(*ChanSeries).FilterChannelRange
    1.26MB 0.014% 96.37%      112MB  1.27%                | github.com/lightningnetwork/lnd/graph/db.(*KVStore).FilterChannelRange
                                          110.74MB 98.88% |   github.com/lightningnetwork/lnd/kvdb.View (inline)
----------------------------------------------------------+-------------
                                           77.39MB   100% |   fmt.Sprintf
         0     0% 96.37%    77.39MB  0.88%                | fmt.(*pp).doPrintf
                                           77.39MB   100% |   fmt.(*pp).printArg
----------------------------------------------------------+-------------
                                           77.39MB   100% |   fmt.(*pp).printArg
         0     0% 96.37%    77.39MB  0.88%                | fmt.(*pp).handleMethods
                                           67.98MB 87.85% |   github.com/lightningnetwork/lnd/lnutils.LogClosure.String
----------------------------------------------------------+-------------
                                           77.39MB   100% |   fmt.(*pp).doPrintf
         0     0% 96.37%    77.39MB  0.88%                | fmt.(*pp).printArg
                                           77.39MB   100% |   fmt.(*pp).handleMethods
----------------------------------------------------------+-------------
                                           90.12MB 99.45% |   github.com/lightningnetwork/lnd/peer.(*Brontide).logWireMessage
         0     0% 96.37%    90.62MB  1.03%                | github.com/btcsuite/btclog/v2.(*sLogger).Tracef
                                           90.62MB   100% |   fmt.Sprintf
----------------------------------------------------------+-------------
                                          165.25MB 91.14% |   github.com/lightningnetwork/lnd/kvdb.View
         0     0% 96.37%   181.32MB  2.05%                | github.com/btcsuite/btcwallet/walletdb/bdb.(*db).View
                                          110.74MB 61.08% |   github.com/lightningnetwork/lnd/graph/db.(*KVStore).FilterChannelRange.func1
                                            9.50MB  5.24% |   github.com/lightningnetwork/lnd/graph/db.deserializeChanEdgeInfo
----------------------------------------------------------+-------------
                                           56.18MB   100% |   github.com/davecgh/go-spew/spew.(*dumpState).dump
         0     0% 96.37%    56.18MB  0.64%                | github.com/davecgh/go-spew/spew.(*dumpState).dumpPtr
                                           56.18MB   100% |   github.com/davecgh/go-spew/spew.(*dumpState).dump
----------------------------------------------------------+-------------
                                           67.98MB   100% |   github.com/lightningnetwork/lnd/peer.(*Brontide).logWireMessage.SpewLogClosure.func2
         0     0% 96.37%    67.98MB  0.77%                | github.com/davecgh/go-spew/spew.Sdump
                                           56.18MB 82.64% |   github.com/davecgh/go-spew/spew.fdump
----------------------------------------------------------+-------------
                                           56.18MB   100% |   github.com/davecgh/go-spew/spew.Sdump
         0     0% 96.37%    56.18MB  0.64%                | github.com/davecgh/go-spew/spew.fdump
                                           56.18MB   100% |   github.com/davecgh/go-spew/spew.(*dumpState).dump
----------------------------------------------------------+-------------
                                          988.40MB   100% |   github.com/lightningnetwork/lnd/discovery.(*AuthenticatedGossiper).splitAndSendAnnBatch.func2
         0     0% 96.37%   988.40MB 11.20%                | github.com/lightningnetwork/lnd/discovery.(*AuthenticatedGossiper).sendRemoteBatch
                                          988.40MB   100% |   github.com/lightningnetwork/lnd/discovery.(*GossipSyncer).FilterGossipMsgs
----------------------------------------------------------+-------------
         0     0% 96.37%   988.40MB 11.20%                | github.com/lightningnetwork/lnd/discovery.(*AuthenticatedGossiper).splitAndSendAnnBatch.func2
                                          988.40MB   100% |   github.com/lightningnetwork/lnd/discovery.(*AuthenticatedGossiper).sendRemoteBatch
----------------------------------------------------------+-------------
                                             112MB   100% |   github.com/lightningnetwork/lnd/discovery.(*GossipSyncer).replyChanRangeQuery
         0     0% 96.37%      112MB  1.27%                | github.com/lightningnetwork/lnd/discovery.(*ChanSeries).FilterChannelRange
                                             112MB   100% |   github.com/lightningnetwork/lnd/graph/db.(*KVStore).FilterChannelRange
----------------------------------------------------------+-------------
         0     0% 96.37%  1415.90MB 16.04%                | github.com/lightningnetwork/lnd/discovery.(*GossipSyncer).ApplyGossipFilter.func2
                                         1415.90MB   100% |   github.com/lightningnetwork/lnd/fn/v2.(*ContextGuard).Create
----------------------------------------------------------+-------------
                                          988.40MB   100% |   github.com/lightningnetwork/lnd/discovery.(*AuthenticatedGossiper).sendRemoteBatch
         0     0% 96.37%   988.40MB 11.20%                | github.com/lightningnetwork/lnd/discovery.(*GossipSyncer).FilterGossipMsgs
                                          988.40MB   100% |   github.com/lightningnetwork/lnd/fn/v2.(*ContextGuard).Create
----------------------------------------------------------+-------------
         0     0% 96.37%  6156.50MB 69.74%                | github.com/lightningnetwork/lnd/discovery.(*GossipSyncer).replyHandler
                                         6156.50MB   100% |   github.com/lightningnetwork/lnd/discovery.(*GossipSyncer).replyPeerQueries
----------------------------------------------------------+-------------
                                         6156.50MB   100% |   github.com/lightningnetwork/lnd/discovery.(*GossipSyncer).replyHandler
         0     0% 96.37%  6156.50MB 69.74%                | github.com/lightningnetwork/lnd/discovery.(*GossipSyncer).replyPeerQueries
                                         6019.57MB 97.78% |   github.com/lightningnetwork/lnd/discovery.(*GossipSyncer).replyShortChanIDs
                                          136.93MB  2.22% |   github.com/lightningnetwork/lnd/discovery.(*GossipSyncer).replyChanRangeQuery
----------------------------------------------------------+-------------
                                         6019.57MB   100% |   github.com/lightningnetwork/lnd/discovery.(*GossipSyncer).replyPeerQueries
         0     0% 96.37%  6019.57MB 68.19%                | github.com/lightningnetwork/lnd/discovery.(*GossipSyncer).replyShortChanIDs
                                         6019.57MB   100% |   github.com/lightningnetwork/lnd/fn/v2.(*ContextGuard).Create
----------------------------------------------------------+-------------
                                         6019.57MB 71.43% |   github.com/lightningnetwork/lnd/discovery.(*GossipSyncer).replyShortChanIDs
                                         1415.90MB 16.80% |   github.com/lightningnetwork/lnd/discovery.(*GossipSyncer).ApplyGossipFilter.func2
                                          988.40MB 11.73% |   github.com/lightningnetwork/lnd/discovery.(*GossipSyncer).FilterGossipMsgs
         0     0% 96.37%  8427.50MB 95.46%                | github.com/lightningnetwork/lnd/fn/v2.(*ContextGuard).Create
                                         6975.90MB 82.78% |   github.com/lightningnetwork/lnd/fn/v2.(*ContextGuard).ctxQuitUnsafe
                                         1451.60MB 17.22% |   context.WithCancel
----------------------------------------------------------+-------------
                                          110.74MB 67.01% |   github.com/lightningnetwork/lnd/graph/db.(*KVStore).FilterChannelRange (inline)
         0     0% 96.37%   165.25MB  1.87%                | github.com/lightningnetwork/lnd/kvdb.View
                                          165.25MB   100% |   github.com/btcsuite/btcwallet/walletdb/bdb.(*db).View
----------------------------------------------------------+-------------
                                           67.98MB   100% |   fmt.(*pp).handleMethods
         0     0% 96.37%    67.98MB  0.77%  

[MegalithicBTC]

Output of pprof "top":

File: lnd
Type: inuse_space
Time: 2025-06-02 06:24:47 PDT
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 8680.12MB, 95.73% of 9067.57MB total
Dropped 452 nodes (cum <= 45.34MB)
Showing top 10 nodes out of 34
      flat  flat%   sum%        cum   cum%
 3043.85MB 33.57% 33.57%  4807.53MB 53.02%  context.(*cancelCtx).propagateCancel
 1763.69MB 19.45% 53.02%  1763.69MB 19.45%  context.(*cancelCtx).Done
 1743.69MB 19.23% 72.25%  6551.22MB 72.25%  context.AfterFunc
 1268.60MB 13.99% 86.24%  1268.60MB 13.99%  context.withCancel (inline)
  356.27MB  3.93% 90.17%   356.27MB  3.93%  github.com/lightningnetwork/lnd/fn/v2.(*ContextGuard).addCancelFnUnsafe (inline)
     259MB  2.86% 93.02%  7166.49MB 79.03%  github.com/lightningnetwork/lnd/fn/v2.(*ContextGuard).ctxQuitUnsafe
  224.50MB  2.48% 95.50%  1493.10MB 16.47%  context.WithCancel
   15.53MB  0.17% 95.67%    98.24MB  1.08%  fmt.Sprintf
    3.50MB 0.039% 95.71%    64.11MB  0.71%  github.com/davecgh/go-spew/spew.(*dumpState).dump
    1.50MB 0.017% 95.73%    64.11MB  0.71%  github.com/davecgh/go-spew/spew.(*dumpState).dumpPtr
(pprof) 

[ziggie1984]

could you share the pprof file or the graphical representation of the heap ?

[MegalithicBTC]

@ziggie1984 shared via private message, let me know if I can help with anything else

[Elbandi]

maybe this line: ctx, _ := g.cg.Create(context.Background())

the ignored cancel fn is added to a map, but newer removed.

[ziggie1984]

This is already fixed in current master: 2a5235a#diff-5321d5dad7ab003eff5e595aef273619fd9c22586f28f1b141940932b93c6ec7L1264

need to cherry-pick for 19.1.

[ziggie1984]

For better visualization: