Per-peer memory usage

This is a thread to discuss the memory usage of peers.

Interesting things:

  • average case / expected memory usage per peer
  • difference in average case memory usage for block-relay-only peer vs full-relay peer
  • worst case memory usage for a single peer
  • worst case memory usage for a block-relay-only peer vs full-relay peer
  • how different network conditions impact the expected memory usage per peer

Memory monitoring:

  • I have written a patch to monitor the memory usage of different types of connections. It currently reports (1) current CNode memory usage (2) current Peer memory usage & (3) max Peer memory usage.
  • Status: the broad strokes are in place, but there are some fields that are not yet accounted for (details in PR description). Also, I would love review to know if the code actually does what I think it does!
  • Desired improvement: update the max peer memory usage to incorporate CNode memory, so it will be a better representation of the entire connection.
  • This spreadsheet are the results from running the patch on my node with inbounds enabled for approximately 5 days (nov 10-15). it shows a dramatic difference between the max memory usage of block-relay-only peers vs full-relay peers.

this graph shows memory usage broken down by max & current, for connections with relay enabled or not. y axis: number of bytes, x axis: different nodes.


Warnet can help us observe different network conditions. Here are some questions I’m curious about, where I think warnet can create/isolate behaviors we sometimes see:

  • When LinkingLion is on the prowl, there is high churn of inbound connections. How much memory does each short-lived connections use?
  • Especially with ordinals/transcriptions, we sometimes see really high transaction volume. How does that impact the expected/average memory usage of our full-relay connections?

Do you have more ideas? :slight_smile:


Thanks for sharing this! I think the labels for max_no_relay and cur_no_relay are mixed up (max is lower than current memory usage). If I’m reading this correctly it’s just below 1 MB per full-relay peer?

Curious about this too! I’m seeing more than a 100 connections per minute to a few of my nodes. I’m wondering when the memory allocations are made. When the peer connects or during connection lifetime. I think observing memory usage over time would be interesting here too!

1 Like

The maximums in that graph counts just the Peer memory usage; the current counts both Peer and CNode


Looks great! Even though the analysis isn’t complete yet (some field not yet accounted for as you mention), I’m just trying to come up with a few high-level takeaways so far.

  • Dynamic memory usage per tx-relaying peers is ~600kb - 1.4MB per tx-relaying peer, a large part of it is probably the rolling bloom filter m_tx_inventory_known_filter.
  • Block-Relay-Only peers only take up a very small fraction of that memory (<10kb)
  • CNode average memory is small in comparison with Peer on average - however, I’d expect hat there would be large spikes when we send / receive a large message that we haven’t processed yet - these likely wouldn’t be seen in this data.
  • CNode memory considerations are mostly independent on whether the connection is block-relay-only or not.

Does this make sense? Any important takeaways so far that I am missing?