Per-peer memory usage

Implementation wg-blockrelay
1

This is a thread to discuss the memory usage of peers.

Interesting things:

  • average case / expected memory usage per peer
  • difference in average case memory usage for block-relay-only peer vs full-relay peer
  • worst case memory usage for a single peer
  • worst case memory usage for a block-relay-only peer vs full-relay peer
  • how different network conditions impact the expected memory usage per peer

Memory monitoring:

  • I have written a patch to monitor the memory usage of different types of connections. It currently reports (1) current CNode memory usage (2) current Peer memory usage & (3) max Peer memory usage.
  • Status: the broad strokes are in place, but there are some fields that are not yet accounted for (details in PR description). Also, I would love review to know if the code actually does what I think it does!
  • Desired improvement: update the max peer memory usage to incorporate CNode memory, so it will be a better representation of the entire connection.
  • This spreadsheet are the results from running the patch on my node with inbounds enabled for approximately 5 days (nov 10-15). it shows a dramatic difference between the max memory usage of block-relay-only peers vs full-relay peers.

this graph shows memory usage broken down by max & current, for connections with relay enabled or not. y axis: number of bytes, x axis: different nodes.

Screen Shot 2023-11-15 at 7.11.34 PM
Screen Shot 2023-11-15 at 7.11.34 PM1324×812 36.1 KB

Warnet:

Warnet can help us observe different network conditions. Here are some questions I’m curious about, where I think warnet can create/isolate behaviors we sometimes see:

  • When LinkingLion is on the prowl, there is high churn of inbound connections. How much memory does each short-lived connections use?
  • Especially with ordinals/transcriptions, we sometimes see really high transaction volume. How does that impact the expected/average memory usage of our full-relay connections?

Do you have more ideas? :slight_smile:

6 Likes
2

Thanks for sharing this! I think the labels for max_no_relay and cur_no_relay are mixed up (max is lower than current memory usage). If I’m reading this correctly it’s just below 1 MB per full-relay peer?

Curious about this too! I’m seeing more than a 100 connections per minute to a few of my nodes. I’m wondering when the memory allocations are made. When the peer connects or during connection lifetime. I think observing memory usage over time would be interesting here too!

2 Likes
3

The maximums in that graph counts just the Peer memory usage; the current counts both Peer and CNode

3 Likes
4

Looks great! Even though the analysis isn’t complete yet (some field not yet accounted for as you mention), I’m just trying to come up with a few high-level takeaways so far.

  • Dynamic memory usage per tx-relaying peers is ~600kb - 1.4MB per tx-relaying peer, a large part of it is probably the rolling bloom filter m_tx_inventory_known_filter.
  • Block-Relay-Only peers only take up a very small fraction of that memory (<10kb)
  • CNode average memory is small in comparison with Peer on average - however, I’d expect hat there would be large spikes when we send / receive a large message that we haven’t processed yet - these likely wouldn’t be seen in this data.
  • CNode memory considerations are mostly independent on whether the connection is block-relay-only or not.

Does this make sense? Any important takeaways so far that I am missing?

3 Likes