Initially, peer-observer did only extract data from the tracing / eBPF interface. The ebpf-extractor hooks into the tracepoints and passes the events on to tools which then process these events (e.g. create prometheus metrics, publish them as JSON via a websocket for web visualizations, … ). This works well for everything that needs realtime events.
To supplement the real-time event data, I added an RPC-extractor in August with getpeerinfo have staleful data about the connected peers that we can’t get from the tracepoints alone. For example:
- how many connections to spy nodes or nodes on a banlist does the node have?
- what share of connections are connected via BIP324 v2 transport connections?
- how does the mean/median Bitcoin protocol ping to my connections change over time?
- how many peers relay sub-1sat/vbyte transactions?
- …
While only getpeerinfo is implemented for now, there are a bunch of other RPCs that would be useful to have in there. A few examples are listed in rpc-extractor: add more RPC (uptime, getmemoryinfo, ...) · Issue #199 · 0xB10C/peer-observer · GitHub and I also want to explore how to add WIP getpeerinfo fields like cpu_load in there rpc-extractor: explore adding temporary fields and RPCs like `cpu_load` from bitcoin/bitcoin #31672 · Issue #200 · 0xB10C/peer-observer · GitHub.
Recently, I’ve been thinking about how to effectively detect P2P DoS attacks or anomalies (i.e. bugs). While I run a process-exporter to collect data on how much time is spent in e.g. the b-msghand thread, an alternative might to also track the time it takes for the node to respond to a ping via the P2P network (metrics tool: track time it takes for us to respond to an inbound ping with a pong · Issue #212 · 0xB10C/peer-observer · GitHub). This has been a good DoS indicator in Notes on 'DoS due to inv-to-send sets growing too large' from May 2023 since pings are handled in queue with all other messages. It measures processing backlog and network latency. For this, I’ve started working on a p2p-extractor that frequently pings the node from localhost (to minimize network latency) and publishes the time it takes for a pong to arrive. This can then be used in alerting.
As part of Implement more extractors · Issue #141 · 0xB10C/peer-observer · GitHub, I’ve also been thinking about a log-extractor similar to the one used in bmon. However, I’ll probably first explore an IPC-based extractor - that might possibly even replace the ebpf / tracing extractor as it should resolve some of the painpoints of the eBPF based tracing interface (see Tracepoint-like interface via libmultiprocess and IPC communication · Issue #185 · bitcoin-core/libmultiprocess · GitHub and POC: IPC tracing interface by ryanofsky · Pull Request #32898 · bitcoin/bitcoin · GitHub).
In other news, I’ve recently added a Knots node called nico to my infrastructure (the others are all Bitcoin Core). Since people are using it, it makes sense to include it in the monitoring too.