Ethereum Network Struggles With Missed Slots, bloXroute and Lighthouse In Debate

Earlier this week, the Ethereum network experienced a significant surge in missed slots, predominantly originating from blocks relayed by bloXroute relays. Investigations revealed that while the bloXroute relays effectively published blocks and blobs, the rapid propagation of blocks through the BloXroute Distributed Network (BDN) stood in contrast with the slower propagation of blobs via peer-to-peer (p2p) channels. This discrepancy highlighted a specific Client (CL) behavior, leading to client rejection of blocks and subsequent missed slots.

Resolving Ethereum Missed Slots With bloXroute

BloXroute Labs CEO Uri Klarman explained in a detailed thread on Github, what transpired about the missed slots in Ethereum.

In the current Lighthouse version, nodes expect the same peer providing the block to also supply the blobs. However, as the BDN does not propagate blobs, consensus nodes connected to the BDN disregard blocks initially received from it. A recent BDN release aimed to expedite block propagation without blobs, relying on the p2p network to disseminate blobs as necessary. This change inadvertently led to a significant uptick in missed slots.

Klarman explained that the BDN heavily relies on Lighthouse, which constitutes the majority of beacon nodes at bloXroute. Initial post-release observations indicated successful block propagation through the BDN, primarily affecting bloXroute relays due to their close integration with the BDN.

To address the issue, a series of tests were conducted, isolating the problem to Lighthouse’s behavior when encountering blocks first through the BDN. Measures were taken to gradually transition relays away from utilizing the BDN for block publishing and subsequently disabling the BDN’s block propagation containing blobs.

Throughout this period, bloXroute relays continued providing blocks with blobs to validators and publishing blocks with blobs to the BDN and the network of beacon nodes. However, these publish requests returned a 202 response as beacon nodes had already received the block from the BDN.

Lighthouse Chief Responds to the Allegations

Michael Sproul, who’s the chief at Lighthouse, has criticized Klarman’s take on the missed slots, claiming it misrepresents an issue with the Lighthouse p2p bug, which he asserts was caused by an untested interaction between Bloxroute’s centralised “block distribution network” (BDN) and Lighthouse’s HTTP API.

This post-mortem misrepresents the issue as a Lighthouse p2p bug, when in fact it was caused by an untested interaction between Bloxroute's centralised "block distribution network" (BDN) and Lighthouse's HTTP API

Here's an account from my perspective https://t.co/T2i9dbI2zQ

— Michael Sproul (@sproulM_) March 29, 2024

Sproul alleges that Bloxroute has been uncooperative during the incident, refusing to share logs to support their claims. He argues that the premature post-mortem was conducted before necessary information could be obtained from Bloxroute.

According to Sproul, the issue arose from Bloxroute publishing blocks without blobs on the p2p network via the BDN and then attempting to fill in the blobs by POSTing them to Lighthouse as part of an HTTP request. Lighthouse and Prysm’s HTTP APIs, however, assumed that blocks were sent on p2p with blobs in their entirety. Sproul contends that this assumption was not valid in the presence of a “block distribution network” that bypasses the normal flow for publishing blocks.

To address the issue, Sproul suggests short-term solutions such as turning off the BDN when blobs are involved, and long-term solutions like restructuring the PBS ecosystem to prevent similar failures. He also expresses his personal opinion that the BDN is a technology that should be obsoleted due to its centralized nature and potential risks to Ethereum’s decentralization.