#17 relayers are failing with the prysm client post Capella/Shanghai ( 2023/04/12 - mainnet)

Date: 2023/04/12

Authors: Terence Tsao, Potuz, Nishant Das, James He

Status: Rootcause found and fix released. https://github.com/prysmaticlabs/prysm/releases/tag/v4.0.2

Network: Mainnet

Summary: Blinded capella blocks missed populating the new field BlsToExecutionChanges while unblinding the builder (aka relay) blinded block. Missing the bls changes resulted in invalid blocks and affects all prysm nodes connected with relays. Proposers will not be able to propose blocks. The inconsistency of submitting BLS messages (typically mostly at the fork) combined with the complexity of the builder setup and generic error resulted in difficult detection in the testnets.

Impact: All validators that run Prysm using MEV-Boost who propose blocks that have greater than 0 bls-to-exec-changes in them. On avg 40% [1] of the network uses Prysm. 80% of the network uses MEV-Boost [2]. Which translates to 5-6 missing blocks every epoch.

Detection: Lido first asked us to investigate a missing block

Action Items:

Action Item	Type	Owner	Relevant Link	Notes
Produce Fix	code	Terence		Done
Review PR	review	Potuz, Nishant, James, Kasey		Done
Direct Users	communication	Team	Discord, Lido, Rocketpool, Twitter, Flashbots	Done
Determine Release	decision	Team	https://github.com/prysmaticlabs/prysm/releases/tag/v4.0.2	Done
Release Fix	devops	Team	https://github.com/prysmaticlabs/prysm/releases/tag/v4.0.2	Done
E2E Testing With Builders	Feature Addition	Nishant
Investigate builder setting caching bug	Investigate	James
Validators that remove all relay/builder related flags from their beacon nodes and validators will still access builder paths which cause panics. Validators that have registered mev boost cannot unregister	Requires Fix	Team

| this panic appears to be happening in the validator client calling getPayloadHeaderFromBuilder | | UX improvement potential with —enable-builder flag in combination with —proposer-settings flag | Discussion | James | | Create a guide for enabling and disabling builder settings | | Better release structure. Preston is the only person that can sign the release. It becomes a blocker for critical recovery | devops | Preston and team | | | | Increase the number of builder enabled validators run by the Prysm team significantly (10% → 80%) so that any builder issues are much more visible in testnets. | devops | Team | | | | Start having separate builders run by the team so that mismatches between proposers and builders are more visible and allow easier insight into possible bugs on the prysm’s builder client code path. | devops | Team | | |

What went wrong

All Prysm validators using MEV-Boost failed to propose blocks, which resulted in missing block rewards and lower attestation inclusion. The failure mode could have lasted for 4-5 days given there were around 300k pending bls to exec changes to be cleared out.
An error occurred in a testing phase for a different reason and by the time the issue was resolved there were no more BLS messages in the pool to be tested against.
Happy path hive tests for the builder did not work as expected and missed catching this issue.
Unit tests did not fully populate this particular value, the error occurs when the array is above 0 length.

Where we got lucky

MEV-Boost circuit breaker triggered at a condition where missing five slots per epoch. This means this issue would have been worse if it weren’t for the circuit breaker. Probably close to eight slots per epoch.
The Flashbots and Ultra Sound teams reacted quickly and disabled Prysm support from most relayers. This meant that for any Prysm getHeader request, the relayer would look at the user agent string and give no response if the header request came from Prysm. Gnosis’ agnostic relayer operators were asleep and could not short-circuit Prysm request. Any MEV-Boost with an agonistic relayer would have exhibited the same failing behavior.

Lessons Learned

Unit tests for new fields on Go structures need to explicitly populate them with non-default values. This was a critical failure on a rarely tested path that could have been prevented by a simple complete unit test. We need to research on static analyzers or similar that would check coverage automatically.