The burst of unusual activity abated; additional resources were deployed to help services swiftly return to nominal.

resposne times have returned to normal

A burst of unusual activity caused nearly an hour of degraded endpoint performance. The team's efforts to mitigate the impact has allowed services to recover. We continue to investigate avenues to harden our services to prevent similar issues from occurring in future.

We have finished our monitoring phase, the outage is now fully resolved.

Robinhood network has returned to a healthy state.

The Robinhood Chain is experiencing a known issue where delayed transactions can be incorrectly filtered, causing downstream RPC failure modes. The Robinhood team is aware of the issue and is working on a fix.

We have deployed a temporary workaround that will dramatically reduce the impact on our indexing.

Services are back to normal.

Unusual spikes of network activity on Base and Solana caused a temporary delay in the freshness of our published data. Underlying problem was found and has been fixed.

Robinhood balances are up to date.

All systems have recovered

Base network is unstuck and everything is caught up.

Symptoms

Starting at 17:28 UTC most token data (enhanced events, aggregates, both pair and token etc) were degraded. They would delay for ~1-2 minutes and then catch back up.

Causes

We found quickly that this was an issue with our kafka producers timing out when trying to produce messages. When they time out, it causes services to backup. Specifically, our services that generate lots of kafka messages (price, walletBalances, enhancer) would get behind.

A rolling security patch on the AWS MSK Kafka cluster caused broker reboots, which triggered a connection leak in the shared Kafka producer service. This resulted in an exhaustion of broker connections, leading to failures when publishing price data.

Mitigation

We performed a rolling restart of the producer services to clear leaked connections and restore timely price data updates.

Fix

An upgrade to @platformatic/kafka v2.2.3 is planned to permanently fix the connection leak. We will also perform better error handling and async processing here to lessen the impact.

Token balances are now processing up-to-date.