Influx of Solana data put stress on pipeline. Improvements to be made.

Impact
Some Solana tokens & pools reported incorrect values for liquidity & price. filterTokens had incorrect liquidity values, and getTokenPrices had intermittently null values returned.

Timeline

At 23:21 UTC March 31st a release to the Solana data ingestion project was deployed, this included a performance boost, by fetching our confirmed block data through geyser GRPC instead of standard RPC methods. This change was thoroughly tested in our staging environment, and passed all automated tests.
After this change was deployed, the vast majority of the data was accurate and all systems appeared green, and arrived ~200ms faster in most cases.
However, after the change, some Solana pools had incorrect reserves set due to slight differences in some of the ingested data between the two datasources, which was not caught through testing. Dramatic reserve swings are expected as pools get drained/withdrawn from all the time, so none of our alerting caught anything. The issue was that these pairs actually did have reserves, and we were incorrectly reporting it.
After some reports from customers about liquidity data missing, and prices being incorrectly reported, the team started investigations at 17:00 UTC April 1st, discovered the issue and reverted back to the previous method of ingesting data, mitigating the issue.

What we are doing about it

We’ve outlined some much better methods of automated testing using runtime schema enforcement that will catch this class of error in the future, and will be working on that right away.

We’re also implementing a new set of alerts that will find and notify our on-call staff when any data inconsistencies of this type occur in the future.

The team is making the changes to correctly and safely implement the new, faster method of ingesting block data, and will deploy that once we are sure it functions end to end without any issues.

Balances delay has been resolved. Up-to-date data is now available.

All services are operational.

Our team is continuing the recovery effort beyond this resolution, and will be monitoring closely as we work through a port mortem to determine next steps/prevention.

All services are operational.

All services are operational.

We have resolved the degraded FilterTokens performance. All systems are operational.

Experienced a back up in the processing queue. We’ve caught up now and are implementing some optimizations to avoid similar issues in the future.

Current and historic prices have been resolved since the Base pricing degradation between 8:25PM UTC and 12:37AM UTC.

There is minimal cleanup of historical outliers ongoing, but the root issue and the majority of affected historical data has been corrected.