Impact
Some Solana tokens & pools reported incorrect values for liquidity & price. filterTokens had incorrect liquidity values, and getTokenPrices had intermittently null values returned.
Timeline
At 23:21 UTC March 31st a release to the Solana data ingestion project was deployed, this included a performance boost, by fetching our confirmed block data through geyser GRPC instead of standard RPC methods. This change was thoroughly tested in our staging environment, and passed all automated tests.
After this change was deployed, the vast majority of the data was accurate and all systems appeared green, and arrived ~200ms faster in most cases.
However, after the change, some Solana pools had incorrect reserves set due to slight differences in some of the ingested data between the two datasources, which was not caught through testing. Dramatic reserve swings are expected as pools get drained/withdrawn from all the time, so none of our alerting caught anything. The issue was that these pairs actually did have reserves, and we were incorrectly reporting it.
After some reports from customers about liquidity data missing, and prices being incorrectly reported, the team started investigations at 17:00 UTC April 1st, discovered the issue and reverted back to the previous method of ingesting data, mitigating the issue.
What we are doing about it
We’ve outlined some much better methods of automated testing using runtime schema enforcement that will catch this class of error in the future, and will be working on that right away.
We’re also implementing a new set of alerts that will find and notify our on-call staff when any data inconsistencies of this type occur in the future.
The team is making the changes to correctly and safely implement the new, faster method of ingesting block data, and will deploy that once we are sure it functions end to end without any issues.
No components marked as affected
Resolved
Impact
Some Solana tokens & pools reported incorrect values for liquidity & price. filterTokens had incorrect liquidity values, and getTokenPrices had intermittently null values returned.
Timeline
At 23:21 UTC March 31st a release to the Solana data ingestion project was deployed, this included a performance boost, by fetching our confirmed block data through geyser GRPC instead of standard RPC methods. This change was thoroughly tested in our staging environment, and passed all automated tests.
After this change was deployed, the vast majority of the data was accurate and all systems appeared green, and arrived ~200ms faster in most cases.
However, after the change, some Solana pools had incorrect reserves set due to slight differences in some of the ingested data between the two datasources, which was not caught through testing. Dramatic reserve swings are expected as pools get drained/withdrawn from all the time, so none of our alerting caught anything. The issue was that these pairs actually did have reserves, and we were incorrectly reporting it.
After some reports from customers about liquidity data missing, and prices being incorrectly reported, the team started investigations at 17:00 UTC April 1st, discovered the issue and reverted back to the previous method of ingesting data, mitigating the issue.
What we are doing about it
We’ve outlined some much better methods of automated testing using runtime schema enforcement that will catch this class of error in the future, and will be working on that right away.
We’re also implementing a new set of alerts that will find and notify our on-call staff when any data inconsistencies of this type occur in the future.
The team is making the changes to correctly and safely implement the new, faster method of ingesting block data, and will deploy that once we are sure it functions end to end without any issues.
