Postmortem: 2019-08-07 Tendermint KMS-related Cosmos Hub Validator Incident
Last night and early this morning we encountered a series of small outages in iqlusion’s Cosmos Hub validator related to a recently released version of Tendermint KMS: v0.6.1. We are the primary contributors to Tendermint KMS and generally try to run the latest version of the code at all times, in order to smoke test each release prior to a wider announcement. Generally this has been going pretty well and we have not had an outage like this before.
Unfortunately, while the v0.6.1 release appeared to work during the day, to the point we made a release announcement about it on Twitter, as sometimes happens issues didn’t crop up until late at night and early next morning. We missed around 200 blocks when the KMS crashed originally last night, and another 300 when it crashed again this morning. We weren’t alone, there were two issues opened by people who encountered the same problems on...
Continue reading →