By Tony Arcieri

The iqlusion team has just released a new version of Tendermint KMS (tmkms): v0.8.0. Tendermint KMS is a signature service with support for Hardware Security Modules (HSMs), such as YubiHSM2 and Ledger Nano . It’s intended to be run alongside Cosmos Validators, ideally on separate physical hosts, providing defense-in-depth for online validator signing keys, double signing protection, and functioning as a central signing service that can be used when operating multiple validators in several Cosmos Zones.

This release includes some interesting new features which we’re excited to share with you, along with important bugfixes.

For installation instructions and more details, please see the Tendermint KMS repository on GitHub:

https://github.com/iqlusioninc/tmkms

Transaction Signing Support (alpha)

The most exciting feature in this release is newly added support for...

By Tony Arcieri

Today we experienced our worst Cosmos Hub Validator outage yet: we were down for approximately 4 hours. In this post, we’d like to share the details of that outage with you, along with our plans for avoiding future outages and some clues to what went wrong which may be helpful to other validator operators.

Let’s begin with a timeline.

Incident timeline

All times are in PST (i.e. this was a 5AM outage):

• 04:54: outage detected by Hubble
• 04:57: Zaki is notified of outage by Adriana Kalpa, confirms outage via Hubble, and alerts the rest of the team. Unfortunately the rest of the team remained asleep for the next 3 hours
• 07:42: Tony begins responding to outage
• 07:53: Outage detected by our internal monitoring and first PagerDuty alert is sent
• 08:27: Sentries isolated as the cause of the problem and fixed. Validator begins syncing back up
• 08:52: Signing resumed

• By Zaki Manian

Iqlusion has been helping Oasis get ready for the launch of it’s first public testnet for the last couple months. Getting to public testnets is a major milestone for the project and I’m excited about everything Tendermint powered that reaches the market.

I’m particularly excited about Oasis because it’s the first instance of.a blockchain architecture that I think will represent a fruitful direction in blockchain architecture. In 2014, the blockchain community started exploring doing state machine replication in blockchain architecture rather than just time stamping/data availability system that is Bitcoin. Ethereum has demonstrated the potential of this architecture and Tendermint is a general purpose state machine replication engine that has been widely adopted.

But the SMR paradigm leaves us wanting in some ways. The strong fault tolerance is great for staking...

Last night and early this morning we encountered a series of small outages in iqlusion’s Cosmos Hub validator related to a recently released version of Tendermint KMS: v0.6.1. We are the primary contributors to Tendermint KMS and generally try to run the latest version of the code at all times, in order to smoke test each release prior to a wider announcement. Generally this has been going pretty well and we have not had an outage like this before.

Unfortunately, while the v0.6.1 release appeared to work during the day, to the point we made a release announcement about it on Twitter, as sometimes happens issues didn’t crop up until late at night and early next morning. We missed around 200 blocks when the KMS crashed originally last night, and another 300 when it crashed again this morning. We weren’t alone, there were two issues opened by people who encountered the same problems on...

By Tony Arcieri

Earlier this month we released Abscissa: our security-oriented Rust application framework. After releasing v0.1, we’ve spent the past few weeks further polishing it up in tandem with this blog post, and just released a follow-up v0.2. The Abscissa source repository is hosted on GitHub.

Here at iqlusion we have developed a number of Rust applications, ranging from CLI applications (e.g. devops tools) to network services including the Tendermint Key Management System. Many of these applications used a common set of crate dependencies and copy/paste boilerplate. After a lot of work, we have moved (or are in the process of moving) all of these tools to Abscissa.

Don’t let the “version 0.2” fool you: the codebase that would become Abscissa is now a little over a year old. We have deliberately left the version number in the “0.0.x” range because much of it started as ugly...

It began with a series of PagerDuty alerts on our phones. We occasionally have false positives, but this was different: several alarms in a row. We looked up at the display in our NOC (above photo, although from a different day) to see that this was not a false alarm: Hubble was also showing we were down. At least our alerts were working!

Though we had been making several changes throughout the day to move onto a new internal sentry architecture, almost all of these changes we additive, so it was surprising to see we had an outage. After hopping onto our redundant validator hosts and checking on the process status, we saw something rather scary: both the active validator and backup validator processes had crashed!

A quick examination of the stack trace found the cause: DNS. We had just deleted a DNS record we thought was no-longer in use, which seemed like a likely culprit. So first...

By Tony Arcieri and Shella Stephens

Here at iqlusion, we have taken our past professional experiences from infrastructure and security teams at notable Silicon Valley companies and applied them in building what we believe is one of the most sophisticated proof-of-stake (PoS) validators in existence today.

In this post, we’d like to provide a deep dive into how we built our Cosmos validator, the experiences that shaped our decisions, and hopefully offer some general insights into how to build a high availability datacenter networks and hybrid clouds in general.

Rather than keep this information a trade secret, we prefer to share it to help promote the security of the overall proof-of-stake validator ecosystem, and though sharing it may slightly reduce our defense-in-depth, we are firm believers in Kerckhoffs’ principle - that our infrastructure is designed in such a way that its...

By Zaki Manian and Kristi Põldsam

The first Cosmos governance proposal is live. It is an interesting proposition to adjust a parameter in the Cosmos Hub Network related to inflation and rewards rate. If passed, it would largely impact the rate at which new Atoms are being created and distributed to stakers and validators.

Below, we are going to analyze several parts of the voting proposal.

Iqlusion will back proposals that increase the rate of Atom issuance in the short term. However, we expect to switch our preferences after IBC and other features are in place.

1. Why do we use blocks per year to compute block rewards vs using BFT time?

The rate at which blocks are being produced in the Cosmos Hub is dependent on the distribution of voting power, timeout settings in validator configuration and network topology. As a result, it is difficult to accurately predict the future rate...

Iqlusion validator is excited to participate in the launch of the Cosmos Hub Mainnet!

We are accepting new delegations on the Cosmos Hub. After the launch of the mainnet, individuals and funds are able to stake Atoms to earn transaction fees and block rewards. Staking will be conducted directly via the Cosmos command line. Delegators to Iqlusion will get our support in managing their delegation and making governance choices as the Cosmos Hub upgrades itself.

Delegation Instructions

• Atoms at stake: 1,317,502
• Commission Rate: 10%
• Iqlusion Address: cosmosvaloper1grgelyng2v6v3t8z87wu3sxgt9m5s03xfytvz7

Bond a certain amount of Atoms to given validator:

gaiacli tx staking delegate 
cosmosvaloper1grgelyng2v6v3t8z87wu3sxgt9m5s03xfytvz7 10000uatom --from delegatorKeyName --gas auto --gas-prices 0.001uatom

Iqlusion offers zero fee delegation via an out of band submission process. If...