the iqlusion blog

Blog for iqlusion, a Silicon Valley-based proof-of-stake validator hosting company deploying the Cosmos Network.

Read this first

Postmortem: 2019-08-07 Tendermint KMS-related Cosmos Hub Validator Incident

Last night and early this morning we encountered a series of small outages in iqlusion’s Cosmos Hub validator related to a recently released version of Tendermint KMS: v0.6.1. We are the primary contributors to Tendermint KMS and generally try to run the latest version of the code at all times, in order to smoke test each release prior to a wider announcement. Generally this has been going pretty well and we have not had an outage like this before.

Unfortunately, while the v0.6.1 release appeared to work during the day, to the point we made a release announcement about it on Twitter, as sometimes happens issues didn’t crop up until late at night and early next morning. We missed around 200 blocks when the KMS crashed originally last night, and another 300 when it crashed again this morning. We weren’t alone, there were two issues opened by people who encountered the same problems on...

Continue reading →

Introducing Abscissa: iqlusion’s security-oriented Rust application framework

By Tony Arcieri


Earlier this month we released Abscissa: our security-oriented Rust application framework. After releasing v0.1, we’ve spent the past few weeks further polishing it up in tandem with this blog post, and just released a follow-up v0.2. The Abscissa source repository is hosted on GitHub.

Here at iqlusion we have developed a number of Rust applications, ranging from CLI applications (e.g. devops tools) to network services including the Tendermint Key Management System. Many of these applications used a common set of crate dependencies and copy/paste boilerplate. After a lot of work, we have moved (or are in the process of moving) all of these tools to Abscissa.

Don’t let the “version 0.2” fool you: the codebase that would become Abscissa is now a little over a year old. We have deliberately left the version number in the “0.0.x” range because much of it started as ugly...

Continue reading →

Postmortem: 2019-03-29 DNS-related Cosmos Hub Validator Incident


It began with a series of PagerDuty alerts on our phones. We occasionally have false positives, but this was different: several alarms in a row. We looked up at the display in our NOC (above photo, although from a different day) to see that this was not a false alarm: Hubble was also showing we were down. At least our alerts were working!

Though we had been making several changes throughout the day to move onto a new internal sentry architecture, almost all of these changes we additive, so it was surprising to see we had an outage. After hopping onto our redundant validator hosts and checking on the process status, we saw something rather scary: both the active validator and backup validator processes had crashed!

A quick examination of the stack trace found the cause: DNS. We had just deleted a DNS record we thought was no-longer in use, which seemed like a likely culprit. So first...

Continue reading →

A look inside iqlusion’s Cosmos Hub Validator architecture

By Tony Arcieri and Shella Stephens

Here at iqlusion, we have taken our past professional experiences from infrastructure and security teams at notable Silicon Valley companies and applied them in building what we believe is one of the most sophisticated proof-of-stake (PoS) validators in existence today.

In this post, we’d like to provide a deep dive into how we built our Cosmos validator, the experiences that shaped our decisions, and hopefully offer some general insights into how to build a high availability datacenter networks and hybrid clouds in general.

Rather than keep this information a trade secret, we prefer to share it to help promote the security of the overall proof-of-stake validator ecosystem, and though sharing it may slightly reduce our defense-in-depth, we are firm believers in Kerckhoffs’ principle - that our infrastructure is designed in such a way that its...

Continue reading →

Iqlusion on the Cosmos Hub Governance Proposal 1

By Zaki Manian and Kristi Põldsam

The first Cosmos governance proposal is live. It is an interesting proposition to adjust a parameter in the Cosmos Hub Network related to inflation and rewards rate. If passed, it would largely impact the rate at which new Atoms are being created and distributed to stakers and validators.

Below, we are going to analyze several parts of the voting proposal.

Iqlusion will back proposals that increase the rate of Atom issuance in the short term. However, we expect to switch our preferences after IBC and other features are in place.

1. Why do we use blocks per year to compute block rewards vs using BFT time?

The rate at which blocks are being produced in the Cosmos Hub is dependent on the distribution of voting power, timeout settings in validator configuration and network topology. As a result, it is difficult to accurately predict the future rate...

Continue reading →

Iqlusion Is Open for Delegation

Iqlusion validator is excited to participate in the launch of the Cosmos Hub Mainnet!

We are accepting new delegations on the Cosmos Hub. After the launch of the mainnet, individuals and funds are able to stake Atoms to earn transaction fees and block rewards. Staking will be conducted directly via the Cosmos command line. Delegators to Iqlusion will get our support in managing their delegation and making governance choices as the Cosmos Hub upgrades itself.

Delegation Instructions

  • Atoms at stake: 1,317,502
  • Commission Rate: 10%
  • Iqlusion Address: cosmosvaloper1grgelyng2v6v3t8z87wu3sxgt9m5s03xfytvz7

Bond a certain amount of Atoms to given validator:

gaiacli tx staking delegate 
cosmosvaloper1grgelyng2v6v3t8z87wu3sxgt9m5s03xfytvz7 10000uatom --from delegatorKeyName --gas auto --gas-prices 0.001uatom

Iqlusion offers zero fee delegation via an out of band submission process. If...

Continue reading →

Staking as a Service: Beyond Asset Custodianship

By Zaki Manian and Kristi Põldsam

staking service image.jpg

The emergence of Staking-as-a-Service providers is one of the rising trends in the blockchain industry. This is a trend that has been talked about since 2017 and it is incredibly satisfying to see it beginning to blossom.

There is a strong demand for such service by hedge funds, VC’s and other entities holding large amounts of illiquid tokens. Those tokens can be used to support securing the network and to generate additional returns. However, setting up and managing a secure and reliable validator node is not and should not be in the core competence of money managers. Node mismanagement and faulty behavior, be it intentional or not, can result in slashings by the protocol with penalties far outweighing the transaction fees and block rewards that can be earned. The responsibility for maintaining validator network infrastructures should be left to...

Continue reading →