Announcing interchaintest
Aug 08, 2023
Justin Tieri
Interchaintest is a framework for testing blockchain functionality and interoperability between chains, primarily with the Inter-Blockchain Communication (IBC) protocol. The framework utilizes Golang tests that orchestrate Docker containers for blockchain validators and fullnodes, IBC relayers, and other processes that may be needed for a given test case. It allows users to quickly spin up local testnets, providing a dev environment to test IBC, chain infrastructures, smart contracts, and more.
There are other testing frameworks for IBC interactions, but they tend to be custom built for the needs of a specific open source application or repository. Interchaintest is designed to be adaptable to most IBC contexts. We welcome issues or pull requests from the community if you see areas to expand the scope and utility of interchaintest.
Background
Historically there has been a lack of test harnesses for Cosmos and the Interchain. Many aspects of web3 make testing more challenging. Applications may need to model or mock out consensus engines to evaluate even the simplest flows. The web of connections between parties is more complex than a simple browser/server interaction. There are faucets and tokens and gas fees needed for every chain being tested, and tests may need to bootstrap a complex state for each to evaluate the function under scrutiny.
These many challenges present a steep learning curve for developers to climb if they are new to IBC or to web3 in general. It is difficult to build confidence with small tests and then combine those into larger integration tests when each layer in the stack is using a different, repo-specific testing method. This leads to test failures that aren’t diagnostic, but merely reflective of the complicated and brittle architectural relationships between components. Many teams in Cosmos and IBC end up testing features in production or near-production environments because success with simpler unit-level testing doesn’t build confidence in the behaviors at scale.
In order to test those complex interactions, many teams end up with Dockerfiles and/or shell scripts that replicate the complex network and chain environments. These solutions are often highly localized to particular CPUs or OSes, making them less portable than desired. The hand-built nature of the environments is less likely to generate reproducible results (“It works on my machine”) and this causes difficulty evaluating and addressing issues reported by the community.
Motivation
Strangelove developers have built a number of repo-specific, Docker- and shell-based solutions to meet our immediate testing needs, particularly as we were developing and pushing the initial versions of Go Relayer and Horcrux. We noticed a lot of duplication of effort and also the difficulty in repurposing the existing testing harnesses for new problem domains.
As experienced OSS developers, this pain highlighted the need for a generic test harness for IBC interactions. In order to ship production-grade software for the Interchain, we needed sophisticated developer tooling. We took the time to extract the patterns and create a use-case-agnostic framework for generating repeatable, diagnostic tests for every aspect of IBC.
Use cases
IBC Fundamentals
Strangelove developers have built interchaintest with some primary use cases in mind. Chief among them is validating the conformance of your application to the Inter-Blockchain (IBC) protocol and the various Interchain Standards (ICS). By incorporating interchaintest into your chain during the earliest phases of development you and your team can ensure that your chain is ready to inter-operate from day one.
The modular structure of the tests means you can focus on verifying a particular area of compatibility or interoperability while the other aspects are still in design or development. This empowers an incremental development approach where you validate each successive layer of functionality while creating a test suite that protects against regressions and unintended consequences of later development work. Test that client creation functions, then work on creating connections, then move to initializing channels, and finally into ICS-20 token transfers. You don’t need to build the full steel thread through your chain to know that each previous chunk of work is functioning as needed to support the future features.
Interchaintest tests have already caught bugs during development in Interchain Accounts (ICA), Interchain Queries (ICQ), Packet Forward Middleware (PFM), Swap and Forward, Localhost IBC, and more (see footnote “interchaintest in action”). Strangelove focuses strongly on shipping production IBC-related code and interchaintest is at the backbone of that process for us and many other teams.
Interchaintest works with any IBC stack, not just Cosmos ecosystem applications. Strangelove leverages interchaintest in our work with Composable to bring Polkadot IBC to fruition. We use it with CIDT to bring Avalanche IBC to market following the same play book, and we are working with other light client developers to bring their IBC implementation to the Cosmos. Interchaintest allows Strangelove and our partners to find and address IBC integration concerns at the method and object level, long before testnets and chain code is live.
Chain Upgrades and Migrations
Interchaintest is not only for end-to-end testing of IBC compatibility and functions. It also allows for end-to-end testing and simulation of live chain behaviors scoped purely within your chain. By running chain behavior validation tests powered by interchaintest, you can be even more confident in the production success of your planned chain upgrades, relayer changes, or sidecar process rollouts. You can fine-tune the parameters for epoch-based execution models and simulate events with precision. And your team won’t have to build the environments from scratch. Strangelove builds Heighliner, a tool for building Docker images for your chain, which speeds up the local development cycle. The tagged releases for registered chains are built and published with out of the box support for a majority of the currently live Cosmos chains.
SDK Modules and Smart Contracts
Interchaintest is not just for full chain development, either. It enables robust test cases for comprehensive testing of new Cosmos SDK modules. You can simulate networks, including relayers and smart contracts, to explore the end-to-end behavior and performance of your new module without the hassle of standing up a full testnet infrastructure each time. Interchaintest has solid CosmWasm coverage and has been central to Strangelove’s efforts on the Wasm IBC light client. Use it to rapidly iterate on smart contract development, finding edge cases and validating behaviors while still in a development environment. A good test suite is an excellent tool for preventing unintended behaviors in production.
In the increasingly heterogeneous world of web3 applications, validating inter-chain transaction behavior can be quite challenging. Most teams are expert at their chain and protocol, but not necessarily all the chains and protocols against which they might wish to integrate. With interchaintest, you can programmatically explore and validate multi-chain, multi-module smart contract environments, all utilizing the world-class developer tools available in the Go community.
Future Plans
Non-Cosmos IBC chains
Interchaintest is an active part of our daily development cycle at Strangelove so you can rely on it continuing to expand and mature. We are currently adding support for the Polkadot, Penumbra, and Avalanche chains, and we are committed to fully supporting all valid IBC transactions within the framework.
Richer Smart Contract tools
Our CosmWasm support is present but somewhat basic, and it requires more mocking and scaffolding than we prefer. We envision a robust API where developers can test smart contracts simply, quickly, and with low overhead, including significant Solidity support. We also want to enable long-running environments where infrastructure can exist for significant calendar time, rather than the ephemeral environments currently created. Some issues can hide when everything is re-initialized between tests. There is an in-flight pull request adding this feature. You can review the proposed actions in the API docs.
Testing Against Live Environments
We see significant value in enabling the same integration tests you might run against a local branch to be executed against a testnet following a chain launch, upgrade, or other significant infrastructure change. Many types of performance testing, like load, soak, stress, or throughput tests, are much more valuable and informative when run against a production-like environment.
Extending Beyond IBC
Interchaintest is currently focused on IBC use cases as the primary stack, but as web3 ecosystems become more interconnected, Strangelove intends to make interchaintest a unified testing framework for all relevant web3 activities. Teams should be able to swap out any part of their stack and continue using their deep set of tests, ensuring compatibility and preventing regressions.
Footnote: interchaintest in action
Juno chain halt reproduction and fix
Non-determinism bug in Interchain Accounts demo
https://github.com/cosmos/interchain-accounts-demo/pull/108
Failure to send ICS-20 token transfers on localhost IBC
https://github.com/cosmos/ibc-go/issues/3568
