Pigeonfall Vulnerability Report

Background

On Wednesday October 11th, 2023, a Strangelove Engineer proactively found a high-severity vulnerability affecting our packet-forward-middleware (PFM) v7.0.0 release. The issue allows a malicious actor to halt the chain upon select forwarded IBC packets. The issue could also be triggered accidentally. This issue, dubbed Pigeonfall (SL-001PF), was discovered while improving the end-to-end testing of PFM to catch adverse events such as this.

Technical Issue

The transfer module in ibc-go v7 (ADR-011) introduces additional escrow balance accounting. It further adds an invariant check of the expected balance within the transfer keeper key/value store compared with the actual balance of the escrow account. This escrow balance accounting was not taken into account in the v7.0.0 release of PFM.

As a result, forwarded packets that fail due to either a receive error on the destination or a packet timeout will halt the chain if the invariant is registered. This is the default behavior for all chains affected on SDK v47:

• Terra-Money

• Juno

• PersistenceOne

• Terp Network

• Xion

• Carbon

While the discovery of a security vulnerability is not a security incident, the commonality of IBC packet timeouts and failed acknowledgements pointed to a higher likelihood of an accidental halt. Due to this risk, the patching process qualified as emergency coordination and was handled accordingly.

Issue Timeline

Discovery

On Wednesday, October 11th around 2pm CDT, Reece Williams discovered panic and halting behavior when writing further end-to-end testing for PFM IBC packet timeouts and failed acknowledgements. Previously testing had been written for older versions of PFM, but it had not been ported to the latest ibc-go v7 and SDK v47 release.

Within 30 minutes the issue was identified and the v7 patch was written and tested locally. Reece disclosed the findings to Strangelove’s CTO, Andrew Gouin. They discussed the problem, scope, and mitigation strategies that should take place. Andrew brought in Jack Zampolin, Sean Beckett, and Jessy from the Amulet security team for guidance and review.

By 3:30pm a list of all chains had been compiled that use any version of Packet-Forward-Middleware. Initially we believed this affected every version of PFM (v3 to v7). After further investigation, we found the escrow accounting behavior was a new addition in ibc-go v7. The reason for this is that previous versions of the SDK did not register the crisis module properly. The IBC release documents made no mention of this in v7, so we assumed the behavior had existed for some time. Discovering the behavior was limited to v7 narrowed the list from 26 chains to 6.

Migration Patch

Overnight, we wrote a proactive mitigation in the event a chain was halted before the patch was applied. This was tested and ready to go on Thursday, October 12th at 11am CDT, 21 hours after initial discovery.

For the patch, we wanted to keep the affected code private to limit risk. To do this, we pushed the patched commit to the cosmos/ibc-apps repo, then deleted the branch. With this approach, the code is not publicly available on GitHub but the commit remains cached for 90 days. Once all chains were patched, we re-pushed the code and it was compiled into the latest release. Only those aware of the commit could see the patch code locally.

On October 12th at 1:20pm CDT (23 hours since discovery), we distributed a single line patch command to security contacts along  with the information about the vulnerability. Further connections were made via Discord, Telegram, and Slack channels to confirm that all affected parties had acknowledged the issue.

Teams began rolling out release tags for their validator sets over the coming hours. Strangelove was able to advise and answer questions teams had about the vulnerability, stepping in to assist as needed.

As of October 19th, all affected chains have successfully migrated and are no longer vulnerable to halting. We applaud the teams and validator sets for their quick reaction time and tight collaboration to keep IBC networks secure and live.

Next Steps

For chain developers, if you are upgrading from an older IBC version or launching your network soon, make sure you apply the latest v7 PFM patch before your update.

For Module developers, always use an end-to-end testing framework like interchaintest. If your module introduces invariance checks, make sure to add this into the upgrade release notes. Developers need to be aware of such large changes, especially for Middleware applications where funds are moved.

For Cosmos networks, please make sure to have a security@ email or social media contact to make the process of notifying easier for future events. A great spot to put critical information is in the public github README or SECURITY file.

The Strangelove team will be writing conformance tests for patch-based upgrades. These will run multiple networks on both the old and new version, and execute a simulation of every transaction against both.