Mandrake (PFM) Vulnerability

Mandrake Vulnerability Report

Background

On Thursday, February 15th, 2024, Composable Finance submitted a bug report through the HackerOne bug bounty program with regards to packet-forward-middleware. The bug was determined to affect every version of the middleware. This bug could result in an escrow account, associated with an IBC port and channel, having a balance that is less than the total supply of an IBC denom token on a counterparty chain. This could result in users being unable to unwind their tokens back through the traversed path they had taken to reach the chain where their assets are currently located. The Mandrake vulnerability was discovered via manual debugging of a problematic IBC client on a production network.

Technical Issue

In packet-forward-middleware, if a token was transferred from a source chain, forwarded through an intermediary chain(s) using packet-forward-middleware, received on a destination chain, and then the user attempted to do another multi-hop tx to send those same tokens back through the intermediary chain(s) to the original source chain, the internal bookkeeping in the escrow accounts was not properly handled. Essentially, when the multi-wrapped IBC denom token was moved back through the intermediary chain, packet-forward-middleware was failing to properly handle timeouts and acknowledgement errors,  such that these failures, in this particular scenario, resulted in the escrow account on the intermediary chain not being properly updated to reflect the total supply of the asset on the counterparty chain.

This issue was present in all supported versions of packet-forward-middleware. While the bug could result in users temporarily being unable to unwind their assets back to the original source chain, it was determined that this bug could not result in users losing funds or in a chain halt and so the vulnerability was determined to be medium severity. We chose to move forward with a private disclosure process to allow teams a period of time to patch so that the discrepancies between any problematic escrow accounts and the counterparty total supply would not be made worse by a malicious actor.

Issue Timeline

Discovery

On Thursday, February 15th, 2024, the Amulet team reached out to the Strangelove team with the details from the bug report that was submitted via the HackerOne bug bounty program. At the time of the initial bug report some of the relevant members of the Strangelove team were in Spain at the ICF offsite and others were traveling for ETHDenver. This led to a slightly slower than normal response time. Over the course of the next few days, we performed extensive testing to determine the severity and impact of the bug and authored a patch to rectify the issue. As of Tuesday, February 20th, the patch was tested and finalized along with the initial security advisory report.

Migration Path

Strangelove determined that in addition to providing teams with a patch we would also provide tooling for teams to programmatically check the balances of every escrow account against the total supply of each asset on the counterparty chains. On Monday, February 26th, the Strangelove team finished the escrow account checker tool and applied it to several mainnet networks. We determined the bug had actually been invoked in multiple production environments throughout the IBC ecosystem. The Strangelove team compiled an impacted parties spreadsheet for the private disclosure notification process and on Friday, March 1st, we sent the private disclosure notice to the impacted parties. By Tuesday, March 5th, all impacted parties identified via the GitHub dependency graph had been contacted and received the private disclosure notification.

Next Steps

Chain developers who have not yet upgraded to the patched version of packet-forward-middleware need to upgrade as soon as reasonable. Please refer to the packet-forward-middleware releases in the ibc-apps repository to find the appropriate patch release for the version of packet-forward-middleware your chain is currently using. In addition to patching packet-forward-middleware, it is imperative that you check the balances of each escrow account against the total supply of each asset on the associated counterparty chain. Failure to verify parity between the escrow accounts and the counterparty total supply can result in a type of Denial-of-Service where users may not be able to unwind their assets through your chain. You can use our escrow checker tool or whatever means you deem fit to validate these balances against the counterparty total supply. If a discrepancy is found, you will need to build an upgrade handler that mints and transfers assets to the escrow account(s) with the discrepancy. Strangelove has provided an example upgrade handler.

This bug was ultimately the result of our e2e test coverage not accounting for particular edge cases. It makes a strong case for why it is important to ensure your application has full unit, integration, and e2e test coverage, and that the tests should be reviewed by as many developers as feasible. Thinking through every edge case is not always straightforward, but by increasing visibility and ensuring quality assurance and testing is performed by many individuals you can develop much more robust testing scenarios. “With enough eyes, all bugs are shallow”.

P.S. One of the most challenging aspects of the private disclosure process was identifying valid points of contact for each of the impacted parties. It is important to have an actively monitored `security@your.domain` email setup, along with a security.md file in your repo(s) and/or a bug submission form on your website. Now is a great time to check if that’s something you have, and put a story in your backlog if not!