BNB Beacon Chain: Not A Blockchain?
Besides being the world’s largest cryptocurrency exchange by market volume, Binance also operates several “blockchains” (the need for parenthesis will become apparent shortly) which on the surface appear to operate as normal.
The BNB Beacon Chain provides governance for the ecosystem described here. It is a Cosmos-derived project with heavy modifications and tightly integrated with BNB Smart Chain, formerly Binance Smart Chain, which is ethereum-derived.
It is no longer clear BNB Beacon Chain should properly be called a blockchain as there are issues with the process where “blocks are chained” such that “the work to change the block would include redoing all the blocks after it.” The docs are here. Those quotes come from here.
Some initial evidence of odd behaviours is provided in this Twitter thread:
Continuing that work has explained why people see magical balances appearing and disapeparing within the Binance chains. The BNB Chain Chief Scientist has seen fit to acknowledge there are issues here which is an excellent first step. However things remain confusing to say the least:
v_bnbchain there is, or at least claims to be, the Chief Scientist. If you read those threads it is immediately apparent we need to figure out what not-in-the-explorer transactions there are and generate some kind of extract-transform-load process to, um, explore them. Well let’s just say that did not go well.
We have more to say on confusing balances, transfers and so on as well. That is for later. The evidence presented here raises deep questions about what the BNB Beacon Chain is doing. As the BNB Beacon Chain provides governance for BNB Smart Chain, and BNB is that chain’s native token, this raises questions about BNB and everything these chains touch.
BNB Beacon Chain Errors
While trying to sync the BNB Beacon Chain from genesis we got the following error:
panic: Failed to process committed block (285075852:2BDC391C402FF452B83AD484D5C40DA615133C25E60C07352CBC6E45435EA873): Wrong Block.Header.AppHash. Expected 3E60F1573122DC7FAD2C5E4779A21BFEEB578422C915A16DEA70A1A617314720, got 1EDDADB1DC0B8E67A3F10FCE05201A4A59A7C380EC54F3CA83D400317CC49685
Let’s translate that error into English: while trying to validate the 285,075,852th block the hashes do not match. It’s not a valid blockchain. Sorry the node gives up.
If you look this block up on the relevant explorer at https://explorer.bnbchain.org/block/285075852 the hash matches the public chain. This is the mainnet data. We are not syncing some incorrect data or the wrong chain or anything like that.
The exact check which fails is in this file. This file, this check, does not exist in Cosmos. In fact the entire local region of this code does not exist in Cosmos. These are all custom modifications for these chains. The BNB Beacon Chain is Cosmos-derived but broke off several years ago and is extensively modified. It is certainly not documented as well as Cosmos. And, as we will see below, while there are users among the general public they struggle mightily.
Back to our node. It is now a bit stuck. But let’s say we wait a day and try again, also from genesis. Well now the error is:
panic: Failed to process committed block (285268100:BE648E40618C858CD5FEB6D83517121BAF0F9624DE7A2A6666BD5E12F8C9BB34): Wrong Block.Header.AppHash. Expected 44D7905CE27CD8035800CB05689C59FF2577F6B1A2545DDFD74B5FDEA3F8F584, got 872C99D77FB32857030F253947EC17F29A12536056F93B64F299D2E7FB24DE46
Just a later block. This continues day after day.
So now we run modified code that explores the data structures and generates a lot more useful logging information. And what do we find? We find the chain breaking at these timestamps
The chain breaks every 24 hours. And yet the chain is still running. Validation pushes past previous breaks, somehow, and blocks march on.
BTW this also suggests this chain has a blocktime of about 0.45 seconds as of now. That’s not documented anywhere so just putting it out there.
This Is Not New
Having now read at least some of the code carefully and worked a bit with the project we know which flags and functions etc are involved. This makes Googling a lot easier. And what do we find? Relevant bug reports galore:
Some of those are over two years old. Let’s quote a few of the reports:
- Why does binance actually make the snapshots if they are useless as far as node can sync from them?
- The problem is with wrong block hash as you can see from the logs.
- it seems to work fine for a while, when it fails I cannot re-join the netowork.
- yes, I stoped and sync again, and it works. interesting….from node-binary.
- I’m afraid it’s not possible. you can only state-sync from heights with snapshots.
- I also tried recover tool, it worked fine in this situation
Oh there is a “recover tool” is there? Any technical folks reading this should check out this file. These lines will surely “repair” a broken chain of blocks:
// Purpose:
// Reset node to a specific height and continue block from this height
//
// Usage:
// 1. go build state_recover.go
// 2. ./state_recover height_to_reset home_path1 home_path2 ...
blockState.LastResultsHash = nextBlock.LastResultsHash
blockState.AppHash = nextBlock.AppHashAlso note quite a few of those issues reference “node-binary.” This is a github repo that contains pre-compiled binaries for running the governance chain. There is a heavier reliance on pre-built code and pre-existing snapshots here than in other parts of the blockchain space.
WTF?
If your blockchain has a “state recover tool” to rearrange hashes it may not be a blockchain. Similarly folks have been reporting problems with AppHash for years. Were we not already successfully running nodes on TRON, Huobi Eco Chain and other, um, projects with limited documentation we probably would have given up. Alas here we are.
BNB Beacon Chain uses Tendermint for consensus per the docs. And Tendermint is a well-documented consensus mechanism. You can read a semi-technical treatment here. Let’ quote just a bit:
Validation
Here we describe the validation rules for every element in a block. Blocks which do not satisfy these rules are considered invalid…
AppHash
block.AppHash == state.AppHash
So this AppHash thing is kind of a big deal.
Overwriting this data is concerning. That the SOP here is to edit hash histories is terrifying. That’s not a blockchain at all.
Now, for completeness, the BNB code is heavily modified from Cosmos and we have clearly not read every single line carefully. Further, the way the code was “forked” is odd. The projects are not actually forks on github — which means there is no simple way to identify differences and merge code. In fact almost all the Cosmos and Tendermint repos are copied and renamed from things like “cosmos-sdk” and “tendermint” to “bnc-cosmos-sdk” and “bnc-tendermint.” And then, for anyone familiar with the go programming language’s modules feature the go.mod files contain passages like:
replace (
github.com/cosmos/cosmos-sdk => github.com/bnb-chain/bnc-cosmos-sdk v0.25.5
github.com/tendermint/go-amino => github.com/bnb-chain/bnc-go-amino v0.14.1-binance.2
github.com/tendermint/iavl => github.com/bnb-chain/bnc-tendermint-iavl v0.12.0-binance.4
github.com/tendermint/tendermint => github.com/bnb-chain/bnc-tendermint v0.32.3-binance.7
github.com/zondax/ledger-cosmos-go => github.com/bnb-chain/ledger-cosmos-go v0.9.9-binance.3
golang.org/x/crypto => github.com/tendermint/crypto v0.0.0-20190823183015-45b1026d81ae
)Not the most transparent way of forking your dependencies. They’ve copied and renamed quite a few projects and then relinked them in a way you cannot spot simply reading the code or running a normal build.
But the problematic hashes above match the public explorer as we have shown so there are issues with these “forks.” Now we need more information to figure this thing out. And, obviously, this post does not contain everything we already know.
If this is all just a documentation problem we will acknowledge that. But someone needs to explain these error messages, patterns and suspicious code snippets in a manner that passes a careful inspection.
Implications
If this thing is not really a blockchain it means whoever is repairing these hashes is operating the system. We can debate how and to what degree Bitcoin, Ethereum and other blockchains are decentralized. But nobody there is repairing hashes looking backwards whereas here things look decidedly less like Satoshi’s vision.
Among other things whoever is operating this is then arguably the operator of the Tornado Cash contracts on BNB Smart Chain fka Binance Smart Chain. That’s before we start discussing odd flows of BNB around the ecosystem.
