Node Sync

Disclaimer: This is a summary of patterns we have observed during our research and should not be considered any form of technical or investment advice. Also, the given “known examples” do not imply they are the best implementations of the said pattern or any superior to any other implementation of the pattern not listed.

Summary

Create a clone of a blockchain node by synchronizing blocks and history.

Context

The user wants to add new nodes or swap nodes to change the physical location/node, improve performance, or create multiple instances of the source blockchain. The consensus rules of the target blockchain platform are compatible with the source blockchain. A snapshot of the source blockchain is available.

Problem

How to load states and smart contracts to the target blockchain?

Forces

• Consistency – States such as blockchain native assets cannot be arbitrarily created because migration must preserve system invariants. It is difficult to create a complete copy of the ledger and block history, as the system is distributed, new blocks are built continuously, and finality is not immediate.
• Accountability – Initiation of new accounts and their states on the target blockchain must be recorded with proof.
• Size – A blockchain may contain a large number of accounts, states, smart contracts, and transactions. Moreover, history may consist of a large number of blocks. A large subset of these may need to be initiated on the target blockchain.
• Latency – Time to create accounts and initialize their states is proportional to the number of accounts and their states. Given the large size of the global state and history, time to make a copy could be very long. It takes even more time to recreate and validate the global state by applying all transactions and running smart contracts.
• Cost – Each account creation and state assignment may needs to pay a transaction fee. The transaction fee could also be proportional to the size of the state.
• Consensus – To accept a state, smart contract, transaction, or block as valid, both the sender and receiver nodes must follow the same consensus rules.
• Governance – The consensus of the blockchain’s governance body is required to update any state or blockchain platform software.

Solution

Steps shown in the following figure can be used to synchronize the new node using the blockchain platform-specific sync tool. First, install the source blockchain platform’s client software (or an updated version that is backward compatible) on the new node. Also, configure the new node to connect to other members of the source blockchain. Second, enable the sync tool on the node to copy various data structures representing the global state, smart contracts, transactions, and blocks from other blockchain nodes. Next, the node should rebuild and validate all the transactions from the genesis to verify the global state. Any errors, such as failed data transfer, need to be resolved by requesting further data. Finally, reconfigure the node to accept new transactions.

The volume of the global state, smart contract, transaction, and block data of a highly-active blockchain could be in excess of hundreds of Gigabytes. Hence, many days to weeks could be required to download and validate them depending on bandwidth, disk IO, and CPU limitations, as well as ongoing transactions.

Several optimizations could be applied to reduce the sync time. For example, the new node can be bootstrapped using a snapshot of the source blockchain. Then only the data from new blocks created after the snapshot need to be synchronized. Moreover, validation could be performed at different levels, such as validation of the last n blocks, random blocks, entire block history, and entire block history and transactions.

If the objective is to swap blockchain nodes, decommission the old nodes. Whereas if the objective is to create multiple blockchain instances, reconfigure the new node(s) to behave as an independent blockchain after a set block number. This is usually achieved by setting a new blockchain identifier and disconnecting from the parent blockchain’s nodes.

Benefits

• Consistency, Accountability, Integrity, and Immutability – Blockchain sync tools rely on asynchronous messaging, immutability (to identify states updated after a given block number), consensus (to provide consistency), and validation. Hence, no state, history, and IDs are lost or changed in the process, ensuring consistency, accountability, integrity, and immutability.
• Governance – No explicit approval of the blockchain governance body is needed to add a new node.
• Cost – No transaction fees as data synchronization is at the blockchain data structure level.
• Compatibility – This pattern works on any blockchain where source and target blockchain platforms are compatible (i.e., follow same consensus rules), and the target node(s) is new.

Drawbacks

• Latency – Time to synchronize a large volume of states and history could vary from hours to days. Time can be substantially reduced by bootstrapping a node using a snapshot and synchronizing only the ledger state.

Related patterns

• Snapshotting pattern can be used to speed up the bootstrap process.

Known uses

• This pattern is frequently used to connect new nodes to an existing blockchain. Bitcoin, Ethereum, and Bithereum allow new nodes to download a snapshot file that reflects the global state and blocks at a set block number, and then sync with other nodes to speed up the process.
• Ethereum supports syncing data at different granularity levels, such as full, fast, and light. Similarly, Bitcoin supports full node and lightweight node syncing.