You may have noticed a stall in the mainnet since UTC 03:00 hrs (15th July). We sincerely apologise for all inconvenience caused due to this. To avoid any confusion, we’ve compiled a summary of the incident and everything we know so far:
What Happened - Due to an unexpected edge case of smart contract transactions flooding the network at the block height 170901, the nodes are currently having different views of the network and cannot agree on the formation of this particular block. Hence, the network is currently going through continuous view change in order to recover itself, but in this scenario the view change corrective mechanism cannot resume block propagation because too many nodes are having different views.
How we are addressing this - Zilliqa is currently in the midst of manually recover the mainnet from the block height of 170901 in order to resume block propagation.
The Planned recovery process - We have informed all the miners and exchanges about this incident. We will be working round-the-clock to get our platform up-and-running.
As part of this, we did a hot fix to the issue by lowering the maximum gas limit (from 500k -> 300k) that can be processed within a micro-block. This will prevent nodes from having different views as lower-performant nodes will not be overwhelmed by the flood of these smart contract transactions coming in. We are currently exploring several long-term solutions and will share them via a detailed technical blogpost.
Measures for Preparedness - We were anticipating a large amount of smart contracts transactions coming from UnstoppableDomains DApp today. However, we were not expecting to hit this edge case in the consensus mechanism.
We will work to include such cases within our internal stress testing for future releases of the protocol to prevent such incidents from happening again.
Instructions Moving Forward - Miners will need to patch their mining nodes when the recovery is completed, we will be issuing a notice to them on the mining TG/Forum. The recovered mainnet will also be upgraded to v4.7.0 in order to fix some persisting issues such as missing state deltas and state root hash check errors.
Once again, we apologise for this interruption and will keep you posted once it is resolved. Thank you for your understanding!