Post Mortem — Aergo Stall, February 1st 2024: Medium статья от Aergo Official

https://medium.com/aergo/post-mortem-aergo-stall-february-1st-2024-5cfbc56a677e

On February 1st at 13:13 KST, Aergo’s mainnet halted new block generations, which resulted in a temporary outage. The core dev team immediately responded to the issue, successfully restoring the network by applying a hotfix to address the root cause.

To be clear, no funds were at risk as a result of the network halt.

Cause

Block generation was halted due to a bug in the Fee Delegation transaction processing within the Smart Contract engine.

Recovery Timeline

February 1st, 2024:

• 13:13 — Block generation stopped at block number 150677807.
• 15:30 — Blocked External nodes to enable communication exclusively between the Trusted Nodes (whitelisted based on IPs). The initial block regeneration attempt failed.
• 17:00 — Commenced testing the hotfix development version and synchronization test among the BPs and the Trusted Nodes.

Continued on February 1st:

• 19:00 — Snapshot data prepared.
• 21:00 — Completed the hotfix development and the local test was successful.

February 2nd, 2024:

• 01:30 — Block generation resumed.
• 08:00 — Ready to snapshot Block Producers (BPs) and initiated synchronization between BPs.
• 08:50 — Snapshot prepared for the Trusted Nodes, and the hotfix was tested on the Alphanet and the testnet.
• 10:20 — Initiated synchronization test of the hotfix on the mainnet.
• 13:10 — Hotfix development version (2.5.1-dev) deployed on the mainnet with BPs and the Trusted Nodes.
• 14:20 — Initiated synchronization for the mainnet Trusted Node 1 and dApp operation resumed.
• 16:00 — Completed recovery based on Mainnet BP/Trusted Nodes.

We will continue to strive to keep the Aergo blockchain platform stable. We apologize for any inconvenience this may cause and appreciate your understanding.