Chang Upgrade #1 lessons learned

The Chang upgrade #1 was a critical step in Cardano's journey toward decentralized governance. It laid the foundation for on-chain governance and launched the technical bootstrapping phase of CIP-1694. Coordinated by the hard fork working group, in collaboration with the community and ecosystem partners like CF and IOI the hard fork was executed smoothly. This lessons learned report reflects on the key successes and challenges during the Chang #1 upgrade and identifies areas for improvement as we look ahead.

The lessons learned were put together by all members of the hard fork working group. Below is a summarized report:

What went well

Smooth hard fork process:

Chang #1 was executed successfully, with a 92% upgrade from stake pool operators (SPOs) and exchanges. The hard fork itself went smoothly, supported by strong coordination and communication across teams.

Incident procedure tested successfully:

The hard fork incident management process was tested during the upgrade, and the rapid deployment of hotfix 9.1.1 ensured minimal disruption. The clear communication across internal teams enabled fast action when needed.

Tooling updates completed without incident:

The readiness tracker and compatible wallets were ready for GovTool, ensuring all stakeholders were well-prepared for the transition. All tooling updates were deployed smoothly.

Strong internal collaboration:

Good coordination between internal teams and external developers ensured effective collaboration. The lack of daily meetings kept the workflow efficient, and the hard fork date of September 1 was set through solid negotiation and alignment between stakeholders.

Effective communication and readiness:

Good engagement with the community ensured that readiness was well-tracked, and even though there was a slight delay in the hard fork, it did not present significant issues.

What didn't go well

Inconsistent communication:

Some teams lacked clarity on what was happening during the process, and a few DApps were unprepared for the hard fork. The absence of universally recognized communication points made it difficult to ensure everyone was on the same page.

Late detection of issues:

A replay from genesis issue was detected late in the process, and the final steps leading up to the hard fork felt rushed. Additionally, the GovTool was not immediately available post-hard fork.

Testnet readiness was uneven:

Some developers were misaligned with testnet management practices, only testing on preprod instead of on preview or sanchonet. Getting people onto preview versions (9.0.0 and 9.1.0) lacked the confidence needed for a smooth transition.

Coordination and partner engagement:

Key partners were left out of important updates, and critical changes to CDDL and the constitution came too late in the process. ICC readiness was also rated as "just okay."

What was learned

Thresholds for readiness were useful:

The predefined thresholds for readiness were highly effective. However, readiness leads should publicly share their outreach plans to improve communication.

Dedicated communication channels are needed:

Clear, dedicated channels for sharing important updates with developers and DApps need to be established to ensure alignment. Some stakeholders were unaware of the hard fork and its significance, revealing gaps in the communication strategy.

Testnet and developer engagement:

Developers need a better understanding of testnet purposes and sequences. Early engagement with governance was effective, but testnets need to be rebooted and made more credible as testing environments.

What could be improved

Increase engagement with DApps:

Stronger engagement with DApps is needed for future upgrades. Clearer communication and collaboration will ensure all developers are aligned with testnet purposes and ready for major upgrades.

Improve readiness tracking and communication:

A single source of truth should be established where all stakeholders can find updates on readiness and important milestones. The readiness page should be updated more frequently to keep everyone informed.

Avoid late-stage changes:

Late changes to critical components, such as CDDL and governance tools, should be avoided to prevent rushed implementations. Future hard forks should also be scheduled on weekdays to avoid challenges associated with weekend deployments.

Better alignment on tools and updates:

Ensuring everyone uses the same versions of tools and updates is crucial. Consistent communication about updates will help developers stay informed.

Actions for moving forward

Coordination and communication

• Create one central hub where all updates and readiness information can be found.

• Establish clear communication channels for developers and DApps, ensuring they are always aligned with updates.

• Share outreach plans from readiness leads to keep everyone in the loop.

• Avoid last-minute changes by setting clear deadlines for updates.

Development and readiness

• Reboot preview testnets to serve as credible testing environments.

• Increase engagement with DApps to ensure they are fully prepared for future upgrades.

• Work with developers to clarify the role and purpose of testnets and governance tools.

• Define the submission process for changes related to Chang #2.

Process improvements

• Schedule hard forks on weekdays to reduce the risk of weekend-related challenges.

• Assign clear responsibilities for each area of governance and developer engagement.

• Communicate timelines and expectations early and provide regular updates to stakeholders.

Chang upgrade #1 was a major success for the Cardano ecosystem, but there are key areas for improvement as we look toward Chang upgrade #2, where practically possible these lessons learned will be implemented. By strengthening communication, improving engagement with developers and DApps, and ensuring readiness across the network, Cardano can continue its journey toward a fully decentralized governance model.