Github baru saja berbagi tentang bagaimana strategi mereka mengupgrade lebih dari 1200 server mysql meerka ke versi MySQL 8.0. Laman repositori software terbesar didunia ini, sedari awal dikembangkan menggunakan Ruby on Rails memang menggunakan MySQL sebagai database mereka.
Setelah bertahun-tahun menggunakan versi 5.x, proses Upgrade ke MySQL 8 sukses mereka lalui dan mereka ceritakan dalam blog mereka. Sebagai gambaran, MySQL milik Github yang berjumlah 1200 lebih itu tersebar di datacenter Azure dan baremetal di datacenter mereka sendiri. Total mereka melayani 300 TB data dan melayani 5.5 juta query per detik dalam 50 lebih cluster database.
Bagi yang ingin membaca lebih lanjut tentang proses mereka melakukan upgrade, silakan baca blog mereka disini.
GitHub undertook a meticulous process to upgrade over 1200 MySQL servers to version 8.0, with a focus on maintaining high availability and adhering to Service Level Objectives (SLOs) and Service Level Agreements (SLAs). Here is a summary of their key points:
- Preparation for the Upgrade:
- Started in July 2022 with various milestones before upgrading production databases.
- Prepared infrastructure for the upgrade, determining default values for MySQL 8.0 and benchmarking performance.
- Ensured application compatibility by adding MySQL 8.0 to Continuous Integration (CI) and enabling pre-built containers for debugging.
- Communication and Transparency:
- Utilized GitHub Projects for a rolling calendar to communicate and track the upgrade schedule.
- Created issue templates to coordinate the upgrade between application and database teams.
- Upgrade Plan:
- Implemented a gradual upgrade strategy with checkpoints and rollbacks.
- Step 1 involved rolling replica upgrades with careful monitoring.
- Step 2 updated replication topology with both 5.7 and 8.0 replicas.
- Step 3 promoted MySQL 8.0 replica to primary through a graceful failover.
- Step 4 upgraded internal-facing instance types.
- Step 5 involved cleanup, removing 5.7 servers after successful validation.
- Ability to Rollback:
- Maintained the ability to rollback to MySQL 5.7 for read-replicas by keeping enough online.
- Ensured backward data replication between 8.0 and 5.7 for the primary database.
- Overcame challenges related to character collation and role management during rollback.
- Final Steps:
- Upgraded ancillary servers for backups or non-production workloads.
- Conducted thorough validation, including a complete 24-hour traffic cycle, before removing 5.7 servers.
- Challenges Overcome:
- Addressed character collation incompatibility by setting default encoding and collation.
- Managed role management issues during the upgrade window.
- Maintained high confidence in backward replication for critical applications like GitHub.com monolith with consistent Rails configuration.
GitHub’s detailed approach ensured a successful and controlled transition to MySQL 8.0, reflecting their commitment to high standards of availability and performance.