Seamless migration from Bitbucket to GitHub: A step-by-step guide and key takeaways – Part 1

Introduction

In the rapidly changing landscape of software today, companies have to constantly reassess their tools to remain competitive. This article explains the journey of migrating code repositories from Bitbucket to GitHub for an Enterprise customer, a leading university in Australia, as Github has better collaborative features, smoother integrations, and a developer-friendly ecosystem. With 1,600 repositories to be migrated with history intact, branches, tags, and metadata, manual migration was not an option. Therefore, a custom Bash script was designed to fully automate the migration. This blog will examine the difficulties faced, the tactics adopted to surmount them, and how the migration was effectively implemented within the agreed-upon timeline.

Problem statement

The customer had been using Bitbucket as their primary version control platform older than 5 years but decided to transition to GitHub to take advantage of its enhanced collaboration features, tighter integration with CI/CD workflows, and a more developer-friendly ecosystem.

There were 1600 repos in Bitbucket required to be migrated to GitHub.

Known challenges before migration:

The key challenges encountered:

1. Lack of Existing Tools: The search for migration tools that could meet the specifications led to disappointment since there was no direct match. This led to the creation of a custom Bash script for the needs of migration.
2. Repository Structure Differences: In Bitbucket, repositories exist under projects; in GitHub, all repositories need to exist under a single organization directly. Dynamic modification of the API URLs had to be performed during the migration process.
3. Feature and Format Differences: Some of its features e.g. User login exp. Client wanted to have Organization managed users on GitHub (SSO) with an enterprise-managed user base.  On Bitbucket, user authentication was based on traditional SSO. Groups on GitHub were synchronized with the IDP, whereas on Bitbucket the access was managed on the project level which didn’t necessarily require user groups.
4. Metadata Preservation: A critical part of the migration was ensuring that every repository’s commit history, branches, and tags were transferred accurately. Managing this while addressing the platform-specific differences required extra attention.
5. Change in remote URLs: The repository URL has changed due to the migration, particularly in shared library code. This requires removing project-level dependencies in the Jenkinsfile and shared libraries.

Preparation work

The feasibility of the migration has been assessed to determine which features need to be migrated. The checklist is provided below.

checklist

To ensure a smooth and successful migration of repositories from Bitbucket to GitHub, the following key steps were followed:

1. Creation of repos.csv:  A repos.csv file was prepared as input for the migration script, listing project keys and repository names in the following format: PROJECT_KEY, REPO_NAME
2. Configuration of variable.var File which is responsible for carrying certain variables that are needed during migration.

Migration planning

Migration Planning, Script Development, and Execution – TTN Team

The TTN team is responsible for designing the migration strategy, developing the required automation scripts, and executing the migration process. This includes:

1. Identifying repositories and defining the migration approach
2. Creating and testing migration scripts
3. Executing the migration in a structured and phased manner

Review and Validation – Customer Team

The Customer team checks all the repositories migrated and verifies that they are correct and unchanged. Major areas of work are:

1. Verification of repository structure, commit history, metadatas
2. Branch protection and correct permissions
3. Find mismatches and resolve them after migration.

Effective migration requires careful planning to ensure a smooth and successful transition. Below are the key steps involved in the planning phase of the migration:

1. Prepare Input File: Created repos.list containing project keys and repository names.
2. Pre-Checks: preChecks.script Scripts will check the presence of repositories on both Bitbucket and GitHub with their current state and also will reduce the redundant values from the input file created.
3. Migration Execution:  getInventory.script to transfers repositories with full history.
4. Group Permissions: getBitbucketPerm.script & mapPermGithub.scriptUsed scripts to retrieve and map Bitbucket group permissions to GitHub.
5. Default Branch: Retrieved and set the default branch in GitHub using defaultBranch.script.
6. Branch Protection Rules: brProtectRule.script applies branch protection rules after setting the default branch.
7. Submodule URL Update:  updateSubmodUrl.script  applies submodule links to reference GitHub repositories.
8. Jenkins URL Update: applied Jenkins changes and repository URL changes across repo updates using updateJfRepoURL.script
9. Post-Migration Report: Generated a report to identify inconsistencies and verify migration success using migrationReport.script
10. Check Discrepancies/Fix Issues: Reviewed and resolved any issues from the migration report. validateMigration.script
11. Archive GitHub Repositories: Archived successfully migrated repositories after validation archiveGithub.script

Execution flow diagram

execution flow diagrams

Migration Workflow

The repository migration process was carefully structured to ensure consistency, accuracy, and minimal disruption. Here’s a high-level overview of the steps involved, which can be visualized in the accompanying flow diagram:

1. Repository Selection
   We began by finalizing the list of repositories that needed to be migrated. This included identifying repositories across projects, both active and legacy.
2. Environment Variable Update
   Environment variables were updated to reflect the necessary credentials and configurations required for seamless migration.
3. Migration Script Execution
   A custom script (migraterepo) was executed to handle the actual migration. The script preserved the existing directory structure and metadata of each repository.
4. Legacy Project Handling
   If a repository belonged to a legacy project, it was migrated to GitHub.Archived post-migration in GitHub.The original Bitbucket repository was also archived.If the repository was not part of a legacy project, we proceeded with further updates
5. Post-Migration Updates for Active RepositoriesModule Check: We checked if the repository contained any submodules.
   If submodules were found, their URLs were updated accordingly.Go Projects: For Go-based repositories, we made updates to go.mod and other relevant files to reflect new library paths or dependencies post-migration.
6. Inventory Comparison
   After the migration and updates, we compared the GitHub repository inventory with the original Bitbucket inventory to identify any discrepancies and ensure completeness.
7. Batch Processing
   If more repositories were pending, the above process was repeated for the next batch. This cycle continued until all repositories were successfully migrated and verified.

Architecture Diagram

Architecture diagram

Seamless execution:

With around 1,600 repositories to migrate, the process was divided into three manageable batches.

1. Batch 1: The first batch included less complex repositories for a smoother initial migration.
2. Batch 2: Dealing with more complex repositories that involve extra considerations.
3. Batch 3: Resolved the remaining repositories according to the insights from the previous batches.

Note: A topic in GitHub was added, tracking the migrated repository count. The workflow does not incorporate reusing Git LFS, references, or additional difficulties handling.

In the end, the migration was a success, with all repositories successfully moved to GitHub.

In the next part of this blog series, we’ll dive deeper into the challenges we faced, the impact this migration we had, and the lessons we learned along the way – along with the best practices that helped us keep things smooth and scalable.