Seamless Migration from Bitbucket to GitHub: A step-by-step guide and key takeaways – Part 2

In the first part of this blog series, we walked through our decision to migrate from Bitbucket to GitHub, the strategic planning that went into it, and how we laid the groundwork for a successful transition. From aligning stakeholders to building custom automation scripts, every step was designed to support a migration at scale.

In Part 2, we’ll focus on the real-world execution – covering the challenges we faced, the outcomes achieved, and the key learnings and best practices that emerged from moving over 1,600 repositories with minimal disruption.

Challenges during migration

1. Github Org Plugin: Managing Jenkins job creation and organization for multiple repositories across different GitHub organizations during the migration of Jenkins libraries was complex.
   Solution: The GitHub Organization Plugin was used to automate the scanning of GitHub organizations, create Jenkins jobs for each repository, and organize them into folders named after the organizations. All newly migrated Jenkins jobs used configuration directly from their respective GitHub repositories, maintaining consistency and version control within the GitHub organization folder structure.
2. Shared Library: One of the primary challenges was to manage two versions of a shared library. The first one was the existing library which was based on the bitbucket structure.  This library was implicitly loaded as a part of jenkins configuration.The second one contained the modifications that enabled it to run using the repositories on GitHub. Jenkins cannot load two libraries implicitly with both having the same function definition.
   Solution: We moved the First version of Jenkins library configuration to the bitbucket project folder on Jenkins. This allowed us to load it implicity for bitbucket projects only rather than global Jenkins. Similarly, we configured the Second verison of shared library on GitHub-org folder with the implicit load.
3. Closed PR: Closed pull requests could not be migrated as both the source and target branches either don’t exist or are currently not in the state while creating PR.
   Solution: To address this, a separate branch was created with details of historical closed PR with conversation details to store them for future reference.
4. Handling repositories with similar names: While generating the repos.csv file (used as input for migration scripts), it was discovered that Bitbucket allowed repositories with the same slug under different project keys. However, GitHub does not support repositories with duplicate slugs, leading to potential conflicts during migration.
   Solution: To address this issue, a set of utility scripts were created:
   1. precheck_GitHub.sh: Verified the existence of each repository on GitHub listed in repos.csv, preventing duplication errors.
   2. repeated_value.sh: Identified duplicate repository names in repos.csv, helping isolate Bitbucket projects with the same slugs.
   3. current_inventory.sh: Generated a detailed list of the current Bitbucket inventory with filters (e.g., archived true/false) to track the migration status of repositories.
5. Edge cases: During the migration process, some repositories had unique characteristics or language-specific structures (e.g., Go language repositories or Terraform-based projects). These required special handling and could not follow the standard migration flow.
   Solution: Custom scripts were developed to handle these edge cases:
   1. go_exclude.sh: Filtered out Go repositories by excluding those with a go.mod file or any Go-related files, ensuring only non-Go repositories were included in the standard inventory.
   2. project_wise_repository.sh: Generated a list of Bitbucket repositories based on specific naming conventions or project keys, allowing more granular control over inventory segmentation
6. LFS (Large File Storage):At Bitbucket: Many of the repositories from Bitbucket failed to clone from Bitbucket as LFS objects were disabled from the repository due to which the cloning process could not be completed.
   Solution: Enable Allow LFS in GitHub repository settings.
7. Handling different cases of Git LFS, i.e
   1. The difference in size limit of files for Bitbucket(4GB) and Github(100MB).
   2. The referenced file as an LFS object could not be pushed to be Github.
   3. GIT LFS could not track the current working directory.
   4. GIT LFS version mismatch
   
   Solution: Make a bare clone version of the repository and import all the files with git LFS including the extension name of the file.
8. Forked Repositories: A few repositories were forks of other repositories e.g. GOmodules, and tf scripts. This caused the below issues while migrationIssues related to path spec and hidden pull request references
   ! [remote rejected] errors after mirroring a git repository
   Solution : Exclude the refs from the repository while cloning or have a bare clone of the repository this will allow you to push/ mirror without .git configs.
9. .gitignore: Restricted files in .gitignore i.e; .txt .json in some of the repositories and also repo-slugs were also restricted in the .gitignore files which did not allow us to push with the above-mentioned slugs and extensions during the push of information-related to closed pull requests.*The closed pull request data was summarised in a file and pushed in a new branch on GitHub.
   Solution: Change the extension to .pr and remove the repo slug from the closed pr branch.
10. Empty repository: The presence of empty repositories abrupt the flow of script as other processes like pr migration etc. could not be performed on an empty repository.Solution: Created a separate inventory for the empty repository and updated the script to process the same.
11. OS limitation: While using bash we used the sed command for replacements that were being made in the file based on the migration being performed.The local testing was successful but the sed command varies based on the operating system being used.
    Solution: So sed was replaced with awk so that it can be used on any server.
12. Reference URLs: The submodule urls were changed according to the git addresses for the migrated repositories, but during testing, it was found the Jenkinsfile could not find certain dependent files as the git was not in sync with the submodule.
    Solution: Updated the script with the git sync command and tested the builds.
13. Webhooks: During the webhook migration, the Bitbucket repositories had different private / authenticated Webhook URLs which triggered an in-house script that handled Jenkins job execution.
    Solution:   This solution could not be used with GitHub due to which a single org (GitHub plugin) based Webhook URL was created to achieve the requirements.
14. Versioning: Migrated repositories had outdated Jenkins libraries and build-number-based Git tags that were no longer relevant.
    Solution: Groovy scripts in Jenkins libraries were updated, and all existing tags were deleted to reset versioning cleanly.

Migration outcome:

Successful migration of repositories from Bitbucket to GitHub with proper permissions, branch protection rules, submodule updates, Jenkinsfile modifications, webhook configurations, and an archived state for the repositories. A post-migration report verifies the correctness of the setup.

Key learnings

1. Pre-migration planning is essential
   1. Define the migration strategy clearly, indicating repository structures, permissions, and integrations.
   2. Communicate the migration plan to all stakeholders beforehand.
2. Data Verification
   1. Ensure that all repositories, including branches, tags, and commit history, are correctly migrated.
   2. Verify that there is no data loss by doing integrity checks.
   3. Verify the repository structures and permissions in accordance with the original configuration.
   4. Ensure that all the migrated webhooks are correct and in good working condition.
3. Access and Permissions Management
   1. The permission model in GitHub is different from that of Bitbucket; proper adjustment of team access is important.
4. Handling CI/CD Integrations and Managing Webhooks and Third-Party Integrations
   1. Identify differences between Bitbucket → Jenkins and GitHub → Jenkins connection or any CI/CD tools.
   2. Update configurations and secrets to ensure smooth deployment after migration.
   3. Webhooks and integrations (e.g., Jira, Slack, CI-CD tools) may need to be reconfigured post-migration.
   4. Verify API rate limits and authentication mechanisms to avoid service disruptions
5. Resolving Repository-Related Issue
   1. Large repositories may require optimization before migration (e.g., pruning unnecessary history).
   2. Identify and resolve dependency issues related to Git LFS or submodules.
6. Post-migration validation and testing:
   1. Check all branches, tags, and commits are there.
   2. Run test deployments and workflows to validate the setup in GitHub.

Core Migration Time

migration time

Other Key Stats

key stats

Benefits realized:

1. BitBucket Maintenance and upgrades
2. Cost Saving of Bitbucket servers and Licenses
3. Robust Version control, CI/CD, and AI integrations

Final Thoughts

1. Any migration, no matter how large, can become straightforward with the right planning and homework.
2. Most problems seem daunting initially, but they often aren’t as complex as they appear once approached methodically.
3. Proper scripting, proactive identification of issues, and collaborative resolution are critical for seamless execution.

This experience was a rigorous learning exercise, refining our processes and adopting best practices for large-scale repository migrations.