Github Gitlab Sync

Published: Oct 3, 2023

I’ve naturally been drawing lines between “things people should look at” and “things I want in my digital house”. Hence twits vs a real twitter account. I’ve been finding similar dividers wanting to pop up in my software projects, so I setup some of my repos to be synced between Gitlab (where I develop things) and Github (where everyone else develops things). This is a bidirectional sync that allows updates on either Gitlab or Github. You know, just in case any collaborative activity happens on Github. Gotta always be ready to hang with the homies. I haven’t actually had this on a project that has had multiple contributors, so maybe it’ll blow up in that context. In fact, I know it will, but here’s how to set it up:

Initialization

To start we need to have a repository in both Github and Gitlab.

Mirror Gitlab to Github

Generate a token

In Github, under Settings > Developer Settings > Personal Access Tokens > Fine-grained tokens , generate a new token. For “Repository access”, select “Only select repositories” give it access to the repo of interest.

Under “Permissions”, give “Read and Write” access to “Contents” and “Workflows”. The “workflows” permission above allows for committing files under .github/workflows. There might be other permissions missing here for other types of Github specific files, but this can be editted later.

Enable push mirroring

In Gitlab, navigate to your repo and under Settings > Repository > Mirroring repositories, add a new mirrored repo. Supply settings like these, replacing the relevant fields; The password is your Github token you previously created:

Note that here we’ve checked “Keep divergent refs”. This stops the mirror if Github ever has commits that aren’t in the current repo. Which may happen if tokens expire at inconvenient times.

Unfortunately, our later setup for Github to Gitlab sync doesn’t have this safeguard.

Mirror Github to Gitlab

Generate a token

In Gitlab, navigate to your repo and under Settings > Access Tokens add a new token with “api” access. The role should be able to push to the main branch of your repo, by default this is likely only the “Maintainer” role.

Store the token in your Github repo

In Github, navigate to your repo and under Settings > Secretes and variables > Actions add a new repository secret. Give it a name, here we’ll use GIT_SYNC_TOKEN and the value will be the Gitlab access token.

Configure Github to Gitlab mirroring

We’re going to create a Github workflow that pushes to gitlab everytime a modification is made to the repo. Add the following file to your repo, updating where relevant:

.github/workflows/git-repo-sync.yaml
name: build

on:
  - push
  - delete

jobs:
  sync:
    runs-on: ubuntu-latest
    name: Git Repo Sync
    steps:
      - uses: actions/checkout@v2
        with:
          fetch-depth: 0
      - uses: wangchucheng/git-repo-sync@master
        with:
          target-url: 'https://gitlab.com/Meptl/krita-live2d-prep.git'
          target-username: 'Meptl'
          target-token: ${{ secrets.GIT_SYNC_TOKEN }}

Done! Pushes to either gitlab or github should automatically sync. Pushes to both at the same time probably causes data loss.

Additional Notes

Token expiration

You will have to manually reset all these connections whenever your tokens expire. Which is a real bother, but such is life. Gitlab’s max token duration is 1 year. It’s likely similar on Github.

When the access token on Gitlab’s side expires, it disappears from the UI like it was never there. It’s better on Github where you get a dialog that tells you your access tokens have expired. When coming back to a project trying to figure out how I set this up, I was left confused if I had a token or not.

Destructive Github to Gitlab

Looking at the wangchucheng/git-repo-sync action that we’re using, the Github side might destroy the Gitlab side.

wangchucheng/git-repo-sync::entrypoint.sh
git remote add target https://${INPUT_TARGET_USERNAME}:${INPUT_TARGET_TOKEN}@${INPUT_TARGET_URL#https://}

case "${GITHUB_EVENT_NAME}" in
    push)
        git push -f --all target
        git push -f --tags target
        ;;
    delete)
        git push -d target ${GITHUB_EVENT_REF}
        ;;
    *)
        break
        ;;
esac

But again, I don’t have a high traffic setup. The Gitlab side has a “Keep Divergent Refs” flag that we’ve checked to prevent crushing the Github side.

If you need to fixup the sync and force push from Gitlab, you have to use the Gitlab API to toggle this flag. Here’s a list of relevant commands that assume a user Gitlab access token stored in a token file:

curl --header "PRIVATE-TOKEN: $(cat token)" "https://gitlab.com/api/v4/projects?owned=true" | jq | less
curl --header "PRIVATE-TOKEN: $(cat token)" "https://gitlab.com/api/v4/projects/50599232/remote_mirrors" | jq | less
curl --header "PRIVATE-TOKEN: $(cat token)" --request PUT --data "keep_divergent_refs=false" "https://gitlab.com/api/v4/projects/50599232/remote_mirrors/1664838"

Troubleshooting Gitlab to Github

Shitty interface, but to view Gitlab mirror errors you hover the Error status badge. Docs.

Weird issue

I had this strange issue where one my repos on gitlab.com failed to sync git LFS files to github. The repo mirroring wasn’t failing, just when viewing the file on Github it wasn’t being recognized as an LFS file. Basically it was missing, this particular tag on Github: Despite this, Github was properly reading the file and telling me it was a 3.7MB in size as specified by the LFS file description. Cloning the Github repo had an unresolved file.

I did all the machinations to determine it was:

  • not an issue with repo mirroring and git LFS (I’ve synced other LFS files)
  • not the files of the repo (I’ve force pushed an entirely different working repo)
  • not the github repo (I’ve mirrored from another working gitlab repo)

so the issue is one specific repo on my Gitlab account failing to sync to Github…

The resolution was to delete my Gitlab repository, wait a while (this is actually important… I tried immediately recreating the repo), and then recreate the sync setup.

How absolutely maddening. And it seems still slightly bugged, the lfs files still need to be manually pushed to the Github side.

Turnkey Solutions

Technically Gitlab has bidirectional mirroring. But it is a premium feature. This particular workflow can likely be improved with the notes in this document.