Chunked git-push into Azure DevOps

The other day I encountered a fun little problem while trying to import a large git repository into Azure DevOps:

Very large pushes use up a lot of resources, blocking or slowing other parts of the service. Such pushes often don’t map to normal software development activities. Someone may have inadvertently checked in build outputs or a VM image, for example. For these reasons and more, pushes are limited to 5GB at a time.

– Azure DevOps Documentation (emphasis mine)

So, well .. this sucks as my repository at hand is way bigger than 5GB 😢

❌ Solution: Web Import

What does Microsoft suggest as a solution? Just import the git repository via the web import feature!

Sadly this was not an option as the source git repository is not directly accessible by Azure DevOps 😢 😢

✅ Solution: Chunked git push

Well, as you might have noticed only a single push can not exceed the 5GB limit as the repository itself can easily grow up to 250GB.

Ha! Now we have an angle of attack! We simply push our repository in small chunks, ensure that we never exceed the limit for a single git push and some time later we should arrive at the current HEAD 😎

And with the power of bash the whole taks was as simple as:

 1$ # Ensure we're starting from the master branch
 2$ git checkout master
 3
 4$ # First we need to count the total amount of commits
 5$ TOTAL_COMMITS=$(git log --pretty=oneline | wc -l)
 6
 7$ # How many commits should be in a single chunk?
 8$ COMMITS_PER_CHUNK=50
 9
10$ # Push our commits in small chunks
11$ for CURSOR in $(seq $TOTAL_COMMITS -$COMMITS_PER_CHUNK 0); do \
12    git push origin master~$CURSOR:master;  \
13  done
14
15$ # And finally push the missing rest (tags, etc.)
16$ git push --mirror origin

In my case it was enought to run this once on the master branch, as the final git push --mirror will push everything else (tags and all other branches).

The last git push --mirror might also run into the same 5GB limit if you have a lot and/or big commits outside the master branch! If so, just repeat the steps from above until git push --mirror also stays within the limit.