Chunked git-push into Azure DevOps
The other day I encountered a fun little problem while trying to import a large git repository into Azure DevOps:
Very large pushes use up a lot of resources, blocking or slowing other parts of the service. Such pushes often don’t map to normal software development activities. Someone may have inadvertently checked in build outputs or a VM image, for example. For these reasons and more, pushes are limited to 5GB at a time.
– Azure DevOps Documentation (emphasis mine)
So, well .. this sucks as my repository at hand is way bigger than 5GB 😢
❌ Solution: Web Import
What does Microsoft suggest as a solution? Just import the git repository via the web import feature!
Sadly this was not an option as the source git repository is not directly accessible by Azure DevOps 😢 😢
✅ Solution: Chunked
Well, as you might have noticed only a single push can not exceed the 5GB limit as the repository itself can easily grow up to 250GB.
Ha! Now we have an angle of attack! We simply push our repository in small chunks, ensure that we never exceed the limit for a single
git push and some time later we should arrive at the current
And with the power of bash the whole taks was as simple as:
1$ # Ensure we're starting from the master branch 2$ git checkout master 3 4$ # First we need to count the total amount of commits 5$ TOTAL_COMMITS=$(git log --pretty=oneline | wc -l) 6 7$ # How many commits should be in a single chunk? 8$ COMMITS_PER_CHUNK=50 9 10$ # Push our commits in small chunks 11$ for CURSOR in $(seq $TOTAL_COMMITS -$COMMITS_PER_CHUNK 0); do \ 12 git push origin master~$CURSOR:master; \ 13 done 14 15$ # And finally push the missing rest (tags, etc.) 16$ git push --mirror origin
In my case it was enought to run this once on the
master branch, as the final
git push --mirror will push everything else (tags and all other branches).
git push --mirrormight also run into the same 5GB limit if you have a lot and/or big commits outside the
masterbranch! If so, just repeat the steps from above until
git push --mirroralso stays within the limit.