Git/Shallow: Difference between revisions

From Omnia
< Git
Jump to navigation Jump to search
Line 31: Line 31:
Reference:
Reference:
* git "shallow clone + unshallow" vs "normal clone" - Stack Overflow - https://stackoverflow.com/questions/67838180/git-shallow-clone-unshallow-vs-normal-clone
* git "shallow clone + unshallow" vs "normal clone" - Stack Overflow - https://stackoverflow.com/questions/67838180/git-shallow-clone-unshallow-vs-normal-clone
NOTE: Eventually, the final --unshallow gets you everything you would have gotten, had you been able to do a full clone all at once without error. Note that you may want to use --no-single-branch during the initial shallow clone, or fix up the fetch refspec after the initial shallow clone.


=== Example ===
=== Example ===
Line 89: Line 91:
  * [new tag]          Releases/SOMETAG1 -> Releases/SOMETAG1
  * [new tag]          Releases/SOMETAG1 -> Releases/SOMETAG1
  * [new tag]          Releases/SOMETAG2 -> Releases/SOMETAG2
  * [new tag]          Releases/SOMETAG2 -> Releases/SOMETAG2




Line 110: Line 111:


$ git fetch --unshallow
$ git fetch --unshallow
remote: Enumerating objects: 1737369, done.
remote: Counting objects: 100% (1719287/1719287), done.
remote: Compressing objects: 100% (455766/455766), done.
remote: Total 1710144 (delta 1268632), reused 1647283 (delta 1208565), pack-reused 0 (from 0)
Receiving objects: 100% (1710144/1710144), 1010.28 MiB | 11.93 MiB/s, done.
Resolving deltas: 100% (1268632/1268632), completed with 5822 local objects.
remote: Enumerating objects: 9, done.
remote: Total 9 (delta 0), reused 0 (delta 0), pack-reused 9 (from 1)
Unpacking objects: 100% (9/9), 2.69 KiB | 2.69 MiB/s, done.
From github.com:MYREPO/MYORG
* [new tag]                SOMETAG  -> SOMETAG


$ du ../shallow --si --max-depth=0
$ du ../shallow --si --max-depth=0
2.6G    ../shallow
</pre>
</pre>


Line 123: Line 136:


$ git fetch --unshallow
$ git fetch --unshallow
remote: Enumerating objects: 1719297, done.
remote: Counting objects: 100% (1719291/1719291), done.
remote: Compressing objects: 100% (455705/455705), done.
remote: Total 1710222 (delta 1268719), reused 1647432 (delta 1208703), pack-reused 0 (from 0)
Receiving objects: 100% (1710222/1710222), 1010.08 MiB | 11.94 MiB/s, done.
Resolving deltas: 100% (1268719/1268719), completed with 5822 local objects.
remote: Enumerating objects: 9, done.
remote: Total 9 (delta 0), reused 0 (delta 0), pack-reused 9 (from 1)
Unpacking objects: 100% (9/9), 2.69 KiB | 2.69 MiB/s, done.
From github.com:MYREPO/MYORG
* [new tag]                Releases/SOMETAG1 -> Releases/SOMETAG1
* [new tag]                Releases/SOMETAG2 -> Releases/SOMETAG2
* [new tag]                SOMETAG  -> SOMETAG


$ du ../shallow --si --max-depth=0
$ du ../shallow --si --max-depth=0
2.6G    ../shallow
</pre>
</pre>



Revision as of 23:12, 22 March 2026

Shallow Clone

git clone --filter=blob:none <url> creates a blobless clone. ...
git clone --filter=tree:0 <url> creates a treeless clone. ...
git clone --depth=1 <url> creates a shallow clone.

Unshallow Clone

To Unshallow a Shallow Clone

git clone <url> --depth 1
git fetch --unshallow

vs

git clone <url>

Incrementally:

git clone <url> --depth 1
git fetch --deepen
git fetch --deepen
...
# or
git fetch --depth 3
git fetch --depth 6
..
git fetch --unshallow

Reference:

NOTE: Eventually, the final --unshallow gets you everything you would have gotten, had you been able to do a full clone all at once without error. Note that you may want to use --no-single-branch during the initial shallow clone, or fix up the fetch refspec after the initial shallow clone.

Example

Deep:

$ git clone org-123@github.com:MYORG/MYLARGEREPO.git deep
Cloning into 'deep'...
remote: Enumerating objects: 2261419, done.
remote: Counting objects: 100% (5695/5695), done.
remote: Compressing objects: 100% (1506/1506), done.
remote: Total 2261419 (delta 5130), reused 4298 (delta 4183), pack-reused 2255724 (from 2)
Receiving objects: 100% (2261419/2261419), 2.07 GiB | 14.10 MiB/s, done.
Resolving deltas: 100% (1694598/1694598), done.
Updating files: 100% (26095/26095), done.

$ du deep --si --max-depth=0
3.4G    deep

Shallow:

$ git clone org-123@github.com:MYORG/MYLARGEREPO.git shallow --depth 1
Cloning into 'shallow'...
remote: Enumerating objects: 25232, done.
remote: Counting objects: 100% (25232/25232), done.
remote: Compressing objects: 100% (16961/16961), done.
remote: Total 25232 (delta 9701), reused 17625 (delta 7165), pack-reused 0 (from 0)
Receiving objects: 100% (25232/25232), 266.90 MiB | 10.68 MiB/s, done.
Resolving deltas: 100% (9701/9701), done.
Updating files: 100% (26095/26095), done.

$ du shallow --si --max-depth=0
1.4G    shallow

Deepend and Depth and Unshallow:

$ rsync -avrt shallow/ shallow-test/

$ cd shallow-test/

$ git fetch --deepen 1
remote: Enumerating objects: 37, done.
remote: Counting objects: 100% (37/37), done.
remote: Compressing objects: 100% (18/18), done.
remote: Total 19 (delta 17), reused 2 (delta 0), pack-reused 0 (from 0)
Unpacking objects: 100% (19/19), 2.31 KiB | 473.00 KiB/s, done.

$ git fetch --deepen 1
remote: Enumerating objects: 173, done.
remote: Counting objects: 100% (120/120), done.
remote: Compressing objects: 100% (57/57), done.
remote: Total 59 (delta 51), reused 9 (delta 2), pack-reused 0 (from 0)
Unpacking objects: 100% (59/59), 8.22 KiB | 290.00 KiB/s, done.
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
From github.com:MYORG/MYREPO
 * [new tag]           Releases/SOMETAG1 -> Releases/SOMETAG1
 * [new tag]           Releases/SOMETAG2 -> Releases/SOMETAG2


$ git fetch --depth 1
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)

$ git fetch --depth 2
remote: Enumerating objects: 13, done.
remote: Counting objects: 100% (13/13), done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 5 (delta 4), reused 1 (delta 0), pack-reused 0 (from 0)
Unpacking objects: 100% (5/5), 1.36 KiB | 1.36 MiB/s, done.

$ git fetch --depth 2
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)

$ git fetch --depth 2
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)


$ git fetch --unshallow
remote: Enumerating objects: 1737369, done.
remote: Counting objects: 100% (1719287/1719287), done.
remote: Compressing objects: 100% (455766/455766), done.
remote: Total 1710144 (delta 1268632), reused 1647283 (delta 1208565), pack-reused 0 (from 0)
Receiving objects: 100% (1710144/1710144), 1010.28 MiB | 11.93 MiB/s, done.
Resolving deltas: 100% (1268632/1268632), completed with 5822 local objects.
remote: Enumerating objects: 9, done.
remote: Total 9 (delta 0), reused 0 (delta 0), pack-reused 9 (from 1)
Unpacking objects: 100% (9/9), 2.69 KiB | 2.69 MiB/s, done.
From github.com:MYREPO/MYORG
 * [new tag]                 SOMETAG   -> SOMETAG

$ du ../shallow --si --max-depth=0
2.6G    ../shallow

Full Unshallow Test:

$ rm -rf shallow-test/

$ rsync -avrt shallow/ shallow-test/

$ cd shallow-test/

$ git fetch --unshallow
remote: Enumerating objects: 1719297, done.
remote: Counting objects: 100% (1719291/1719291), done.
remote: Compressing objects: 100% (455705/455705), done.
remote: Total 1710222 (delta 1268719), reused 1647432 (delta 1208703), pack-reused 0 (from 0)
Receiving objects: 100% (1710222/1710222), 1010.08 MiB | 11.94 MiB/s, done.
Resolving deltas: 100% (1268719/1268719), completed with 5822 local objects.
remote: Enumerating objects: 9, done.
remote: Total 9 (delta 0), reused 0 (delta 0), pack-reused 9 (from 1)
Unpacking objects: 100% (9/9), 2.69 KiB | 2.69 MiB/s, done.
From github.com:MYREPO/MYORG
 * [new tag]                 Releases/SOMETAG1 -> Releases/SOMETAG1
 * [new tag]                 Releases/SOMETAG2 -> Releases/SOMETAG2
 * [new tag]                 SOMETAG   -> SOMETAG

$ du ../shallow --si --max-depth=0
2.6G    ../shallow

Convert Cloned Repo to Shallow

git clone --depth-1 <repo> -b <branch>
git clone --depth 1  org-1234@github.com:MYORG/MYREPO.git -b MYREPO/MYBRANCH  MYBRANCH


git clean -x -f -d .

git pull --depth 1

git fetch --depth 1

git reflog expire --expire=0
git reflog expire --expire=now --all

git show-ref -s HEAD
git show-ref -s HEAD > .git/shallow

git prune
git prune-packed
git repack

git pull --shallow-since=1999-01-01
git pull --shallow-since=2026-03-19

git gc
git gc --prune=all
git gc --prune=now
git gc --aggressive


git tag -d $(git tag -l)

git fsck --full --no-reflogs


# fix:
#   $ git gc
#   warning: reflog of 'HEAD' references pruned commits
git reflog expire --all --stale-fix
git gc


du --si --max-depth=1

References:

UPDATED BRUTE FORCE CONVERSION METHOD

git config remote.origin.fetch "+refs/heads/MYREPO/MYBRANCH:refs/remotes/origin/MYREPO/MYBRANCH"
rm -rf .git/objects/*
## optional cleanup: rm -f .git/info/refs
## optional cleanup: rm -f .git/{COMMIT_EDITMSG,}
# optional:  rm .git/hooks/*
# optional:  rm -rf .git/lfs
# optional:  rm -rf .git/refs/*
# optional:  rm -rf .git/logs/*
rm .git/packed-refs
git reflog expire --all --stale-fix
git fetch
git checkout MYREPO/MYBRANCH
git pull
git gc
du --si --max-depth=1

BRUTE FORCE CONVERSION METHOD

cd .git

# delete top level files ---
#    dir:  branches/  hooks/  info/  lfs/  logs/  objects/  refs/
#    files:  COMMIT_EDITMSG  config  description  FETCH_HEAD  HEAD  index  ORIG_HEAD  packed-refs  shallow
#  removes extras: COMMIT_EDITMSG, FETCH_HEAD
#  ignore unused: description
rm -f *

# remove lfs/
rm -rf lfs/

# clear logs/ and sub-dirs
rm -rf logs/*

# clear hooks/
rm hooks/*

# clear write-projected objects/pack
#rm -f objects/pack/*
#rm objects/info/packs
rm -rf objects/*

# remove refs
rm -rf refs/*

rm info/refs

rm COMMIT_EDITMSG
rm packed-refs

# will be recreated:
rm FETCH_HEAD

# make origin head shallow
cat ORIG_HEAD > shallow


# fix config:
#  [origin]
# replace:  fetch = +refs/heads/*:refs/remotes/origin/*
# with: fetch = +refs/heads/MYREPO/MYBRANCH:refs/remotes/origin/MYREPO/MYBRANCH

git config --get remote.origin.fetch
  +refs/heads/*:refs/remotes/origin/*
# git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*"
## Limit fetch to just the one branch:
git config remote.origin.fetch "+refs/heads/MYREPO/MYBRANCH:refs/remotes/origin/MYREPO/MYBRANCH"
git config --get remote.origin.fetch
  +refs/heads/MYREPO/MYBRANCH:refs/remotes/origin/MYREPO/MYBRANCH

# copy shallow copies .git/* -- config, description, FETCH_HEAD, HEAD, ORIG_HEAD, packed-refs, shallow
#    extras:
cp ../../shallow/.git/* ./

cd ..

# Fetch - should only fetch the current branch only
git fetch