Git/Shallow: Difference between revisions

From Omnia
< Git
Jump to navigation Jump to search
 
(7 intermediate revisions by the same user not shown)
Line 4: Line 4:
  git clone --filter=tree:0 <url> creates a treeless clone. ...
  git clone --filter=tree:0 <url> creates a treeless clone. ...
  git clone --depth=1 <url> creates a shallow clone.
  git clone --depth=1 <url> creates a shallow clone.
git clone --depth=1 <url> -b <branch>  # clone at branch
References:
* Get up to speed with partial clone and shallow clone - The GitHub Blog - https://github.blog/open-source/git/get-up-to-speed-with-partial-clone-and-shallow-clone/


== Unshallow Clone ==
== Unshallow Clone ==
Line 32: Line 38:
* git "shallow clone + unshallow" vs "normal clone" - Stack Overflow - https://stackoverflow.com/questions/67838180/git-shallow-clone-unshallow-vs-normal-clone
* git "shallow clone + unshallow" vs "normal clone" - Stack Overflow - https://stackoverflow.com/questions/67838180/git-shallow-clone-unshallow-vs-normal-clone


=== Example ===
NOTE: Eventually, the final --unshallow gets you everything you would have gotten, had you been able to do a full clone all at once without error. Note that you may want to use --no-single-branch during the initial shallow clone, or fix up the fetch refspec after the initial shallow clone.
 
Deep:
<pre>
$ git clone org-123@github.com:MYORG/MYLARGEREPO.git deep
Cloning into 'deep'...
remote: Enumerating objects: 2261419, done.
remote: Counting objects: 100% (5695/5695), done.
remote: Compressing objects: 100% (1506/1506), done.
remote: Total 2261419 (delta 5130), reused 4298 (delta 4183), pack-reused 2255724 (from 2)
Receiving objects: 100% (2261419/2261419), 2.07 GiB | 14.10 MiB/s, done.
Resolving deltas: 100% (1694598/1694598), done.
Updating files: 100% (26095/26095), done.
 
$ du deep --si --max-depth=0
3.4G    deep
</pre>
 
Shallow:
<pre>
$ git clone org-123@github.com:MYORG/MYLARGEREPO.git shallow --depth 1
Cloning into 'shallow'...
remote: Enumerating objects: 25232, done.
remote: Counting objects: 100% (25232/25232), done.
remote: Compressing objects: 100% (16961/16961), done.
remote: Total 25232 (delta 9701), reused 17625 (delta 7165), pack-reused 0 (from 0)
Receiving objects: 100% (25232/25232), 266.90 MiB | 10.68 MiB/s, done.
Resolving deltas: 100% (9701/9701), done.
Updating files: 100% (26095/26095), done.
 
$ du shallow --si --max-depth=0
1.4G    shallow
</pre>
 
Deepend and Depth and Unshallow:
<pre>
$ rsync -avrt shallow/ shallow-test/
 
$ cd shallow-test/
 
$ git fetch --deepen 1
remote: Enumerating objects: 37, done.
remote: Counting objects: 100% (37/37), done.
remote: Compressing objects: 100% (18/18), done.
remote: Total 19 (delta 17), reused 2 (delta 0), pack-reused 0 (from 0)
Unpacking objects: 100% (19/19), 2.31 KiB | 473.00 KiB/s, done.
 
$ git fetch --deepen 1
remote: Enumerating objects: 173, done.
remote: Counting objects: 100% (120/120), done.
remote: Compressing objects: 100% (57/57), done.
remote: Total 59 (delta 51), reused 9 (delta 2), pack-reused 0 (from 0)
Unpacking objects: 100% (59/59), 8.22 KiB | 290.00 KiB/s, done.
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
From github.com:MYORG/MYREPO
* [new tag]          Releases/SOMETAG1 -> Releases/SOMETAG1
* [new tag]          Releases/SOMETAG2 -> Releases/SOMETAG2
 
 
 
$ git fetch --depth 1
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
 
$ git fetch --depth 2
remote: Enumerating objects: 13, done.
remote: Counting objects: 100% (13/13), done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 5 (delta 4), reused 1 (delta 0), pack-reused 0 (from 0)
Unpacking objects: 100% (5/5), 1.36 KiB | 1.36 MiB/s, done.
 
$ git fetch --depth 2
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
 
$ git fetch --depth 2
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
 
 
$ git fetch --unshallow
 
$ du ../shallow --si --max-depth=0
</pre>
 
Full Unshallow Test:
<pre>
$ rm -rf shallow-test/
 
$ rsync -avrt shallow/ shallow-test/
 
$ cd shallow-test/
 
$ git fetch --unshallow
 
$ du ../shallow --si --max-depth=0
</pre>


== Convert Cloned Repo to Shallow ==
== Convert Cloned Repo to Shallow ==
Line 182: Line 95:
rm -rf .git/objects/*
rm -rf .git/objects/*
## optional cleanup: rm -f .git/info/refs
## optional cleanup: rm -f .git/info/refs
## optional cleanup: rm -f .git/{COMMIT_EDITMSG,}
## optional cleanup: rm -f .git/{COMMIT_EDITMSG,FETCH_HEAD}
# optional:  rm .git/hooks/*
# optional:  rm .git/hooks/*
# optional:  rm -rf .git/lfs
# optional:  rm -rf .git/lfs
Line 239: Line 152:
#  [origin]
#  [origin]
# replace:  fetch = +refs/heads/*:refs/remotes/origin/*
# replace:  fetch = +refs/heads/*:refs/remotes/origin/*
# with: fetch = +refs/heads/MYREPO/MYBRANCH:refs/remotes/origin/MYREPO/MYBRANCH
# with: fetch = +refs/heads/MYBRANCH:refs/remotes/origin/MYBRANCH


git config --get remote.origin.fetch
git config --get remote.origin.fetch
Line 245: Line 158:
# git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*"
# git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*"
## Limit fetch to just the one branch:
## Limit fetch to just the one branch:
git config remote.origin.fetch "+refs/heads/MYREPO/MYBRANCH:refs/remotes/origin/MYREPO/MYBRANCH"
git config remote.origin.fetch "+refs/heads/MYBRANCH:refs/remotes/origin/MYBRANCH"
git config --get remote.origin.fetch
git config --get remote.origin.fetch
   +refs/heads/MYREPO/MYBRANCH:refs/remotes/origin/MYREPO/MYBRANCH
   +refs/heads/MYREPO/MYBRANCH:refs/remotes/origin/MYREPO/MYBRANCH
Line 258: Line 171:
git fetch
git fetch
</pre>
</pre>
=== Unshallow Clone Example ===
Deep:
<pre>
$ git clone org-123@github.com:MYORG/MYLARGEREPO.git deep
Cloning into 'deep'...
remote: Enumerating objects: 2261419, done.
remote: Counting objects: 100% (5695/5695), done.
remote: Compressing objects: 100% (1506/1506), done.
remote: Total 2261419 (delta 5130), reused 4298 (delta 4183), pack-reused 2255724 (from 2)
Receiving objects: 100% (2261419/2261419), 2.07 GiB | 14.10 MiB/s, done.
Resolving deltas: 100% (1694598/1694598), done.
Updating files: 100% (26095/26095), done.
$ du deep --si --max-depth=0
3.4G    deep
</pre>
Shallow:
<pre>
$ git clone org-123@github.com:MYORG/MYLARGEREPO.git shallow --depth 1
Cloning into 'shallow'...
remote: Enumerating objects: 25232, done.
remote: Counting objects: 100% (25232/25232), done.
remote: Compressing objects: 100% (16961/16961), done.
remote: Total 25232 (delta 9701), reused 17625 (delta 7165), pack-reused 0 (from 0)
Receiving objects: 100% (25232/25232), 266.90 MiB | 10.68 MiB/s, done.
Resolving deltas: 100% (9701/9701), done.
Updating files: 100% (26095/26095), done.
$ du shallow --si --max-depth=0
1.4G    shallow
</pre>
Deepend and Depth and Unshallow:
<pre>
$ rsync -avrt shallow/ shallow-test/
$ cd shallow-test/
$ git fetch --deepen 1
remote: Enumerating objects: 37, done.
remote: Counting objects: 100% (37/37), done.
remote: Compressing objects: 100% (18/18), done.
remote: Total 19 (delta 17), reused 2 (delta 0), pack-reused 0 (from 0)
Unpacking objects: 100% (19/19), 2.31 KiB | 473.00 KiB/s, done.
$ git fetch --deepen 1
remote: Enumerating objects: 173, done.
remote: Counting objects: 100% (120/120), done.
remote: Compressing objects: 100% (57/57), done.
remote: Total 59 (delta 51), reused 9 (delta 2), pack-reused 0 (from 0)
Unpacking objects: 100% (59/59), 8.22 KiB | 290.00 KiB/s, done.
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
From github.com:MYORG/MYREPO
* [new tag]          Releases/SOMETAG1 -> Releases/SOMETAG1
* [new tag]          Releases/SOMETAG2 -> Releases/SOMETAG2
$ git fetch --depth 1
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
$ git fetch --depth 2
remote: Enumerating objects: 13, done.
remote: Counting objects: 100% (13/13), done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 5 (delta 4), reused 1 (delta 0), pack-reused 0 (from 0)
Unpacking objects: 100% (5/5), 1.36 KiB | 1.36 MiB/s, done.
$ git fetch --depth 2
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
$ git fetch --depth 2
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
$ git fetch --unshallow
remote: Enumerating objects: 1737369, done.
remote: Counting objects: 100% (1719287/1719287), done.
remote: Compressing objects: 100% (455766/455766), done.
remote: Total 1710144 (delta 1268632), reused 1647283 (delta 1208565), pack-reused 0 (from 0)
Receiving objects: 100% (1710144/1710144), 1010.28 MiB | 11.93 MiB/s, done.
Resolving deltas: 100% (1268632/1268632), completed with 5822 local objects.
remote: Enumerating objects: 9, done.
remote: Total 9 (delta 0), reused 0 (delta 0), pack-reused 9 (from 1)
Unpacking objects: 100% (9/9), 2.69 KiB | 2.69 MiB/s, done.
From github.com:MYREPO/MYORG
* [new tag]                SOMETAG  -> SOMETAG
$ du ../shallow-test --si --max-depth=0
2.6G    ../shallow-test
---
# Fix refspec to not target single branch
$ git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*"
$ git fetch --unshallow
fatal: --unshallow on a complete repository does not make sense
# OOPS!  Needed to fix the refspec before running unshallow
</pre>
Full Unshallow Test:
<pre>
$ rm -rf shallow-test/
$ rsync -avrt shallow/ shallow-test/
$ cd shallow-test/
$ git fetch --unshallow
remote: Enumerating objects: 1719297, done.
remote: Counting objects: 100% (1719291/1719291), done.
remote: Compressing objects: 100% (455705/455705), done.
remote: Total 1710222 (delta 1268719), reused 1647432 (delta 1208703), pack-reused 0 (from 0)
Receiving objects: 100% (1710222/1710222), 1010.08 MiB | 11.94 MiB/s, done.
Resolving deltas: 100% (1268719/1268719), completed with 5822 local objects.
remote: Enumerating objects: 9, done.
remote: Total 9 (delta 0), reused 0 (delta 0), pack-reused 9 (from 1)
Unpacking objects: 100% (9/9), 2.69 KiB | 2.69 MiB/s, done.
From github.com:MYREPO/MYORG
* [new tag]                Releases/SOMETAG1 -> Releases/SOMETAG1
* [new tag]                Releases/SOMETAG2 -> Releases/SOMETAG2
* [new tag]                SOMETAG  -> SOMETAG
$ du ../shallow-test --si --max-depth=0
2.6G    ../shallow-test
---
# Fix refspec to not target single branch
$ git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*"
$ git fetch --unshallow
fatal: --unshallow on a complete repository does not make sense
# OOPS!  Needed to fix the refspec before running unshallow
</pre>
Full No-Single-Branch Unshallow Test:
<pre>
$ rm -rf shallow-test/
$ rsync -avrt shallow/ shallow-test/
$ cd shallow-test/
# Fix refspec:
$ git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*"
$ git fetch --unshallow
???
???  SOOO MUCH is scrolled off the screen passed my buffer
???
* [new tag] ...
* [new tag] ...
* [new tag] ...
$ du ../shallow-test --si --max-depth=0
2.6G    ../shallow-test
</pre>
== keywords ==

Latest revision as of 06:49, 24 March 2026

Shallow Clone

git clone --filter=blob:none <url> creates a blobless clone. ...
git clone --filter=tree:0 <url> creates a treeless clone. ...
git clone --depth=1 <url> creates a shallow clone.


git clone --depth=1 <url> -b <branch>  # clone at branch

References:

Unshallow Clone

To Unshallow a Shallow Clone

git clone <url> --depth 1
git fetch --unshallow

vs

git clone <url>

Incrementally:

git clone <url> --depth 1
git fetch --deepen
git fetch --deepen
...
# or
git fetch --depth 3
git fetch --depth 6
..
git fetch --unshallow

Reference:

NOTE: Eventually, the final --unshallow gets you everything you would have gotten, had you been able to do a full clone all at once without error. Note that you may want to use --no-single-branch during the initial shallow clone, or fix up the fetch refspec after the initial shallow clone.

Convert Cloned Repo to Shallow

git clone --depth-1 <repo> -b <branch>
git clone --depth 1  org-1234@github.com:MYORG/MYREPO.git -b MYREPO/MYBRANCH  MYBRANCH


git clean -x -f -d .

git pull --depth 1

git fetch --depth 1

git reflog expire --expire=0
git reflog expire --expire=now --all

git show-ref -s HEAD
git show-ref -s HEAD > .git/shallow

git prune
git prune-packed
git repack

git pull --shallow-since=1999-01-01
git pull --shallow-since=2026-03-19

git gc
git gc --prune=all
git gc --prune=now
git gc --aggressive


git tag -d $(git tag -l)

git fsck --full --no-reflogs


# fix:
#   $ git gc
#   warning: reflog of 'HEAD' references pruned commits
git reflog expire --all --stale-fix
git gc


du --si --max-depth=1

References:

UPDATED BRUTE FORCE CONVERSION METHOD

git config remote.origin.fetch "+refs/heads/MYREPO/MYBRANCH:refs/remotes/origin/MYREPO/MYBRANCH"
rm -rf .git/objects/*
## optional cleanup: rm -f .git/info/refs
## optional cleanup: rm -f .git/{COMMIT_EDITMSG,FETCH_HEAD}
# optional:  rm .git/hooks/*
# optional:  rm -rf .git/lfs
# optional:  rm -rf .git/refs/*
# optional:  rm -rf .git/logs/*
rm .git/packed-refs
git reflog expire --all --stale-fix
git fetch
git checkout MYREPO/MYBRANCH
git pull
git gc
du --si --max-depth=1

BRUTE FORCE CONVERSION METHOD

cd .git

# delete top level files ---
#    dir:  branches/  hooks/  info/  lfs/  logs/  objects/  refs/
#    files:  COMMIT_EDITMSG  config  description  FETCH_HEAD  HEAD  index  ORIG_HEAD  packed-refs  shallow
#  removes extras: COMMIT_EDITMSG, FETCH_HEAD
#  ignore unused: description
rm -f *

# remove lfs/
rm -rf lfs/

# clear logs/ and sub-dirs
rm -rf logs/*

# clear hooks/
rm hooks/*

# clear write-projected objects/pack
#rm -f objects/pack/*
#rm objects/info/packs
rm -rf objects/*

# remove refs
rm -rf refs/*

rm info/refs

rm COMMIT_EDITMSG
rm packed-refs

# will be recreated:
rm FETCH_HEAD

# make origin head shallow
cat ORIG_HEAD > shallow


# fix config:
#  [origin]
# replace:  fetch = +refs/heads/*:refs/remotes/origin/*
# with: fetch = +refs/heads/MYBRANCH:refs/remotes/origin/MYBRANCH

git config --get remote.origin.fetch
  +refs/heads/*:refs/remotes/origin/*
# git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*"
## Limit fetch to just the one branch:
git config remote.origin.fetch "+refs/heads/MYBRANCH:refs/remotes/origin/MYBRANCH"
git config --get remote.origin.fetch
  +refs/heads/MYREPO/MYBRANCH:refs/remotes/origin/MYREPO/MYBRANCH

# copy shallow copies .git/* -- config, description, FETCH_HEAD, HEAD, ORIG_HEAD, packed-refs, shallow
#    extras:
cp ../../shallow/.git/* ./

cd ..

# Fetch - should only fetch the current branch only
git fetch

Unshallow Clone Example

Deep:

$ git clone org-123@github.com:MYORG/MYLARGEREPO.git deep
Cloning into 'deep'...
remote: Enumerating objects: 2261419, done.
remote: Counting objects: 100% (5695/5695), done.
remote: Compressing objects: 100% (1506/1506), done.
remote: Total 2261419 (delta 5130), reused 4298 (delta 4183), pack-reused 2255724 (from 2)
Receiving objects: 100% (2261419/2261419), 2.07 GiB | 14.10 MiB/s, done.
Resolving deltas: 100% (1694598/1694598), done.
Updating files: 100% (26095/26095), done.

$ du deep --si --max-depth=0
3.4G    deep

Shallow:

$ git clone org-123@github.com:MYORG/MYLARGEREPO.git shallow --depth 1
Cloning into 'shallow'...
remote: Enumerating objects: 25232, done.
remote: Counting objects: 100% (25232/25232), done.
remote: Compressing objects: 100% (16961/16961), done.
remote: Total 25232 (delta 9701), reused 17625 (delta 7165), pack-reused 0 (from 0)
Receiving objects: 100% (25232/25232), 266.90 MiB | 10.68 MiB/s, done.
Resolving deltas: 100% (9701/9701), done.
Updating files: 100% (26095/26095), done.

$ du shallow --si --max-depth=0
1.4G    shallow

Deepend and Depth and Unshallow:

$ rsync -avrt shallow/ shallow-test/

$ cd shallow-test/

$ git fetch --deepen 1
remote: Enumerating objects: 37, done.
remote: Counting objects: 100% (37/37), done.
remote: Compressing objects: 100% (18/18), done.
remote: Total 19 (delta 17), reused 2 (delta 0), pack-reused 0 (from 0)
Unpacking objects: 100% (19/19), 2.31 KiB | 473.00 KiB/s, done.

$ git fetch --deepen 1
remote: Enumerating objects: 173, done.
remote: Counting objects: 100% (120/120), done.
remote: Compressing objects: 100% (57/57), done.
remote: Total 59 (delta 51), reused 9 (delta 2), pack-reused 0 (from 0)
Unpacking objects: 100% (59/59), 8.22 KiB | 290.00 KiB/s, done.
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
From github.com:MYORG/MYREPO
 * [new tag]           Releases/SOMETAG1 -> Releases/SOMETAG1
 * [new tag]           Releases/SOMETAG2 -> Releases/SOMETAG2


$ git fetch --depth 1
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)

$ git fetch --depth 2
remote: Enumerating objects: 13, done.
remote: Counting objects: 100% (13/13), done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 5 (delta 4), reused 1 (delta 0), pack-reused 0 (from 0)
Unpacking objects: 100% (5/5), 1.36 KiB | 1.36 MiB/s, done.

$ git fetch --depth 2
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)

$ git fetch --depth 2
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)


$ git fetch --unshallow
remote: Enumerating objects: 1737369, done.
remote: Counting objects: 100% (1719287/1719287), done.
remote: Compressing objects: 100% (455766/455766), done.
remote: Total 1710144 (delta 1268632), reused 1647283 (delta 1208565), pack-reused 0 (from 0)
Receiving objects: 100% (1710144/1710144), 1010.28 MiB | 11.93 MiB/s, done.
Resolving deltas: 100% (1268632/1268632), completed with 5822 local objects.
remote: Enumerating objects: 9, done.
remote: Total 9 (delta 0), reused 0 (delta 0), pack-reused 9 (from 1)
Unpacking objects: 100% (9/9), 2.69 KiB | 2.69 MiB/s, done.
From github.com:MYREPO/MYORG
 * [new tag]                 SOMETAG   -> SOMETAG

$ du ../shallow-test --si --max-depth=0
2.6G    ../shallow-test

---

# Fix refspec to not target single branch
$ git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*"

$ git fetch --unshallow
fatal: --unshallow on a complete repository does not make sense

# OOPS!  Needed to fix the refspec before running unshallow

Full Unshallow Test:

$ rm -rf shallow-test/

$ rsync -avrt shallow/ shallow-test/

$ cd shallow-test/

$ git fetch --unshallow
remote: Enumerating objects: 1719297, done.
remote: Counting objects: 100% (1719291/1719291), done.
remote: Compressing objects: 100% (455705/455705), done.
remote: Total 1710222 (delta 1268719), reused 1647432 (delta 1208703), pack-reused 0 (from 0)
Receiving objects: 100% (1710222/1710222), 1010.08 MiB | 11.94 MiB/s, done.
Resolving deltas: 100% (1268719/1268719), completed with 5822 local objects.
remote: Enumerating objects: 9, done.
remote: Total 9 (delta 0), reused 0 (delta 0), pack-reused 9 (from 1)
Unpacking objects: 100% (9/9), 2.69 KiB | 2.69 MiB/s, done.
From github.com:MYREPO/MYORG
 * [new tag]                 Releases/SOMETAG1 -> Releases/SOMETAG1
 * [new tag]                 Releases/SOMETAG2 -> Releases/SOMETAG2
 * [new tag]                 SOMETAG   -> SOMETAG

$ du ../shallow-test --si --max-depth=0
2.6G    ../shallow-test

---

# Fix refspec to not target single branch
$ git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*"

$ git fetch --unshallow
fatal: --unshallow on a complete repository does not make sense

# OOPS!  Needed to fix the refspec before running unshallow


Full No-Single-Branch Unshallow Test:

$ rm -rf shallow-test/

$ rsync -avrt shallow/ shallow-test/

$ cd shallow-test/

# Fix refspec:
$ git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*"

$ git fetch --unshallow
???
???  SOOO MUCH is scrolled off the screen passed my buffer
???
 * [new tag] ...
 * [new tag] ...
 * [new tag] ...

$ du ../shallow-test --si --max-depth=0
2.6G    ../shallow-test

keywords