Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[funtoo / git] cleaning of .git
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
avx
Advocate
Advocate


Joined: 21 Jun 2004
Posts: 2152

PostPosted: Sat Sep 05, 2009 1:53 pm    Post subject: [funtoo / git] cleaning of .git Reply with quote

I noticed the free space on my partition holding portage is getting smaller and smaller and after some investigation, I found out it's related to the .git-directory inside. This dir now has a whooping weigth of 310MB (and counting). I know that git stores some - or better, a whole lot - of metadata in there, most of which I'll probably never use, so is there a safe way to do some spring-cleaning in there?

thanks,
ph

PS: Before you ask, diskspace does matter on this box and mounting portage via i.e. nfs is no option here.
Back to top
View user's profile Send private message
Mike Hunt
Watchman
Watchman


Joined: 19 Jul 2009
Posts: 5287

PostPosted: Sat Sep 05, 2009 2:11 pm    Post subject: Reply with quote

The only way I was able to "spring clean" funtoo git portage is unmounting and reformatting the partition, then untar and git checkout a fresh snapshot.

Otherwise I have found that the funtoo portage is quite sensitive to "fiddling".

The funtoo FAQ "Forking the Portage Tree" may provide some clues though. :)
Back to top
View user's profile Send private message
avx
Advocate
Advocate


Joined: 21 Jun 2004
Posts: 2152

PostPosted: Sat Sep 05, 2009 2:26 pm    Post subject: Reply with quote

Erm, yay, that's surely a way, but... one major problem with this is, that the filesystem resides on a cf-card(pda, so no better way) and this would pretty surely trash the disk in "no time".

Thinking about it, how big must .git on kernel.org be, there the people are checking-in/-out 24/7 and I'm only doing it ~twice a weak and for ~3 months now.

Edit, normal portage is no option, at least not as long as it takes ages to bring some stuff from the overlays into portage and I don't want to manage dozens of overlays myself.
Back to top
View user's profile Send private message
Naib
Watchman
Watchman


Joined: 21 May 2004
Posts: 5712
Location: Removed by Neddy

PostPosted: Sat Sep 05, 2009 2:33 pm    Post subject: Reply with quote

try:
git clean -X && git gc

it should clean it up a bit
_________________
The best argument against democracy is a five-minute conversation with the average voter
Great Britain is a republic, with a hereditary president, while the United States is a monarchy with an elective king
Back to top
View user's profile Send private message
Earthwings
Bodhisattva
Bodhisattva


Joined: 14 Apr 2003
Posts: 7753
Location: Karlsruhe, Germany

PostPosted: Sat Sep 05, 2009 3:26 pm    Post subject: Reply with quote

Moved from Other Things Gentoo to Unsupported Software.
_________________
KDE
Back to top
View user's profile Send private message
avx
Advocate
Advocate


Joined: 21 Jun 2004
Posts: 2152

PostPosted: Sat Sep 05, 2009 5:31 pm    Post subject: Reply with quote

Mh, is there a way to generate the needed temporary data in another place? I've only got ~150MB left on this partition and that's not enough.
Back to top
View user's profile Send private message
Mike Hunt
Watchman
Watchman


Joined: 19 Jul 2009
Posts: 5287

PostPosted: Sat Sep 05, 2009 6:13 pm    Post subject: Reply with quote

git-gc looks promising. The manpage describes several options. Let me know how that goes (if you try it).

git-fsck might also help.

You could symlink /usr/portage/distfiles to some other place.
Back to top
View user's profile Send private message
avx
Advocate
Advocate


Joined: 21 Jun 2004
Posts: 2152

PostPosted: Sat Sep 05, 2009 6:57 pm    Post subject: Reply with quote

Now out to the airport, trying it later. But symlinking distfiles, mh, won't help, nothing in there and git's tmpfiles seem to go to .git/objects/... as default.
Back to top
View user's profile Send private message
avx
Advocate
Advocate


Joined: 21 Jun 2004
Posts: 2152

PostPosted: Sat Sep 05, 2009 9:42 pm    Post subject: Reply with quote

Tried `git gc --aggressive` which reduced the size to ~240MB, imho still pretty heavy. Someone using funtoo can tell me, what their `du -h $PORTDIR/.git` gives?

Edit, did an `cd $portage/.git ; rm -rf objects refs; emerge --sync` and now I'm at 13MB, will see how this one grows in the next few days or if I suffered from a bug before.
Back to top
View user's profile Send private message
Mike Hunt
Watchman
Watchman


Joined: 19 Jul 2009
Posts: 5287

PostPosted: Wed Sep 09, 2009 11:43 pm    Post subject: Reply with quote

Wow 268M and that's after running git-gc.

Code:
# du -h .git
3.0K    .git/info
19K     .git/logs/refs/remotes/origin
20K     .git/logs/refs/remotes
13K     .git/logs/refs/heads
34K     .git/logs/refs
47K     .git/logs
1.0K    .git/refs/tags
2.0K    .git/refs/remotes/origin
3.0K    .git/refs/remotes
1.0K    .git/refs/heads
6.0K    .git/refs
19K     .git/hooks
2.0K    .git/objects/info
255M    .git/objects/pack
255M    .git/objects
268M    .git


Looks like it's the objects/pack/pack-nnn.pack file that's taking 95% of the diskspace.
Back to top
View user's profile Send private message
avx
Advocate
Advocate


Joined: 21 Jun 2004
Posts: 2152

PostPosted: Thu Sep 10, 2009 12:00 am    Post subject: Reply with quote

Yep, annoying. Just setup a new system, /portage is on reiser4, complete /portage is 493MB and .git is 272MB - still looking for good solutions :roll:
Back to top
View user's profile Send private message
yoshi314
l33t
l33t


Joined: 30 Dec 2004
Posts: 848
Location: PL

PostPosted: Thu Sep 10, 2009 10:54 am    Post subject: Reply with quote

Quote:
Mh, is there a way to generate the needed temporary data in another place?
there is a way to mount two dirs via aufs/unionfs so that writes are redirected to one directory, and reads are taken from union of both.

this way you could keep ebuilds in one filesystem and the git metadata on another. and set up some kind of rsync job to move ebuilds and assorted files to first filesystem and keep the git data on another one.

it's just a rough idea, but it definitely is possible. it's a variation on keeping your portage on squashfs with writable /dev/shm on top of it (portage is really small this way - nowadays ~48mb)
_________________
~amd64
shrink your /usr/portage with squashfs+aufs
Back to top
View user's profile Send private message
Naib
Watchman
Watchman


Joined: 21 May 2004
Posts: 5712
Location: Removed by Neddy

PostPosted: Thu Sep 10, 2009 11:47 am    Post subject: Reply with quote

Code:

git repack -a -d -l


might be worth a shot
_________________
The best argument against democracy is a five-minute conversation with the average voter
Great Britain is a republic, with a hereditary president, while the United States is a monarchy with an elective king
Back to top
View user's profile Send private message
avx
Advocate
Advocate


Joined: 21 Jun 2004
Posts: 2152

PostPosted: Thu Sep 10, 2009 12:43 pm    Post subject: Reply with quote

Quote:
might be worth a shot
Yay, another 2MB won. :/

yoshi314, thanks, I'm considering this, weekend's coming.
Back to top
View user's profile Send private message
avx
Advocate
Advocate


Joined: 21 Jun 2004
Posts: 2152

PostPosted: Wed Sep 23, 2009 4:30 pm    Post subject: Reply with quote

Woha, just cloned the zen-sources...
Code:
du -hs /usr/src/zen -> 916M
du -hs /usr/src/zen/.git -> 410M
even worse, this is on Reiser4, again.

Edit, after `git clean -Xf; git repack -a -d -l; git gc --agressive`, which took about one hour total, .git is down to 355M - still sucks.
Back to top
View user's profile Send private message
Naib
Watchman
Watchman


Joined: 21 May 2004
Posts: 5712
Location: Removed by Neddy

PostPosted: Wed Sep 23, 2009 7:26 pm    Post subject: Reply with quote

ph030 wrote:
Woha, just cloned the zen-sources...
Code:
du -hs /usr/src/zen -> 916M
du -hs /usr/src/zen/.git -> 410M
even worse, this is on Reiser4, again.

Edit, after `git clean -Xf; git repack -a -d -l; git gc --agressive`, which took about one hour total, .git is down to 355M - still sucks.


well the zen-sources are taken straight from the main linux src, check the history
Code:

git log $(git rev-list HEAD | tail -n1)

commit 1da177e4c3f41524e886b7f1b8a0c1fc7321cac2
Author: Linus Torvalds <torvalds@ppc970.osdl.org>
Date:   Sat Apr 16 15:20:36 2005 -0700

    Linux-2.6.12-rc2
   
    Initial git repository build. I'm not bothering with the full history,
    even though we have it. We can create a separate "historical" git
    archive of that later if we want to, and in the meantime it's about
    3.2GB when imported into git - space that would just make the early
    git days unnecessarily complicated, when we don't have a lot of good
    infrastructure for it.
   
    Let it rip!


4years worth of commits,branches,merges,rebases, patches.
do a gitk on the folder (althought gitg is better ;) and see how much is actually stored

I do see the merit that downstream users probably don't have a use for all this but hey \o/
_________________
The best argument against democracy is a five-minute conversation with the average voter
Great Britain is a republic, with a hereditary president, while the United States is a monarchy with an elective king
Back to top
View user's profile Send private message
avx
Advocate
Advocate


Joined: 21 Jun 2004
Posts: 2152

PostPosted: Wed Sep 23, 2009 11:21 pm    Post subject: Reply with quote

Mh, reading the `git help clone` I found the --depth X option, setting X=1 brings .git down to 134M - way better, yet not percect.

edit, the same "trick" brought down the funtoo-tree from 505M(incl. .git 283M) to 314M(incl. .git 93M).

Now trying what git-gc and repack can do.
Back to top
View user's profile Send private message
tcx
n00b
n00b


Joined: 23 Mar 2005
Posts: 32
Location: Porto, Portugal

PostPosted: Tue Oct 06, 2009 2:39 pm    Post subject: Reply with quote

8O
I was totally unaware of that.
Apparently I haven't been checking my portage tree with enough care.
Has anyone found a better solution?
Back to top
View user's profile Send private message
Mike Hunt
Watchman
Watchman


Joined: 19 Jul 2009
Posts: 5287

PostPosted: Fri Oct 09, 2009 3:03 am    Post subject: Reply with quote

ph030, is git clone --depth 1 affecting performance ?
Back to top
View user's profile Send private message
avx
Advocate
Advocate


Joined: 21 Jun 2004
Posts: 2152

PostPosted: Fri Oct 09, 2009 3:55 pm    Post subject: Reply with quote

Good question, haven't `time`d that. Doing it right now with zen-stable, stay tuned.

Edit, didn't use `time`, instead used
Code:
date > file.txt ; git clone git://zen-sources.org/kernel/zen-stable.git zen && sync && date >> file.txt
To be sure, I rebooted between the tries.

timediff normal run: ~17 minutes
timediff depth 1 run: ~4.5 minutes

Specs: CPU 1.2GHz Pentium M, 4.200rpm HDD, max. download capacity 1.4MB/s (execpt the checkout it's been idle), filesystem is reiser4 (not the latest version with the fast-sync patches). If needed, request `time`d results.
Back to top
View user's profile Send private message
Mike Hunt
Watchman
Watchman


Joined: 19 Jul 2009
Posts: 5287

PostPosted: Mon Oct 12, 2009 10:57 pm    Post subject: Reply with quote

That is nice improvement if a truncated history is sufficient. Have you tried it with git-portage?
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum