Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Keeping /usr/portage/distfiles tidy
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Installing Gentoo
View previous topic :: View next topic  
Author Message
niranjana_km
n00b
n00b


Joined: 15 Mar 2014
Posts: 14

PostPosted: Wed Dec 31, 2014 8:22 am    Post subject: Keeping /usr/portage/distfiles tidy Reply with quote


Keeping /usr/portage/distfiles tidy


When we keep updating gentoo, over a time the source packages accumulate in /usr/portage/distfiles. This consumes a lot of disk space because of source packages of old versions of installed programs still sitting there. To clean it manually is tedious as there are thousands of packages. And deleting all the source packages in it is not a wise thing as they are required when rebuilding something due to dependency or USE change. Here is a handy script which lists the source packages used for current installation and those packages not used (due to an update) but still sitting in /usr/portage/distfiles. After listing here is another script which can move the 'not used' packages into a directory, which you can then move it to your storage devices for backup (for any future needs) or delete it. The listing script is as follows,

Code:

#!/bin/bash

# Old package source files lister for Gentoo.

# The package database location: "/var/db/pkg"

bzcat -k /var/db/pkg/*/*/environment.bz2 | grep "declare -x A=" >Adistlist
bzcat -k /var/db/pkg/*/*/environment.bz2 | grep "declare -x AA=" >AAdistlist

sed -e "s/declare -x A=\"//g" -e "s/\"//g" -e "s/ /\n/g" Adistlist >installed
sed -e "s/declare -x AA=\"//g" -e "s/\"//g" -e "s/ /\n/g" AAdistlist >>installed

sort -u installed >xinstalled
mv xinstalled installed

ls -1 /usr/portage/distfiles >allfiles
cp allfiles uninstalled
for i in $(cat installed)
do
sed --in-place -e "s/$i//1" uninstalled
done

cat uninstalled | tr -s '\n' >xuninstalled
mv xuninstalled uninstalled

# EOF



And the moving script is,

Code:

#!/bin/bash

# Old package source files cleaner for Gentoo.

# You need not to run this script as root. Because, "/usr/portage/distfiles"
# directory is owned by `portage` and belongs to `portage` group. It has read/write
# permission for all the group members and if you are a member of the `portage` group you can also run this script.

distdir="/usr/portage/distfiles"
listdir="./"  # Directory where the above prepared list is sitting.
mvtodir="/usr/portage/distfiles/duninstalled-$(date +%Y%m%d)"

cd $distdir
mkdir $mvtodir

for i in $(cat $listdir/uninstalled)
do
mv -t $mvtodir $i
done


## Alternate method:
## First, the name of the source files needed in the current installation are
## listed, sorted and uniquified using the script 'clean-distfiles/list-distfiles.sh'.
## Then the source files listed are moved to a directory named 'installed' and
## whatever remains is moved to a directory named 'uninstalled'. Then the contents
## of the directory 'installed' are moved back into the '/usr/portage/distfiles/' directory.
## Then the directory 'uninstalled' is moved to external harddisk with date for backup.
# for i in $(cat installed)
# do
# mv -t dinstalled $i
# done
#
# for i in *
# do
# mv -t duninstalled $i
# done


# EOF


Quote:

Note: Not tested with -9999 ebuilds i.e the program source fetched with a revision control system like git, subversion, bazar etc...


Happy new year Gentoo. Love you....
Back to top
View user's profile Send private message
charles17
Advocate
Advocate


Joined: 02 Mar 2008
Posts: 2616

PostPosted: Wed Dec 31, 2014 10:51 am    Post subject: Reply with quote

What is the point of not using eclean-dist?
Quote:
$ eclean-dist -d
Back to top
View user's profile Send private message
Clad in Sky
l33t
l33t


Joined: 04 May 2007
Posts: 852
Location: Germany

PostPosted: Wed Dec 31, 2014 11:17 am    Post subject: Reply with quote

Probably OP wasn't aware of this. Neither was I. So thanks to OP and to you for pointing me at this.
_________________
Kali Ma
Now it's autumn of the aeons
Dance with your sword
Now it's time for the harvest
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43212
Location: 56N 3W

PostPosted: Wed Dec 31, 2014 11:43 am    Post subject: Reply with quote

Clad in Sky,

eclean works for packages too.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
niranjana_km
n00b
n00b


Joined: 15 Mar 2014
Posts: 14

PostPosted: Thu Jan 01, 2015 8:12 am    Post subject: Reply with quote

Ya, I was not aware of 'eclean', thanks charles17 for pointing it..
Back to top
View user's profile Send private message
niranjana_km
n00b
n00b


Joined: 15 Mar 2014
Posts: 14

PostPosted: Thu Jan 01, 2015 8:23 am    Post subject: Reply with quote

`man eclean` gave the following two links where `eclean` was born in 2002,

https://forums.gentoo.org/viewtopic.php?t=3011
https://bugs.gentoo.org/show_bug.cgi?id=33877

Need is the mother of invention....
Back to top
View user's profile Send private message
charles17
Advocate
Advocate


Joined: 02 Mar 2008
Posts: 2616

PostPosted: Sat Jan 03, 2015 9:19 am    Post subject: Reply with quote

niranjana_km wrote:
Ya, I was not aware of 'eclean', thanks charles17 for pointing it..
Another nice tool you should try is
Quote:
$ eix-test-obsolete
Back to top
View user's profile Send private message
niranjana_km
n00b
n00b


Joined: 15 Mar 2014
Posts: 14

PostPosted: Wed Jan 28, 2015 5:55 am    Post subject: Reply with quote

Yes, since it uses cache the search is very quick and nice. I started using eix instead of equery. Thanks...
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6281

PostPosted: Wed Jan 28, 2015 7:17 am    Post subject: Reply with quote

I had bad experiences with eclean since it does not honour your useflags and thus keeps too many files.

What I use is the trickyfetch script from the mv overlay and its "obsolete" command: This moves all your sourcefiles to $DISTDIR/.obsolete and causes portage to fetch all files of all of your packages (according to your useflags); "fetching" means first to move from $DISTDIR/.obsolete (and some other directories), and only if it is not found to fallback to wget (or whatever you want to use as FETCHCOMMAND). Hence, in the end, all your "unneeded" files remain in $DISTDIR/.obsolete, and you can decide whether to clean them all or keep some of them. (And BTW you also realize if there were file bumps without a revision bump so that you are guaranteed to rebuild all of your packages without internet access.)
Back to top
View user's profile Send private message
szczerb
Veteran
Veteran


Joined: 24 Feb 2007
Posts: 1701
Location: Poland => Lodz

PostPosted: Wed Jan 28, 2015 10:40 am    Post subject: Reply with quote

I'm just using this:
Code:
find /usr/portage/distfiles -maxdepth 1 -type f -mtime +30 -exec rm -v {} \;
Back to top
View user's profile Send private message
cwr
Veteran
Veteran


Joined: 17 Dec 2005
Posts: 1969

PostPosted: Wed Jan 28, 2015 1:48 pm    Post subject: Reply with quote

szczerb wrote:
I'm just using this:
Code:
find /usr/portage/distfiles -maxdepth 1 -type f -mtime +30 -exec rm -v {} \;


I'm surprised that that works - a lot of my portage files are dated before 2010.

Will
Back to top
View user's profile Send private message
niranjana_km
n00b
n00b


Joined: 15 Mar 2014
Posts: 14

PostPosted: Thu Jan 29, 2015 7:27 am    Post subject: Reply with quote

The trickyfetch script method seems to be little inefficient. I have not used it, but its above description by "mv" made me to think so. How much time it takes to complete it?

And the method used by "szczerb" doesn't guarantee retention of all the source packages used in present installation.
Back to top
View user's profile Send private message
charles17
Advocate
Advocate


Joined: 02 Mar 2008
Posts: 2616

PostPosted: Thu Jan 29, 2015 8:56 am    Post subject: Reply with quote

mv wrote:
I had bad experiences with eclean since it does not honour your useflags and thus keeps too many files.
Could you give an example of such file(s) being kept because of an non-honored USE flag?
Me thinks it's worth filing a bug to get eclean improved.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6281

PostPosted: Thu Jan 29, 2015 9:15 am    Post subject: Reply with quote

charles17 wrote:
mv wrote:
I had bad experiences with eclean since it does not honour your useflags and thus keeps too many files.
Could you give an example of such file(s) being kept because of an non-honored USE flag?
Me thinks it's worth filing a bug to get eclean improved.

Examples depend of course on your useflags. For instance, many language files of firefox are only downloaded if certain linguas_...-USE-Flags are set.
This is not a "bug" which can be easily fixed: You need knowledge about the whole user configuration (not only make.conf and /usr/portage/package.use but also selected profile(s), USE-expand variables etc.) to get the currently active USE-flag setting.
Moreover, there are probably some files which you never want to be removed, even if you do not have them installed currently (e.g. because of dead upstream or fetch-restrictions or whatever).
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6281

PostPosted: Thu Jan 29, 2015 9:27 am    Post subject: Reply with quote

niranjana_km wrote:
The trickyfetch script method seems to be little inefficient. I have not used it, but its above description by "mv" made me to think so. How much time it takes to complete it?

The overhead over portage is almost negligible (except for the texlive packages which you can easily manually move out from .obsolete if you do not want to check them, too): The needed time is essentially that of
Code:
emerge -efD --with=bdeps=y @world
On my system (athlon, ~1200 packages, including full texlive) it takes around 10 minutes, because one cannot configure portage to omit doing the checksums for every file. On the other hand, only this way it can be guaranteed that there have not been any changes in the portage tree concerning your downloaded files.
I do not cleanup after every emerge --sync but only e.g. before I make a backup of my main system (which I do less frequently), so these 10 minutes or so are not very disturbing.

The advantage of all this is that you can influence a lot manually: For instance, if you use the same DISTDIR for several machines/chroots (with different configurations), you just have to call the above emerge command on every machine/chroot, and then the files remaining in .obsolete are those used by neither.
Or if you want to keep e.g. the sources for adobe-flash for all architectures, although you currently have installed only amd64, you just call
Code:
emerge -1FO adobe-flash
before cleaning up your .obsolete directory. Moreover, I prefer to keep some source-files (e.g. for games or other optional packages which I install on only a few systems) in separate directories: The trickyfetch script can generate corresponding symlinks and just moves out these symlinks from .save.* directories if needed.
There is also a .restricted directory in which I put symlinks to files with fetch restrictions: The trickyfetch script takes care about moving them out, so that you do not run into errors about nonexisting sources for fetch-restricted packages.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Installing Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum