Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
simple script to sync portage to squashfs with overlayfs
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
yoshi314
l33t
l33t


Joined: 30 Dec 2004
Posts: 848
Location: PL

PostPosted: Thu Feb 12, 2015 9:52 am    Post subject: simple script to sync portage to squashfs with overlayfs Reply with quote

I've made a simplistic script to easily keep portage in squashfs, for some serious space savings, utilizing overlayfs.

I thought this was way too dumb to post here, but few users already expressed interest in it, so i'm posting this.



It should be possible to make a portage sync hook out of it, once the portage version with sync hooks is unmasked and i get to figure this out.
Personally, i use it with paludis - i disabled default gentoo sync, and added the script as sync_all_pre hook, so it runs before everything else is synced.


Code:
#!/bin/sh

#
# directories to use; overlayfs uses ro dir, rw dir and workdir. all of them are combined into final mount
#

sqfs=/dev/shm/portage
mkdir -p ${sqfs}
mkdir -p ${sqfs}-work


#
# where to keep squashfs files
#

storage=/mnt/storage

#
# this is where everything gets mounted eventually
#

tempmount=/mnt/storage/ovfs

# check for recent archives
#
# if there is an archive from last 30 minutes, don't sync again
#
# i do this, because some git syncs sometimes timeout for me (e.g. pentoo or backbone),
# and i re-run entire sync procedure. Re-syncing gentoo repo this often is unecessary.
#

howmany=$(find ${storage} -maxdepth 1 -name "portage*.sqfs" -type f -mmin -30 | wc -l)

[ "$howmany" -gt 0 ] && exit 0


#
# /usr/portage is squashfs mount
#
# we mount ro portion as lowerdir , upperdir will keep the new and modified files and workdir keeps track of things
# ${tempmount} is the final mount location we will be working on
#

#overlayfs
mount -t overlay overlay -olowerdir=/usr/portage,upperdir=${sqfs},workdir=${sqfs}-work ${tempmount}

#
# use your favorite rsync mirror of choice
# exclusion of changelogs can cut down few mb of final squashfs file
#
rsync -avr --delete-during --progress --exclude="*ChangeLog" rsync://rsync.pl.gentoo.org/gentoo-portage/ ${tempmount}/


#
# i name files like portage-YYYY-MM-DD-HH-MM-SS.sqfs
# and link most recent one to portage-current.sqfs and mount that
#

date=$(date +%F-%H-%M-%S)

mksquashfs ${tempmount} /mnt/storage/portage-${date}.sqfs -comp xz -progress


#
# clean up overlayfs mount
#
umount ${tempmount} && rm -rf ${sqfs} && rm -rf ${sqfs}-work

#
# remount portage
#


umount /usr/portage || exit 1
cd /mnt/storage
ln -sf portage-${date}.sqfs portage-latest.sqfs
mount /usr/portage


#
# clean up 7+ days old archives
#
# todo: might need a check not to delete if there is only one file left, for whatever reason
#

find ${storage} -maxdepth 1 -name "portage*sqfs" -mtime +7 -delete

_________________
~amd64
shrink your /usr/portage with squashfs+aufs
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6277

PostPosted: Fri Feb 13, 2015 4:10 pm    Post subject: Reply with quote

[Advertisement]
If you are looking for a more general (and thus more powerful) approach, have a look at squashmount, available from the mv overlay via layman. There is also a dedicated thread in this forum.
[/Advertisement]
Back to top
View user's profile Send private message
yoshi314
l33t
l33t


Joined: 30 Dec 2004
Posts: 848
Location: PL

PostPosted: Fri Feb 13, 2015 4:16 pm    Post subject: Reply with quote

I used it, but it's a bit overengineered for my needs. so i opted for something dumb and simple.
_________________
~amd64
shrink your /usr/portage with squashfs+aufs
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum