Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
a great backup system for gentoo workstations
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
Elm0
Apprentice
Apprentice


Joined: 24 Nov 2002
Posts: 281
Location: UK

PostPosted: Thu Oct 09, 2003 7:54 pm    Post subject: Reply with quote

I added this to the end of my script to allow automatic creation of an iso image of the backups to burn straight onto CD. I included the lst files as well because they are useful for determing whats in each final tar..

Code:

#!/bin/sh
# for FULL backups
# this backs up the important stuffs listed in ${lists} to ${BKPDIR}
# the lists *should* be in ${BKPDIR} and named <whatever>.lst
# the resulting backups will be <whatever>.<timestamp>.tgz
#
# notes:
# variables in CAPS are ok for you to set
# change the other variables if you know what you're doing
# you can get fancy in the lists... (think xargs *wink*)
# assumes /boot is usually unmounted, and has an /etc/fstab entry
# follow your security policy when setting perms on ${BKPDIR}
#
# written by razamatan
# ISO image modification by Elm0
# DISCLAIMER: razamatan didn't write this if something goes wrong

BKPDIR=/home/.backup            # where the backups go
BOOT=sys                        # list that has /boot
NUMBKPS=1                       # how many backups to keep
ISOUSER=jack                    # who you want to own the final ISO image

if [ ! -d ${BKPDIR} ] ; then
        echo ${BKPDIR} is not a valid directory or is non-existant
fi

lists=${BKPDIR}/*.lst
ext=tgz

for list in `ls ${lists}`; do
        type=`basename ${list} .lst`

        if [ ${type} = ${BOOT} ] ; then mount /boot ; fi
        cat ${list} | xargs tar zlcf \
                ${BKPDIR}/${type}.`date +%Y-%m-%d-%H%M`.${ext} > /dev/null 2>&1
        if [ ${type} = ${BOOT} ] ; then umount /boot ; fi

        num=${NUMBKPS}
        for evict in `ls -t ${BKPDIR}/${type}.*.${ext}`; do
                if [ ${num} -le 0 ] ; then rm -f ${evict}
                else num=$((${num}-1)) ; fi
        done
done

mkisofs -o /home/${ISOUSER}/backup.iso -iso-level 2 \
        -max-iso9660-filenames ${BKPDIR} > /dev/null 2>&1
chown ${ISOUSER} /home/${ISOUSER}/backup.iso
chgrp users /home/${ISOUSER}/backup.iso
chmod 0700 /home/${ISOUSER}/backup.iso


Just change the ISOUSER variable to reflect the name of the user who will own the iso file. It will also be moved to the root of their home directory.
Back to top
View user's profile Send private message
Starfox
Tux's lil' helper
Tux's lil' helper


Joined: 04 Sep 2002
Posts: 93

PostPosted: Fri Oct 10, 2003 4:31 pm    Post subject: Reply with quote

Jeedo wrote:

...
compression=tbz2
or tgz og just tar and the script would then add xj, xz or just nothing respectively


Using bzip2 doesn't make much sense for backupping as your backup size will decrease by ~10-15% while your backup time will increase by ~100-150%!!!

decide yourself which one is more valueable to you, i'll keep gzip!
bye fox
Back to top
View user's profile Send private message
Starfox
Tux's lil' helper
Tux's lil' helper


Joined: 04 Sep 2002
Posts: 93

PostPosted: Fri Oct 10, 2003 4:35 pm    Post subject: Reply with quote

Hi folks,

what dirs do i have to put in sys.lst to backup my system, so i will be able to reinstall it by just taking a stage-X-tarball, and running emerge world???

I used:
Quote:

/boot
/etc
/root
/usr/local/portage
--exclude=/usr/local/portage/distfiles
/usr/src/linux/.config
/var/lib
--exclude=/var/lib/texmf
/var/cache/edb/world


Do I need any more???
thanks, fox
Back to top
View user's profile Send private message
Jeedo
Apprentice
Apprentice


Joined: 02 May 2003
Posts: 202
Location: Akureyri, Iceland

PostPosted: Sun Oct 12, 2003 3:32 am    Post subject: Reply with quote

why are you backing up portage and libs?
Back to top
View user's profile Send private message
Starfox
Tux's lil' helper
Tux's lil' helper


Joined: 04 Sep 2002
Posts: 93

PostPosted: Sun Oct 12, 2003 11:42 am    Post subject: Reply with quote

hmm, /usr/locale/portage because of my personal changed ebuilds, and i took /var/lib because it was mentioned just some posts above, but i'm not really sure if i'll need it for a reinstall!?
maybe i'd just take /var/lib/mysql !?
Back to top
View user's profile Send private message
Starfox
Tux's lil' helper
Tux's lil' helper


Joined: 04 Sep 2002
Posts: 93

PostPosted: Thu Oct 16, 2003 7:10 am    Post subject: Reply with quote

hmm, i just reallized that
Code:
/lib/modules
is important for system backup , too. If you won't include it, you won't be able to start your old kernel from /boot corectly.
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Thu Oct 16, 2003 10:36 pm    Post subject: Reply with quote

Splitting into separate archives with tar -L requires all archives to be present to restore, correct? In that case, why not just create one big tarball and use 'split' to split it into 700MB chunks. You need them all to restore (cat them together and pipe into tar) so you don't lose anything (except you have to have them all available at the same time).
Back to top
View user's profile Send private message
cripwalk
n00b
n00b


Joined: 13 Jul 2003
Posts: 59
Location: Socal

PostPosted: Fri Oct 17, 2003 3:58 am    Post subject: Reply with quote

razamatan-
Thanks for the script. This is EXACTLY what I was looking for! It works perfect for me. Thanks again
Back to top
View user's profile Send private message
Starfox
Tux's lil' helper
Tux's lil' helper


Joined: 04 Sep 2002
Posts: 93

PostPosted: Fri Oct 17, 2003 6:18 am    Post subject: Reply with quote

meowsqueak wrote:
Splitting into separate archives with tar -L requires all archives to be present to restore, correct? In that case, why not just create one big tarball and use 'split' to split it into 700MB chunks. You need them all to restore (cat them together and pipe into tar) so you don't lose anything (except you have to have them all available at the same time).


This only works if you have enough free disk space, but anyway a nice idea!
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Fri Oct 17, 2003 9:39 am    Post subject: Reply with quote

Indeed - I implied that, but you are right to clarify. Cheers.
Back to top
View user's profile Send private message
razamatan
Apprentice
Apprentice


Joined: 28 Feb 2003
Posts: 160

PostPosted: Sat Oct 18, 2003 7:39 pm    Post subject: Reply with quote

yeah... split'ing into 700mb chunks and later cat'ing them together would do it. but i accomplish the same thing by having various lst files where each resultant tarball is smaller that 700mb... usually a lot smaller. so if i do decide to archive my backups to cdrom, i just burn the various tarballs individually until i hit 700mb.

i'm glad ppl are making their own solutions off of this. that's what open source is all about. :)
_________________
a razamatan doth speaketh,
"Never attribute to malice, that which can be adequately explained by stupidity"
Back to top
View user's profile Send private message
Souperman
Guru
Guru


Joined: 14 Jul 2003
Posts: 449
Location: Cape Town, South Africa

PostPosted: Sat Oct 18, 2003 8:17 pm    Post subject: Reply with quote

Starfox wrote:
maybe i'd just take /var/lib/mysql !?

It's safer to use the mysqldump program to create a plain text "dump" of your mysql database(s). See 'man mysqldump' for details.
_________________
moo
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Sun Oct 19, 2003 10:43 pm    Post subject: Reply with quote

This looks interesting and relevant:

http://freshmeat.net/branches/29090/

Quote:
About: hdup is used to back up a filesystem. Features include encryption of the archive (with mcrypt), compression of the archive (bzip/gzip/none), the ability to transfer the archive to a remote host or restoring from a remote host (with ssh), the ability to split up archives, and no obscure archive format (it is a normal compressed tar file).
Back to top
View user's profile Send private message
tomapd
n00b
n00b


Joined: 26 Aug 2003
Posts: 22
Location: Sendai

PostPosted: Mon Oct 20, 2003 2:40 am    Post subject: Reply with quote

Thaks for a very nice script!!
I posted just to say thank you. :D
Back to top
View user's profile Send private message
odegard
Guru
Guru


Joined: 08 Mar 2003
Posts: 324
Location: Trondheim, NO

PostPosted: Thu Oct 30, 2003 4:28 am    Post subject: Reply with quote

Great script!!

Since I'm a n00b, could someone tell me what important dirs to backup besides /home/user?

Thanks Gents
Back to top
View user's profile Send private message
Souperman
Guru
Guru


Joined: 14 Jul 2003
Posts: 449
Location: Cape Town, South Africa

PostPosted: Thu Oct 30, 2003 6:17 am    Post subject: Reply with quote

odegard wrote:
Great script!!

Since I'm a n00b, could someone tell me what important dirs to backup besides /home/user?

Thanks Gents

I have the following in my backup script:
Code:
BACKUP_ITEMS='/root /home /etc /var/log \
/var/spool/cron/crontabs /usr/local /var/cache/edb/world'
EXCLUDE_ITEMS='/root/tmp /home/httpd/htdocs/admin/backup/backup.tar.bz2 \
/home/httpd/htdocs/phpmyadmin /home/httpd/htdocs/admin/phpsysinfo'

Your requirements may differ. ;)
_________________
moo
Back to top
View user's profile Send private message
Starfox
Tux's lil' helper
Tux's lil' helper


Joined: 04 Sep 2002
Posts: 93

PostPosted: Thu Oct 30, 2003 12:13 pm    Post subject: Reply with quote

odegard wrote:
Great script!!

Since I'm a n00b, could someone tell me what important dirs to backup besides /home/user?


For a (without warranty) "full" system backup take this as sys.lst:
Quote:

/boot
/etc
/root
/usr/local/portage
--exclude=/usr/local/portage/distfiles
/usr/src/linux/.config
/var/lib
--exclude=/var/lib/texmf
/var/cache/edb/world
/lib/modules


To restore, do

  • untar gentoo-statge-3-tarball to /
  • untar sys.backup to /
  • emerge sync
  • emerge -eb world

after that you should have exactly the same system as before.
bye fox

PS: perhaps you should do some testing before to verfiy completness of the backup
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Thu Oct 30, 2003 8:14 pm    Post subject: Reply with quote

Another relevant backup script that creates uncompressed iso's for burning straight to CD, amongst others:

http://hoopajoo.net/quick_scripts.html

Specifically, take a look at the mkcdbackup script:

http://hoopajoo.net/static/quick_scripts/mkcdbackup
Back to top
View user's profile Send private message
odegard
Guru
Guru


Joined: 08 Mar 2003
Posts: 324
Location: Trondheim, NO

PostPosted: Fri Oct 31, 2003 9:39 am    Post subject: Reply with quote

Hm, I'm no wizard and I can't sort out this problem.

I made /home/.backup where I put my backup.lst with the following content
Code:

/boot
/etc
/root
/usr/local/portage
--exclude=/usr/local/portage/distfiles
/usr/src/linux/.config
/var/lib
--exclude=/var/lib/texmf
/var/cache/edb/world
/lib/modules
/home/odegard


I copied the backup.cron file to cron.daily, made it executable and ran it manually. It executed it and went to make a cup of tee. When I came back it had stopped, but only because I was out of diskspace!! The file was several gigabytes!! Obviously it got looped. I did some small changes to the file:

Code:

#!/bin/sh
# for FULL backups
# this backs up the important stuffs listed in ${lists} to ${BKPDIR}
# the lists *should* be in ${BKPDIR} and named <whatever>.lst
# the resulting backups will be <whatever>.<timestamp>.tgz
#
# notes:
# - assumes ${BKPDIR} is unmounted and has an /etc/fstab entry
# - assumes /boot is unmounted and has an /etc/fstab entry
# - variables in CAPS are ok for you to set... change the other
#   vars if you know what you're doing
# - you can get fancy in the lists... think xargs *wink*, but
#   you can't use thes spanning feature to break up an
#   archive to smaller pieces of arbitrary size
# - follow your security policy when setting perms on ${BKPDIR}
#
# written by razamatan
#
# DISCLAIMER: razamatan didn't write this if something goes wrong

BKPDIR=/home/.backup            # where the backups go
BOOT=sys                        # list that has /boot
NUMBKPS=4                       # how many backups to keep

if [ ! -d ${BKPDIR} ] ; then
        echo ${BKPDIR} is not a valid directory or does not exist
fi

#mount ${BKPDIR}                # i have my backup directory on a seperate partition

lists=${BKPDIR}/*.lst
ext=tgz

for list in `ls ${lists}`; do
        type=`basename ${list} .lst`

        if [ ${type} = ${BOOT} ] ; then mount /boot ; fi
        cat ${list} | xargs tar zlcf \
                ${BKPDIR}/${type}.`date +%Y-%m-%d-%H%M`.${ext} > /dev/null 2>&1
        if [ ${type} = ${BOOT} ] ; then umount /boot ; fi

        num=${NUMBKPS}
        for evict in `ls -t ${BKPDIR}/${type}.*.${ext}`; do
                if [ ${num} -le 0 ] ; then rm -f ${evict}
                else num=$((${num}-1)) ; fi
        done
done

#umount ${BKPDIR}       # and i like to keep it unmounted


Since my backup goes to /home/.backup I commented out the mount/umount. I must confess that I don't quite understand the BOOT=sys thing so I just left it.

So, can anyone spot what's wrong?

EDIT: The only changes to the script is the mentioned commenting of mount/umount and setting BKPDIR to the correct dir.
Back to top
View user's profile Send private message
hadfield
Retired Dev
Retired Dev


Joined: 18 Mar 2003
Posts: 308
Location: Vancouver, BC, Canada

PostPosted: Fri Oct 31, 2003 6:03 pm    Post subject: Reply with quote

I can think of only one thing that might be happening, you have several gigabytes of files in one or more than one of the folders you're trying to backup. Use 'du -sk' on every folder listed in all of your lists. When I tested this script the other day I ran the test on folders that I thought only had about 20 Megs in, but when it got to 40 I knew there was a problem. It turned out that my .ccache folder in my /root directory had 150MB's in it.

If this doesn't help, you could narrow down the problem by taking out some folders from your list files until you've found the exact folder you're having a problem with.

I would bet that the problem is with your home folder run du -sk /home/odegard to see how many KB's are in that folder.

Scott
Back to top
View user's profile Send private message
hadfield
Retired Dev
Retired Dev


Joined: 18 Mar 2003
Posts: 308
Location: Vancouver, BC, Canada

PostPosted: Fri Oct 31, 2003 6:30 pm    Post subject: Reply with quote

first off, razamatan thanks for the script! I've been planning on writing a script to do just this for the past couple months, but I've just been too lazy to write it. So when I found your script yesterday, I just decided I had to finally do it. I've taken your script and a lot of the suggestions here and made the script a little more powerfull. Of course it added a lot of complexity to your script, which is too bad, becaue the simplicity of it is what got me interested in it in the first place.

Anyway, I've made the following additions to the script:
1. I Cleaned up the code a little just to be a bit more to my liking. :-)
2. Modified the ISO creation section, posted by Elm0 to make ISO's no larger than MAXISOSIZE. So multiple ISO's will be created.
3. If a single tgz file is larger than MAXISOSIZE, it splits the tgz file, using the split command into a bunch of seperate tgz files and creates an ISO for each.
4. I added pre and post command options on a per list basis. This allows you to run commands both before and after a list is compressed. I needed this for my ccache problem in the root folder. Just add %pre and %post to the list if you want to perform any additional operations.
Format: %pre command cmd_options ;command2 ;command3 cmd3_options
5. You can call this script with each list you want it to backup. So,
backup.cron (with no options means backup all of the lists in the bkpdir)
backup.cron sys.lst music.lst (backup only sys.list and music.lst)

Here's my example sys.lst with %pre commands:
Code:
%pre ccache -C
/boot
/etc
/root
/usr/local/portage
--exclude=/usr/local/portage/distfiles
/usr/src/linux/.config
--exclude=/var/lib/texmf
/var/cache/edb/world
/lib/modules


Here's the new script, mostly untested with larger backups, so be warned! Once again, sorry for the added complexity, but if you step through it line by line it's not actually all that difficult to understand.
Code:
#!/bin/sh
BKPDIR=/pub/backup              # where the backups go
BOOT=sys                        # list that has /boot
NUMBKPS=3                       # how many backups to keep
ISODIR=${BKPDIR}/iso            # where the iso's go
ISOUSER=root                    # who you want to own the final ISO image
MAXISOSIZE=600000               # the max size of iso files created
MAKEISO=Y                       # Should the script make an ISO

lists=${BKPDIR}/*.lst
ext=tgz

checkdir() {
    if [ -e $1 ] && [ ! -d $1 ]; then
   echo "$1 is not a valid directory!"
   exit 0
    elif [ ! -e $1 ]; then
   mkdir -p $1
    fi
}

makeISO() {
    # Remove any existing iso
    if [ -e $2 ]; then
   rm $2
    fi
    mkisofs -o $2 -iso-level 2 -max-iso9660-filenames $1 > /dev/null 2>&1
    chown ${ISOUSER} $2
    chgrp users $2
    chmod 0700 $2
}

checkdir ${BKPDIR}/tmp
checkdir ${ISODIR}/tmp

origdir=`pwd`  # Save the current directory
cd $BKPDIR

if [ $# -eq 0 ]; then
    lists=`ls ${lists}`
else
    lists="$*"
fi

bkpfiles=""
for list in $lists; do
    type=`basename ${list} .lst`
    bkpname=${type}.$(date +%Y-%m-%d-%H%M).${ext}
   
    if [ ${type} = ${BOOT} ]; then
   mount /boot
    fi

    # Execute pre-tar commands
    grep "%pre" ${list} | awk -F "%pre" {'print $2'} &> /dev/null

    echo "Compressing $bkpname..."
    grep -vE "^%pre|^%post" ${list} | xargs tar zlcf $bkpname > /dev/null 2>&1

    if [ ${type} = ${BOOT} ]; then
   umount /boot
    fi

    # Execute post-tar commands
    grep "%post" ${list} | awk -F "%post" {'print $2'} &> /dev/null
   
    # Add the file to our list of current backup items
    bkpfiles="$bkpfiles $bkpname"
   
    num=${NUMBKPS}
    for evict in `ls -t ${BKPDIR}/${type}.*.${ext}`; do
   if [ ${num} -le 0 ]; then
       rm -f ${evict}
   else
       num=$((${num}-1))
   fi
    done
done

if [ "$MAKEISO" = "Y" ]; then
    # Create ISO's
    echo "Creating ISO's..."
    isosize=0
    isolist=""
    isocount=1
    for bkp in $bkpfiles; do
   bkpsize=$(du -sk $bkp | sed 's/^\([0-9]\+\).*/\1/')
        # Make sure the single tgz file isn't bigger than MAXISOSIZE
   if [ "$bkpsize" -gt "$MAXISOSIZE" ]; then
       split -C ${MAXISOSIZE}k $bkp ${BKPDIR}/tmp/${bkp}.
       partcount=1
       for broken in `ls -t ${BKPDIR}/tmp`; do
      mv ${BKPDIR}/tmp/${broken} ${ISODIR}/tmp
      makeISO ${ISODIR}/tmp \
          ${ISODIR}/backup.${isocount}-${partcount}.iso
      rm ${ISODIR}/tmp/${broken}
      let partcount++
       done
       let isocount++
       continue
   fi
   if [ "$(echo ${isosize}+${bkpsize} | bc)" -ge "$MAXISOSIZE" ]; then
       mv $isolist ${ISODIR}/tmp
       makeISO ${ISODIR}/tmp ${ISODIR}/backup.$isocount.iso
       mv ${ISODIR}/tmp/* ${BKPDIR}
       let isocount++
       isolist=""
       isosize=0
   else
       isolist="$isolist $bkp"
       isosize=$(echo "${isosize}+${bkpsize}" | bc)
   fi
    done

    # Create the last iso
    mv $isolist ${ISODIR}/tmp
    makeISO ${ISODIR}/tmp ${ISODIR}/backup.$isocount.iso
    mv ${ISODIR}/tmp/* ${BKPDIR}

fi
cd $origdir
Back to top
View user's profile Send private message
razamatan
Apprentice
Apprentice


Joined: 28 Feb 2003
Posts: 160

PostPosted: Tue Nov 04, 2003 5:46 am    Post subject: Reply with quote

it's cool man.. i intended for this script to be used however ppl needed to use it. i wanted to implement the "lower level" part of getting stuff into tarballs using lists. that was enough for my backup needs.

in terms of the iso creating stuff... it might be wiser to seperate that into a different script... and call that script from the backup script... iso creating isn't dependent on intermediary steps of this backup creation process... it can do its thing after the backups are generated...

generally... i write scripts on the idea of just doing a small unit of work.. it fits better with the idea of scripts as commands... commands aren't applications. the power of the command line lies in the idea that a command does one thing very well.
_________________
a razamatan doth speaketh,
"Never attribute to malice, that which can be adequately explained by stupidity"
Back to top
View user's profile Send private message
odegard
Guru
Guru


Joined: 08 Mar 2003
Posts: 324
Location: Trondheim, NO

PostPosted: Wed Nov 05, 2003 12:54 am    Post subject: Reply with quote

Hi, new problem.

I just reinstalled my system. I downloaded the script, commented out the mount commands since I don't back up to a mounted volume (yet), made it executable but when I try to execute it I get:
Code:

01:53:40 root /etc/cron.daily # ./backup.cron
ls: /home/.backup/backup.*.tgz: No such file or directory

the list, backup.lst, is in /home/.backup

What is wrong?
Back to top
View user's profile Send private message
hadfield
Retired Dev
Retired Dev


Joined: 18 Mar 2003
Posts: 308
Location: Vancouver, BC, Canada

PostPosted: Wed Nov 05, 2003 2:31 am    Post subject: Reply with quote

The line that is causing this problem is:
Code:
for evict in `ls -t ${BKPDIR}/${type}.*.${ext}`; do

It's looking for files to delete, but none exist yet. However, at least one file should exist, the file that just got backed up.

Are you sure that the backup file is being created? Is it being saved to the proper directory? First make sure that the tar command is running properly, I'm assuming this is where the error lies. Perhaps there's a typo in your .lst or something.
Back to top
View user's profile Send private message
odegard
Guru
Guru


Joined: 08 Mar 2003
Posts: 324
Location: Trondheim, NO

PostPosted: Wed Nov 05, 2003 5:14 am    Post subject: Reply with quote

I'm running the script for the first time so there are no other files besides the backup.lst file there yet.

My backup.lst is

Code:
/boot
/etc
/root
/usr/local/portage
--exclude=/usr/local/portage/distfiles
/usr/src/linux/.config
/var/lib
--exclude=/var/lib/texmf
/var/cache/edb/world
/lib/modules
/home/odegard
--exclude/home/odegard/download


I'm at loss as to why this happens...
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3, 4  Next
Page 2 of 4

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum