Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
a great backup system for gentoo workstations
View unanswered posts
View posts from last 24 hours

Goto page 1, 2, 3, 4  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
razamatan
Apprentice
Apprentice


Joined: 28 Feb 2003
Posts: 160

PostPosted: Mon Jul 07, 2003 4:02 am    Post subject: a great backup system for gentoo workstations Reply with quote

ok... i started looking for simple backup solutions for just the bare essentials (mostly config and irreplaceable files) and i came to this script.

people have extended this script by adding functionality, and i've linked the msgs with their mods at the bottom of this first post....

key points:

    meant for personal workstations, not servers
    allows for scalability by using user definable lists
    extremely small and imho, has a very simple configuration
    basically a fancy and automatic tar based backup solution


i call it backup.cron and put it in /etc/cron.weekly... :wink: :

Code:

#!/bin/sh
# for FULL backups
# this backs up the important stuffs listed in ${lists} to ${BKPDIR}
# the lists *should* be in ${BKPDIR} and named <whatever>.lst
# the resulting backups will be <whatever>.<timestamp>.tgz
#
# notes:
# variables in CAPS are ok for you to set
# change the other variables if you know what you're doing
# you can get fancy in the lists... (think xargs *wink*)
# assumes /boot is usually unmounted, and has an /etc/fstab entry
# follow your security policy when setting perms on ${BKPDIR}
#
# written by razamatan
#
# DISCLAIMER: razamatan didn't write this if something goes wrong

BKPDIR=/home/.backup            # where the backups go
BOOT=sys                        # list that has /boot
NUMBKPS=3                       # how many backups to keep
                                                                               
if [ ! -d ${BKPDIR} ] ; then
        echo ${BKPDIR} is not a valid directory or is non-existant
fi
                                                                               
lists=${BKPDIR}/*.lst
ext=tgz
                                                                               
for list in `ls ${lists}`; do
        type=`basename ${list} .lst`
                                                                               
        if [ ${type} = ${BOOT} ] ; then mount /boot ; fi
        cat ${list} | xargs tar zlcf \
                ${BKPDIR}/${type}.`date +%Y-%m-%d-%H%M`.${ext} > /dev/null 2>&1
        if [ ${type} = ${BOOT} ] ; then umount /boot ; fi

        num=${NUMBKPS}
        for evict in `ls -t ${BKPDIR}/${type}.*.${ext}`; do
                if [ ${num} -le 0 ] ; then rm -f ${evict}
                else num=$((${num}-1)) ; fi
        done
done



you can download the latest version of the script from here: http://www-personal.umich.edu/~jyi/backup.cron

i actually only have one list, the /home/.backup/sys.lst:

Code:

/boot
/etc
/root
/usr/local/portage
--exclude=/usr/local/portage/distfiles
/usr/src/linux/.config
/var/lib
--exclude=/var/lib/texmf


EXTENSIONS & COMMENTS WELCOME!!!

here's some contributed extensions
iso images
iso images, tar splitting, pre & post-backup command processing, command line option to backup specific lists
specification of lists in a file (so you can have lists at various places


**EDIT** fixed numbkps issue
**EDIT** links to messages of extended versions
_________________
a razamatan doth speaketh,
"Never attribute to malice, that which can be adequately explained by stupidity"


Last edited by razamatan on Fri Feb 06, 2004 12:42 am; edited 3 times in total
Back to top
View user's profile Send private message
razamatan
Apprentice
Apprentice


Joined: 28 Feb 2003
Posts: 160

PostPosted: Mon Jul 07, 2003 4:22 am    Post subject: Reply with quote

oh... and if you should ever need to recover from said backups, make sure you use the --same-owner and --preserve flags w/ the tar command...

heh...
_________________
a razamatan doth speaketh,
"Never attribute to malice, that which can be adequately explained by stupidity"
Back to top
View user's profile Send private message
gatiba
Guru
Guru


Joined: 01 Sep 2002
Posts: 434

PostPosted: Mon Jul 07, 2003 10:23 am    Post subject: Reply with quote

Great script !
Would be nice if it could split the backup file in more parts if it's > 700mb for example (for burning ;) )
Back to top
View user's profile Send private message
vulcan_
n00b
n00b


Joined: 06 May 2003
Posts: 61
Location: Gent, Belgium

PostPosted: Mon Jul 07, 2003 11:12 am    Post subject: splitting a tar file Reply with quote

note that GNU tar has this option
Quote:

-L --tape-length N
change tapes after writing N*1024 bytes

but of course, this works to any file system that tar can output to
_________________
vulcan was a Roman myth - god of smiths
Back to top
View user's profile Send private message
razamatan
Apprentice
Apprentice


Joined: 28 Feb 2003
Posts: 160

PostPosted: Mon Jul 07, 2003 12:57 pm    Post subject: Re: splitting a tar file Reply with quote

yeah... you can do that on a per list basis. so say your <username>.lst would be huge cus of all those multimedia files, then you just need something like this to split stuff.

Code:

/home/raza
--exclude=/home/raza/*cache
--exclude=/home/raza/*Cache
-L 7000  # ~700 MB chunks


vulcan_ wrote:
note that GNU tar has this option
Quote:

-L --tape-length N
change tapes after writing N*1024 bytes

but of course, this works to any file system that tar can output to

_________________
a razamatan doth speaketh,
"Never attribute to malice, that which can be adequately explained by stupidity"
Back to top
View user's profile Send private message
gatiba
Guru
Guru


Joined: 01 Sep 2002
Posts: 434

PostPosted: Tue Jul 08, 2003 7:35 am    Post subject: Reply with quote

Mmmhhh i have problems.
This is my sys.lst:

Code:
/boot
/etc
/home/httpd
/windows/D/____progetti
/windows/D/____progetti/progetti_Gabriele/Alberone/vari
--exclude=/windows/D/____progetti/progetti_Gabriele/Alberone/vari/foto
/windows/D/____progetti/progetti_Gabriele/Felici_Editore/sito+ecommerce
--exclude=/windows/D/____progetti/progetti_Gabriele/Felici_Editore/sito+ecommerce/foto
-L 6500


When i start the script i got this error:

Code:
ls: /windows/D/backups/sys.*.tgz: No such file or directory


If i delete the "-L 6500" line it works ...
Back to top
View user's profile Send private message
sessionID
Apprentice
Apprentice


Joined: 11 Nov 2002
Posts: 266
Location: hungary

PostPosted: Tue Jul 08, 2003 2:38 pm    Post subject: Reply with quote

There should be an "umount /boot" line somewhere... :wink:
_________________
((( WeFunk )))
Back to top
View user's profile Send private message
razamatan
Apprentice
Apprentice


Joined: 28 Feb 2003
Posts: 160

PostPosted: Wed Jul 09, 2003 12:27 am    Post subject: Reply with quote

there is... look right below the crazy tar line.. :)

sessionID wrote:
There should be an "umount /boot" line somewhere... :wink:

_________________
a razamatan doth speaketh,
"Never attribute to malice, that which can be adequately explained by stupidity"
Back to top
View user's profile Send private message
razamatan
Apprentice
Apprentice


Joined: 28 Feb 2003
Posts: 160

PostPosted: Wed Jul 09, 2003 12:41 am    Post subject: Reply with quote

hmmm...

this script doesn't handle archive spanning.. :/ ... spanning requires interaction. i really don't do archive spanning so i didn't test for that functionality. i though the -L would do it, but you'd have to modify the script a bit.

to span archives, you can't compress it. this would require the script to handle compression on a list by list basis, and that would add complexity. i'll see what i can whip up :wink:.

gatiba wrote:
Mmmhhh i have problems.
This is my sys.lst:

Code:
/boot
/etc
/home/httpd
/windows/D/____progetti
/windows/D/____progetti/progetti_Gabriele/Alberone/vari
--exclude=/windows/D/____progetti/progetti_Gabriele/Alberone/vari/foto
/windows/D/____progetti/progetti_Gabriele/Felici_Editore/sito+ecommerce
--exclude=/windows/D/____progetti/progetti_Gabriele/Felici_Editore/sito+ecommerce/foto
-L 6500


When i start the script i got this error:

Code:
ls: /windows/D/backups/sys.*.tgz: No such file or directory


If i delete the "-L 6500" line it works ...

_________________
a razamatan doth speaketh,
"Never attribute to malice, that which can be adequately explained by stupidity"
Back to top
View user's profile Send private message
razamatan
Apprentice
Apprentice


Joined: 28 Feb 2003
Posts: 160

PostPosted: Wed Jul 09, 2003 12:58 am    Post subject: Reply with quote

gah... tar's -L option is weird...

this'll be a bit more research than i planned... in the meantime... treat this backup system for compressed archives... and DOESN'T SUPPORT SPANNING!!

sigh..

sorry spanning folks... i'll program my way outta this.
_________________
a razamatan doth speaketh,
"Never attribute to malice, that which can be adequately explained by stupidity"
Back to top
View user's profile Send private message
gatiba
Guru
Guru


Joined: 01 Sep 2002
Posts: 434

PostPosted: Wed Jul 09, 2003 6:47 am    Post subject: Reply with quote

:wink:
Back to top
View user's profile Send private message
sessionID
Apprentice
Apprentice


Joined: 11 Nov 2002
Posts: 266
Location: hungary

PostPosted: Wed Jul 09, 2003 2:54 pm    Post subject: Reply with quote

razamatan wrote:
there is... look right below the crazy tar line.. :)

sessionID wrote:
There should be an "umount /boot" line somewhere... :wink:


oops :oops: sorry
_________________
((( WeFunk )))
Back to top
View user's profile Send private message
S_aIN_t
Guru
Guru


Joined: 11 May 2002
Posts: 488
Location: Ottawa

PostPosted: Thu Jul 10, 2003 9:38 am    Post subject: Reply with quote

well done. i have been considering something similar.. althought i need a script that will backup linux and (gasp) windows.. but it would run from a linux server. i think there is something that samba has that would allow me to tar over an smb share.

more research is required for that. i'll be sure to post it here once i have something working.
_________________
"That which is overdesigned, too highly
specific, anticipates outcome; the anicipation of
outcome guatantees, if not failure, the
absence of grace."
- William Gibson, "All Tomorrow's Parties"
----
http://petro.tanreisoftware.com
Back to top
View user's profile Send private message
danhan79
n00b
n00b


Joined: 17 Jul 2003
Posts: 7

PostPosted: Thu Jul 17, 2003 4:22 am    Post subject: Reply with quote

odd, i get a bus error when i run your script. queer no?
Back to top
View user's profile Send private message
razamatan
Apprentice
Apprentice


Joined: 28 Feb 2003
Posts: 160

PostPosted: Thu Jul 17, 2003 7:23 am    Post subject: Reply with quote

lol... so you've managed to switch from rh to gen2? :) hehehe...

the moderators on these forums are pretty tuff dan... just im me later.

danhan79 wrote:
odd, i get a bus error when i run your script. queer no?

_________________
a razamatan doth speaketh,
"Never attribute to malice, that which can be adequately explained by stupidity"
Back to top
View user's profile Send private message
Elm0
Apprentice
Apprentice


Joined: 24 Nov 2002
Posts: 281
Location: UK

PostPosted: Sat Jul 19, 2003 12:25 pm    Post subject: Reply with quote

Great little script, just what I've been looking for :) Thanks!
Back to top
View user's profile Send private message
Elm0
Apprentice
Apprentice


Joined: 24 Nov 2002
Posts: 281
Location: UK

PostPosted: Sat Jul 19, 2003 12:47 pm    Post subject: Reply with quote

I don't know if the script is designed to this but...

The expected behavior would be to keep NUMBKPS of EACH list's backup in the backup directory. However, the current behavior means that if you have 5 lists and NUMBKPS = 3, the script will only backup 3 of the 5 and delete two existing files.

I would fix it myself but I'm not really a bash monkey, so maybe if you could take a look at this...
Back to top
View user's profile Send private message
razamatan
Apprentice
Apprentice


Joined: 28 Feb 2003
Posts: 160

PostPosted: Sat Jul 19, 2003 5:10 pm    Post subject: Reply with quote

ahh!!

stupid global/local variable issue.. thanks for spelunking! the code is fixed and reposted.

Elm0 wrote:
I don't know if the script is designed to this but...

The expected behavior would be to keep NUMBKPS of EACH list's backup in the backup directory. However, the current behavior means that if you have 5 lists and NUMBKPS = 3, the script will only backup 3 of the 5 and delete two existing files.

I would fix it myself but I'm not really a bash monkey, so maybe if you could take a look at this...

_________________
a razamatan doth speaketh,
"Never attribute to malice, that which can be adequately explained by stupidity"
Back to top
View user's profile Send private message
Zeitgeist
Apprentice
Apprentice


Joined: 13 Mar 2003
Posts: 165
Location: Ouagadougou, Burkina Faso

PostPosted: Sun Oct 05, 2003 5:32 am    Post subject: Reply with quote

Awsome script.

I have it running in /etc/cron.monthly, /etc/cron.daily, /etc/cron.weekly, modified so they look for different *.suffixs.

I have my schoolwork from /home/zeitgeist/school backedup hourly, my movies and music dirs are backed up weekly, and my root and etc dirs are backed up daily.

It's great :) Thanks
Back to top
View user's profile Send private message
razamatan
Apprentice
Apprentice


Joined: 28 Feb 2003
Posts: 160

PostPosted: Sun Oct 05, 2003 4:42 pm    Post subject: Reply with quote

Zeitgeist wrote:
Awsome script.

I have it running in /etc/cron.monthly, /etc/cron.daily, /etc/cron.weekly, modified so they look for different *.suffixs.

I have my schoolwork from /home/zeitgeist/school backedup hourly, my movies and music dirs are backed up weekly, and my root and etc dirs are backed up daily.

It's great :) Thanks


i feel all squishy and warm... thanks man. :D
_________________
a razamatan doth speaketh,
"Never attribute to malice, that which can be adequately explained by stupidity"
Back to top
View user's profile Send private message
Jeedo
Apprentice
Apprentice


Joined: 02 May 2003
Posts: 202
Location: Akureyri, Iceland

PostPosted: Sun Oct 05, 2003 8:25 pm    Post subject: Reply with quote

Your posted script did not work for me, howver this one did: http://www-personal.umich.edu/~jyi/backup.cron

I've been looking for something like this, i even was going to write one myself but didnt know how (suck at bash)..

Anyway i intend to to what Zeitgeist did and back up my files on a regularly..

However one thing i intend to modify is not using .gz for my important files but just .tar, and perhaps .bz2 for things tha i do intend to compress.

Perhaps that is a good idea for version 2.0 is an option to specify whether to use compression or not, and if so what kind of of compression to use..

Another idea is to add this to the comments:

Code:

# DISCLAIMER: razamatan didn't write this if something goes wrong
#
# The latest version can always be found at : http://www-personal.umich.edu/~jyi/backup.cron


Anyway thanks alot man.
Back to top
View user's profile Send private message
razamatan
Apprentice
Apprentice


Joined: 28 Feb 2003
Posts: 160

PostPosted: Mon Oct 06, 2003 2:45 am    Post subject: Reply with quote

Jeedo wrote:

However one thing i intend to modify is not using .gz for my important files but just .tar, and perhaps .bz2 for things tha i do intend to compress.

Perhaps that is a good idea for version 2.0 is an option to specify whether to use compression or not, and if so what kind of of compression to use..


you can remove the "x" option from tar to remove default compression... you can then put -x or -j in your lst files to do a case by case compression.... but then, you'd have to deal with naming the files with the proper extension for every respective file... so that's why i went with default gzipping... since tar and gzip is usually around, i don't mind compressing even my most important files.

Jeedo wrote:

Another idea is to add this to the comments:

Code:

# DISCLAIMER: razamatan didn't write this if something goes wrong
#
# The latest version can always be found at : http://www-personal.umich.edu/~jyi/backup.cron


Anyway thanks alot man.


no problem... i should put the url for this forum thread instead. more support.. heh.
_________________
a razamatan doth speaketh,
"Never attribute to malice, that which can be adequately explained by stupidity"
Back to top
View user's profile Send private message
nat
Apprentice
Apprentice


Joined: 04 Sep 2002
Posts: 204

PostPosted: Tue Oct 07, 2003 1:41 pm    Post subject: Reply with quote

Here is a small diff that mounts /boot if it finds '/boot' in the .lst file. If /boot already is mounted it doesnt touch it. In other words, you can put your /boot in any .lst file.

Code:
--- backup.cron.orig    2003-10-07 15:25:16.282737029 +0200
+++ backup.cron 2003-10-07 15:41:29.382176277 +0200
@@ -16,7 +16,6 @@
 # DISCLAIMER: razamatan didn't write this if something goes wrong
 
 BKPDIR=/home/.backup           # where the backups go
-BOOT=sys                       # list that has /boot
 NUMBKPS=3                      # how many backups to keep
 
 if [ ! -d ${BKPDIR} ] ; then
@@ -29,10 +28,20 @@
 for list in `ls ${lists}`; do
        type=`basename ${list} .lst`
 
-       if [ ${type} = ${BOOT} ] ; then mount /boot ; fi
+# if the .lst file contains '/boot' lets mount /boot before backing up
+# unless its already mounted.
+
+       if grep '/boot' ${list} > /dev/null && \
+           ! mount | grep /boot > /dev/null ; then
+           BOOT=1
+       else
+           BOOT=""
+       fi
+
+       [ ${BOOT} ] && mount /boot
        cat ${list} | xargs tar zlcf \
                ${BKPDIR}/${type}.`date +%Y-%m-%d-%H%M`.${ext} > /dev/null 2>&1
-       if [ ${type} = ${BOOT} ] ; then umount /boot ; fi
+       [ ${BOOT} ] && umount /boot
 
        num=${NUMBKPS}
        for evict in `ls -t ${BKPDIR}/${type}.*.${ext}`; do
Back to top
View user's profile Send private message
razamatan
Apprentice
Apprentice


Joined: 28 Feb 2003
Posts: 160

PostPosted: Tue Oct 07, 2003 7:55 pm    Post subject: Reply with quote

nat wrote:
Here is a small diff that mounts /boot if it finds '/boot' in the .lst file. If /boot already is mounted it doesnt touch it. In other words, you can put your /boot in any .lst file.


i thougth about this... but then i realized.... there's no need to have boot in more than 1 archive. so let's just specify it. even in my incarnation, you can stick /boot in any list.. just name that list in the env. variable.

your code actually has a slight problem. say that i have a situation where i have boot as a subdir of someplace that i wanted (/this/boot).. or even that i don't wanna include boot (--exclude=/boot)... it greps for any instance of /boot... thereby, it does unnecessary (u)mounts. just a slight problem.
_________________
a razamatan doth speaketh,
"Never attribute to malice, that which can be adequately explained by stupidity"
Back to top
View user's profile Send private message
Jeedo
Apprentice
Apprentice


Joined: 02 May 2003
Posts: 202
Location: Akureyri, Iceland

PostPosted: Wed Oct 08, 2003 11:04 pm    Post subject: Reply with quote

razamatan wrote:
Jeedo wrote:

However one thing i intend to modify is not using .gz for my important files but just .tar, and perhaps .bz2 for things tha i do intend to compress.

Perhaps that is a good idea for version 2.0 is an option to specify whether to use compression or not, and if so what kind of of compression to use..


you can remove the "x" option from tar to remove default compression... you can then put -x or -j in your lst files to do a case by case compression.... but then, you'd have to deal with naming the files with the proper extension for every respective file... so that's why i went with default gzipping... since tar and gzip is usually around, i don't mind compressing even my most important files.



I know, and that is what i did, but what i meant is that it would be nice to add it as a variable, like

compression=tbz2
or tgz og just tar and the script would then add xj, xz or just nothing respectively
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page 1, 2, 3, 4  Next
Page 1 of 4

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum