Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Automated backup of important system wide files
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
Spinner_Kontrol
n00b
n00b


Joined: 16 Jan 2005
Posts: 6

PostPosted: Sun Jan 16, 2005 10:40 am    Post subject: Automated backup of important system wide files Reply with quote

I just thought I'd share this with the gentoo group since I've been a user now for a few years (Gentoo that is) ;-)

This program was written as a little project to learn some perl, back when I wanted to learn at least the basics of this powerful scripting language, and also because some of the system files that I had would disappear after some emerges (fluxbox is mentioned in the file's comments)
Well...basically, the top half of the perl script has all this info, so I'll get right to it.

The archive.critical.files.pl file which I have as /etc/cron.scripts/archive.critical.files.pl :

Code:
#!/usr/bin/perl -w
#####
###
# archive.critical.files.pl
# Written by Scott Walker
# scott @at@ walker .dot. ath .dot. cx
#   This simple perl script is by no means a generic solution, it was written
#   for 2 reasons:
#          1) To refresh my memory (the bit I knew) of PERL.
#          2) To suit the existing needs of my system at the time.
#          3) Because I was pissed off that my fluxbox/[menu|keys] had been lost
#                 after I'd spent so long getting them just right.
#   ****(I know I said 2 reasons, but this was more of a
#             subconscious lingering rather than a reason).
#   This program was written to be called periodically by cron (daily for me)
#   to backup the configuration files that the caller has deemed to be important
#   enough to require periodic archival. Obviously, any files can be archived, but
#   the initial idea was for backing up config files (ie. samba, http, X11, WM's etc.).
#   This script will read from a text file (backup.files), extract all the file names,
#   create a compressed archive containing those files, and change the permissions
#   on the archive based on the values easily configurable at the start of this script.
#   The file "backup.files" may contain your standard characters, or a
#   "quoted string" for any files containing special characters. If a filename has a
#   '#' character in it, then it MUST be quoted. The same goes for most other
#   nonstandard filename characters for example (!, ;, @, <, >, etc.).
#   A comment will be considered anything after the quoted string up until the end
#   of the line, or in the case of non quoted filenames, anything starting with a
#   nonstandard character up until the end of the line.
#   *** Standard characters are taken to be:
#      ***      - (dash), . (dot), _ (underscore), * (asterisk), \ (backslash),
#      ***      / (forwardslash), and all upper/lower case & numbers.
#   *** Refer the the middle of the script for the line parsing acceptable chars.
#   The file "backup.files" can use any nonstandard characters to start a comment.
#   The '#' or ';' character is your best option for the start of a comment.
#   If a filename is quoted, it MUST be on a line by itself (or with a comment), but
#   regular filenames can exist in multiple on the same line. ie. this line:
#         "/etc/fancy filenames/file#2.tx!" /etc/hostname
#      will only archive the quoted 'fancy' file, (since " /etc/hostname" is assumed
#      to be a comment) whereas this line:
#         /etc/crontab /etc/fstab /etc/inittab /home/config.txt
#      will archive all 4 (four) listed files, since the line contains ONLY regular chars.
#   (Escaped sequences have not been accounted for in the parsing of the
#   "backup.files" file, so  writing a filename as
#       /important/files/filename\#2.txt
#   WILL *NOT* work - it MUST be quoted such as:
#       "/important/files/filename#2.txt" #optional comment, note: '#' not required
#   ).
#   Comments can be placed anywhere in the "backup.files" file, but are taken to be
#   everything from the nonstandard character to the end of the line. (Provided that
#   char does not lie within a quoted string.
#   Blank spaces at the start and end of a filename are stripped, so if there are spaces
#   at either end, then yes, that's right, quote it!
#   Finally, the archived file has its permissions charged to 400 to prevent any important
#   data getting into the wrong hands. To prevent this, the archive is only readable by
#   the owner. The permission string can be changed, but I strongly suggest it is left as
#   400 (r--/---/---), and the archive is chmod'ed to the new permissions only if/when required.
###
#####
###
#    Copyright (C) 2004 Scott Walker
#
#    This program is free software; you can redistribute it and/or modify
#    it under the terms of the GNU General Public License as published by
#    the Free Software Foundation; either version 2 of the License, or
#    (at your option) any later version.
#
#    This program is distributed in the hope that it will be useful,
#    but WITHOUT ANY WARRANTY; without even the implied warranty of
#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#    You should have received a copy of the GNU General Public License
#    along with this program; if not, write to the Free Software
#    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
###
#####
### Program starts here --->
# Get the date for creating a unique filename
# (unless some prat has postdated a filename).
chomp($DATE=`date +%Y%m%d-%H%M`);
# BACKUPLIST is a (possibly) commented file containing a list of all the files to be archived
$BACKUPLIST="/etc/cron.scripts/backup.files";
# BACKUP_DIR is the directory name where the archive will be placed
$BACKUP_DIR = "/archive/gentoo/backup";
# EXTENSION is the extension to be appended to the end of the archive - if you change this
# please change the switches handed to tar at the bottom of the script to reflect the filename.
$EXTENSION="tbz2";
$FILENAME="${BACKUP_DIR}/${DATE}.system.backup.${EXTENSION}";
#Some of the archived files more than likely contain sensitive information such as clear
#   text passwords for DynDNS, *ap-secrets, etc. so make the archive owner (root?) only access!!!
$FILE_PERMS="400";
open(IN, $BACKUPLIST)||die "can't open $BACKUPLIST\n";
chomp(@file_list = <IN>);
foreach $line (@file_list){
# Scan line from file. Whitespace is permitted at start of line.
# Anything encapsulated in "'s is taken as a literal string allowing # chars etc.
# If the (supposed) filename has quotes around it, then accept it verbatim
#   until we reach the next quote.
 if($line =~ /^s*(\".*\").*$/){
  $file=$1;
 }
 else{
# If there are no quotes, then treat it as a regular filename
#    Allow chars: - (dash), . (dot), _ (underscore), * (asterisk), \ (backslash),
#    / (forwardslash), and all upper/lower case & numbers.
  if($line =~ /^\s*([-._\*\\\/\w ]*)/){
   $file=$1;
  }
 }
# If we have made a match, then add it to the list of files for tar to archive.
 if ($file){
  $file=~s/\s*$//;$files_to_archive .= " $file";
 }
}
# If the file contained any valid filename strings, then hand them to tar, and
# change the permissions of the newly created archive.
if($files_to_archive){
 system "tar cjf $FILENAME" . $files_to_archive;
 system "chmod $FILE_PERMS $FILENAME";
}


Then there's the list of what you want to backup (absolute path/filename depends on what the variables in the previous file are - be warned that this file is obsolete and files like world and XFree86 prolly aren't the most accurate representations of where these files reside, or even if they're used anymore - ie. xorg.conf):

Code:
# Periodic auto update scripts
/etc/cron.scripts/archive.critical.files.pl
/etc/cron.scripts/backup.files
/etc/cron.scripts/slocate
/etc/cron.scripts/updateportage
/etc/crontab

# Filesystem specific files
/etc/fstab
/etc/hosts
/etc/inittab
/etc/make.conf
/etc/ntp.conf

# GUI config files
/etc/X11/XF86Config

# Fluxbox Windowing Manager config files
/usr/share/fluxbox/init
/usr/share/fluxbox/keys
/usr/share/fluxbox/menu

# Netscape plugin helper config file
/etc/pluggerrc

# Modem dialer config
/etc/wvdial.conf

# Dialup account user/pass combos
/etc/ppp/*-secrets

# DNS files
/etc/resolv.*

# HTTP config files
/etc/apache2/conf/apache2.conf
/etc/apache2/conf/commonapache2.conf

/etc/samba/smb.conf
/etc/squid/squid.conf

# FTP config files
/etc/vsftpd/banner
/etc/vsftpd/vsftpd.conf

# Gentoo world file (emerged packages)
/var/cache/edb/world


and finally the addition to /etc/crontab:
Code:
echo "30 0 * * *      root    /etc/cron.scripts/archive.critical.files.pl" >> /etc/crontab



Just as a side note, not really anything to do with backing up files, but more to do with keeping OS-freshness ;-), I have the emerge sync and emerge -f world scheduled to run 0630 daily:
The automation of keeping portage up to date, and grabbing the source files I might need in advance is done by the file /etc/cron.scripts/updateportage :
Code:
#! /bin/sh
if [ -x /usr/bin/emerge ]
then
        /usr/bin/emerge sync > /dev/null
        /usr/bin/emerge -uf world > /dev/null
fi


We also add this to our crontab to update in the wee hours of the morning (when I should be just starting to fall asleep - well, such was the case when I wrote these files back in my uni days)
Code:
echo "30 6 * * *      root    /etc/cron.scripts/updateportage" >> /etc/crontab


I have more than likely reinvented the wheel here, but it might save someone else from doing it, and at least I know exactly what my system is doing.

Hope I helped someone out ;-)
Back to top
View user's profile Send private message
toniocartonio
n00b
n00b


Joined: 04 Mar 2004
Posts: 38
Location: Malmö - Skåne

PostPosted: Thu Jan 20, 2005 2:55 pm    Post subject: Reply with quote

Hi Spinner_Kontrol,
and thanks for the great script. I am using it right now to do a backup of the important files on my server to another harddisk.

I only have one question: Does the script automatically delete the old backup files?
I am not a coder myself and i could not find the answer in the comments, but since my tarred backup files end up to be pretty big and the space dedicated to backup is limited, it would be useful to remove the old files to make space to the new ones.

Thanks again for your contribution :wink:
_________________
http://www.francescoinsvezia.net - la mia vita fra la gnokka (powered by gentoo linux!)
Back to top
View user's profile Send private message
Spinner_Kontrol
n00b
n00b


Joined: 16 Jan 2005
Posts: 6

PostPosted: Sat Mar 19, 2005 11:21 am    Post subject: Reply with quote

naah...doesn't remove the old files

I have about 400 days worth of files in my /archive/backup/gentoo directory (to be failr - they're all <80kB so no biggie there)
I never thought about deleteing the backups because there have been times where changes are a bad thing (which you don't realise for a while) and you need to go back in time to get a good version - ie 20050314-1230 archived a file which had been modified by etc-update (for example mplayer's input.conf) and then you find you need the old one that was working just before (lets say christmas),
I can just do a
Code:
tar xvjpf /archive/dir/to/file/20041223-0030.system.backup.tbz2 etc/important.file
and get it back to a working state without having to trawl thru the file to find what's causing the problem/make the changes which I may have forgotten by now (save's reading a heap of man pages again to figure it out)

I really do think that for backups etc. rm'ing files should really be left upto the sys admin tho - nothing like coming back at a later date to find that the program/script has so helpfully removed just the file you needed - no one to blame if it's you who did it tho ;-)

just keep in mind that you have this script in cron and perhaps on a weekly/monthly basis go and rm all the files that you don't need anymore - the naming convention should make this pretty easy since you can delete all of this years January and February backups with just a
Code:
rm -fr /path/to/archive/200501* ; rm -fr /path/to/archive/200502*

and if you have some 2004 files that you don't want either then:
Code:
rm -fr /path/to/archive/2004*

etc. - pretty easy

anyway - glad it's making your life easier - I'm sure glad I wrote it coz, like I said - I have about 400 days worth or archived files so it's stood the test of time for me so far
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum