Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
System updater, automagically keep your system up to date!
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
Axelay
n00b
n00b


Joined: 17 Feb 2003
Posts: 4
Location: Omaha

PostPosted: Mon Feb 17, 2003 12:19 pm    Post subject: System updater, automagically keep your system up to date! Reply with quote

Gentoo is a moving target, packages are updated at a rate that I found to be quite annoying (to keep up with)

Therefore, I wrote a script to do it all for me, and let me know when it did.

Here is a list of the things the script does:
1. emerge rsync
2. emerge -u world --deep
3. Keeps and rotates logs containing errors and updates
4. Deletes logs that don't have errors or updates
5. Mails the user of your choice (defined at the head of the script) when updates or errors are performed/encountered. (mail requires that you have working mail)
It was written for the bash shell, should work out fine for most people.

I have only just written it, so far it tests out fine, but there are bound to be problems.

Thanks to all of those in #gentoo who helped me with various oddities while I was writing this (though you probably didn't know this is what you were helping me with.)




Code:

#!/bin/bash
#This script should be run through cron.
#It will run as often as you like.
#It will keep your portage tree up-to-date.
#And update your system packages as needed.
#This will also mail you if anything is updated.
#
#First we set the user preferences
_admin_mail=
_log_dir=/var/log/gen-update/
_logfile=${_log_dir}updater.log
#
#We check to see if the logdir exists (if we have run before)
if [[ -d $_log_dir ]]
then
     _last_log=$(ls -tr ${_logfile}* | tail -1)
#If the user has deleted all of the logs, this var will be undef
#And the rotation will be reset
     _logfile_seq=$(cat $_last_log | grep LOGFILESEQUENCE= | \
     tail -1 | cut -d= -f2)
#Next we check to make sure that we have a valid sequence for rotation
#This is where the rotation is reset if the user nuked his logs
     if [[ ${#_logfile_seq} -ne 0 ]]
     then
          _logfile_seq=$(expr ${_logfile_seq} + 1 )
     else
          _logfile_seq=1
     fi
else
     mkdir $_log_dir
     _logfile_seq=1
fi
#
#Now we set the logfile for the current rotation
_logfile=${_logfile}${_logfile_seq}
#
#This is just formatting for the logfile
echo "The body of this file is taken from $_logfile" > $_logfile
echo "You may wish to view the file for more details" >> $_logfile
echo "This output only contains information preceeding the build" >> $_logfile
echo "It will not contain compile output, or any related errors" >> $_logfile
echo "" >> $_logfile
echo "DO NOT EDIT THIS FILE, IT WILL BREAK THE LOG ROTATION" >> $_logfile
echo "You may freely delete this file, but do not edit!!" >> $_logfile
echo "LOGFILESEQUENCE=${_logfile_seq}" >> $_logfile
echo "" >> $_logfile
echo "" >> $_logfile
echo "" >> $_logfile
echo "------------RSYNC_UPDATE_LOG--------------" >> $_logfile
#
#Now we rsync to make sure we are up-to-date
emerge rsync | grep -v file... >> $_logfile
#
#more formatting
echo "------------EMERGE_PRE_UPDATE_LOG-------------" >> $_logfile
#
#Now we check what we will be updating (if anything)
emerge -pu world --deep >> $_logfile 2>&1
#
#This will show us if there is anything to be updated, or any errors
_updated_query=$(cat $_logfile | sed -n '/EMERGE_PRE_UPDATE_LOG/,$p' | \
      egrep "(\[ebuild|\!\!\!)")
#
#Now lets check our variable to see if we have any work, or errors
if [[ ${#_updated_query} -lt 1 ]]
then
     rm $_logfile #There is nothing to do, so we don't need the log
     exit 0
else
     mail $_admin_mail -s 'Gentoo Updater is performing an update' < $_logfile
     echo "------------EMERGE_UPDATE_LOG-------------" >> $_logfile
     emerge -u world --deep >> $_logfile 2>&1
     _updated_query=$(cat $_logfile | sed -n '/EMERGE_UPDATE_LOG/,$p' | \
      egrep "(\!\!\!)")
     if [[  ${#_updated_query} -lt 1 ]]
     then
          mail $_admin_mail -s 'Gentoo Update is successful' < $_logfile
     else
          mail $_admin_mail -s 'Gentoo Update has failed' < $_logfile
     fi
fi
exit 0

_________________
Mmmmmm
Back to top
View user's profile Send private message
zhenlin
Veteran
Veteran


Joined: 09 Nov 2002
Posts: 1361

PostPosted: Mon Feb 17, 2003 2:51 pm    Post subject: Reply with quote

My system has such a script. Daily, it will mail me a report on what packages I need to upgrade and what use flags will affect them. Of course, I could also make it automagic, but that might break too many things.

I rsync every hour from /etc/cron.hourly. I don't remember when was the last time I typed emerge rsync...
Back to top
View user's profile Send private message
charlieg
Advocate
Advocate


Joined: 30 Jul 2002
Posts: 2149
Location: Manchester UK

PostPosted: Mon Feb 17, 2003 4:05 pm    Post subject: Reply with quote

zhenlin wrote:
My system has such a script. Daily, it will mail me a report on what packages I need to upgrade and what use flags will affect them. Of course, I could also make it automagic, but that might break too many things.


Perhaps you could enlighten us with this script. :) It'd be something to compare to Axelay's script, if nothing else.
_________________
Want Free games?
Free Gamer - open source games list & commentary

Open source web-enabled rich UI platform: Vexi
Back to top
View user's profile Send private message
PARENA
Guru
Guru


Joined: 06 Jan 2003
Posts: 349

PostPosted: Mon Feb 17, 2003 4:11 pm    Post subject: Reply with quote

charlieg wrote:
Perhaps you could enlighten us with this script. :) It'd be something to compare to Axelay's script, if nothing else.


I agree! :)
Back to top
View user's profile Send private message
Axelay
n00b
n00b


Joined: 17 Feb 2003
Posts: 4
Location: Omaha

PostPosted: Mon Feb 17, 2003 4:13 pm    Post subject: Reply with quote

Anything is preferable to "Yeah, I can do that too!"

;)
_________________
Mmmmmm
Back to top
View user's profile Send private message
razamatan
Apprentice
Apprentice


Joined: 28 Feb 2003
Posts: 160

PostPosted: Wed Jul 02, 2003 5:25 am    Post subject: Reply with quote

cron.hourly is in poor taste to the gentoo rsync servers... save bandwidth today!

zhenlin wrote:
My system has such a script. Daily, it will mail me a report on what packages I need to upgrade and what use flags will affect them. Of course, I could also make it automagic, but that might break too many things.

I rsync every hour from /etc/cron.hourly. I don't remember when was the last time I typed emerge rsync...

_________________
a razamatan doth speaketh,
"Never attribute to malice, that which can be adequately explained by stupidity"
Back to top
View user's profile Send private message
tank
Tux's lil' helper
Tux's lil' helper


Joined: 28 Aug 2002
Posts: 116
Location: Virginia

PostPosted: Wed Jul 02, 2003 11:56 am    Post subject: Reply with quote

zhenlin wrote:
My system has such a script. Daily, it will mail me a report on what packages I need to upgrade and what use flags will affect them. Of course, I could also make it automagic, but that might break too many things.

I rsync every hour from /etc/cron.hourly. I don't remember when was the last time I typed emerge rsync...


If you only get an email about updates once a day then why run the rsync hourly?
Back to top
View user's profile Send private message
puggy
Bodhisattva
Bodhisattva


Joined: 28 Feb 2003
Posts: 1992
Location: Oxford, UK

PostPosted: Wed Jul 02, 2003 1:08 pm    Post subject: Reply with quote

razamatan wrote:
cron.hourly is in poor taste to the gentoo rsync servers... save bandwidth today!

zhenlin wrote:
My system has such a script. Daily, it will mail me a report on what packages I need to upgrade and what use flags will affect them. Of course, I could also make it automagic, but that might break too many things.

I rsync every hour from /etc/cron.hourly. I don't remember when was the last time I typed emerge rsync...


This is a valid point. Why waste the limited and stretched resources of these rsync servers. There is really no need to run an rsync 24 times a day. I run one once every few days. I don't think you really need to update that often do you, especially because of tank's point.
tank wrote:
zhenlin wrote:
My system has such a script. Daily, it will mail me a report on what packages I need to upgrade and what use flags will affect them. Of course, I could also make it automagic, but that might break too many things.

I rsync every hour from /etc/cron.hourly. I don't remember when was the last time I typed emerge rsync...


If you only get an email about updates once a day then why run the rsync hourly?


Puggy
_________________
Where there's open source , there's a way.
Back to top
View user's profile Send private message
tgoodaire
Tux's lil' helper
Tux's lil' helper


Joined: 31 Jan 2003
Posts: 145
Location: Dartmouth, Nova Scotia, Canada

PostPosted: Mon Jul 07, 2003 2:57 pm    Post subject: My update script Reply with quote

All that I did is stick this in my cron.daily:

Code:

#!/bin/bash
emerge sync &&
emerge -up world | mail -s "Updates" root


I named it "checkforupdates", but you can call it anything. You'll need a local mta running (postfix, sendmail, qmail, exim...), and the mail command.
_________________
I bent my wookie.
Back to top
View user's profile Send private message
zhenlin
Veteran
Veteran


Joined: 09 Nov 2002
Posts: 1361

PostPosted: Tue Jul 08, 2003 7:43 am    Post subject: Reply with quote

/etc/cron.daily/update-world.cron

Code:

#! /bin/bash
 
export PORTDIR_OVERLAY="$(. /etc/make.conf; echo $PORTDIR_OVERLAY)"
 
(
  echo "Subject: Preliminary World Update";
  echo "To: root";
  echo "From: Cron Daemon <root@$(hostname)>";
  echo;
  emerge -vpul --nospinner --deep world;
) | fold -sw 160 | /usr/lib/sendmail -F"Cron Daemon" -f"root" root
 
if [ -f /etc/autoupdate ]
then
    DATE=`date '+%A, %B %e %Y'`
    emerge -u --deep world >& /var/log/world-update-${DATE}.log
fi


Of course, I have never used the batch update feature, since I don't want to have a emerge openoffice running in the background without my consent.

As for syncing, I now sync once every 4 hours when my computer is online, twice (5PM, 9PM) on weekdays and thrice on weekends (1PM, 5PM, 9PM) and sometimes even at 1AM or 9AM...

I find that this kind of syncing is neccessary when I need to do a GLSA update. No need to waste time syncing to get that one update. Also, this does not constitute as unneccessary strain on rsync servers - there are far, far more than just one. If anything, it is a strain on my Internet connection.

Note, my previous post was made in February.
Back to top
View user's profile Send private message
charlieg
Advocate
Advocate


Joined: 30 Jul 2002
Posts: 2149
Location: Manchester UK

PostPosted: Tue Jul 08, 2003 3:21 pm    Post subject: Reply with quote

zhenlin wrote:
As for syncing, I now sync once every 4 hours when my computer is online, twice (5PM, 9PM) on weekdays and thrice on weekends (1PM, 5PM, 9PM) and sometimes even at 1AM or 9AM...

I find that this kind of syncing is neccessary when I need to do a GLSA update. No need to waste time syncing to get that one update. Also, this does not constitute as unneccessary strain on rsync servers - there are far, far more than just one. If anything, it is a strain on my Internet connection.

If everybody were to sync every 4 hours, it would be an unecessary strain on the Gentoo servers. Fortunately the majority of people are less selfish on this count.

Why on earth do you need to sync every 4 hours? To get the GLSA updates? You say yourself you don't do the 'batch updates' simply because you want to verify each update. Surely then you are aware when an GLSA comes about and could simply sync then.

Also, I find it a little unbelievable that you need to stay that 'up to date' unless your machine is a production server and/or the target of black hats.
_________________
Want Free games?
Free Gamer - open source games list & commentary

Open source web-enabled rich UI platform: Vexi
Back to top
View user's profile Send private message
razamatan
Apprentice
Apprentice


Joined: 28 Feb 2003
Posts: 160

PostPosted: Wed Jul 09, 2003 3:27 am    Post subject: Reply with quote

+1 in favor of charlieg... anything more than 1 sync a day is obsessive compulsive :) ...

even when a glsa gets posted, a 1 day waiting period isn't *that* bad. however, if you must, when a glsa is posted and *if you need it* (you might not have the app in question installed), then go ahead and do a manual sync. i mean... it's not hard. emerge sync. 12 keystrokes. then do the appropriate emerge -u <theglsa'dpackage>. you probably won't do a world because this would be a production server or workstation.. right?? right?? :)

every 4 hours.... 6 times a day... omfg.

i wish there could be a way for people who abuse the rsync servers to temporarily become a mirror so they get flooded by other people who abuse the system.

:X :evil: :twisted:

charlieg wrote:
zhenlin wrote:
As for syncing, I now sync once every 4 hours when my computer is online, twice (5PM, 9PM) on weekdays and thrice on weekends (1PM, 5PM, 9PM) and sometimes even at 1AM or 9AM...

I find that this kind of syncing is neccessary when I need to do a GLSA update. No need to waste time syncing to get that one update. Also, this does not constitute as unneccessary strain on rsync servers - there are far, far more than just one. If anything, it is a strain on my Internet connection.

If everybody were to sync every 4 hours, it would be an unecessary strain on the Gentoo servers. Fortunately the majority of people are less selfish on this count.

Why on earth do you need to sync every 4 hours? To get the GLSA updates? You say yourself you don't do the 'batch updates' simply because you want to verify each update. Surely then you are aware when an GLSA comes about and could simply sync then.

Also, I find it a little unbelievable that you need to stay that 'up to date' unless your machine is a production server and/or the target of black hats.
[/i]
_________________
a razamatan doth speaketh,
"Never attribute to malice, that which can be adequately explained by stupidity"
Back to top
View user's profile Send private message
zhenlin
Veteran
Veteran


Joined: 09 Nov 2002
Posts: 1361

PostPosted: Wed Jul 09, 2003 7:23 am    Post subject: Reply with quote

Read carefully, I only sync 2 times a day on normal days. I am a little obsessive-compulsive with keeping my system up-to-date, true; but syncs are beginning to take up to 8 minutes for me.

I hardly see how accessing a server once a day is called 'unnecessary strain'

Blacklisting people to the DNS round robin is possible, but it would put unnecessary strain on the DNS server, remember the temporary switchover to TCP DNS packets?

My computer is not the target of blackhats, for now, but I know there are people who would very much like to crack into my computer... Thank goodness most of them are script kiddies, and for now, there aren't many Linux cracking scripts.
Back to top
View user's profile Send private message
ebrostig
Bodhisattva
Bodhisattva


Joined: 20 Jul 2002
Posts: 3152
Location: Orlando, Fl

PostPosted: Wed Jul 09, 2003 6:58 pm    Post subject: Reply with quote

This is a good idea as long as you don't edit /usr/portage/profiles/package.mask as I do :)

Then the script will actually downgrade several of my packages and I would be very unhappy :)

Erik
_________________
'Yes, Firefox is indeed greater than women. Can women block pops up for you? No. Can Firefox show you naked women? Yes.'
Back to top
View user's profile Send private message
Chris Finch
Tux's lil' helper
Tux's lil' helper


Joined: 10 Mar 2003
Posts: 106
Location: Darmstadt, Germany

PostPosted: Wed Jul 09, 2003 10:55 pm    Post subject: Reply with quote

Frankly, I think this whole idea of automated updates is completey ill advised. No comment on syncing several times a day, but even besides that: how often do you run etc-update? And revdep-resolve? env-update? See what I mean? It would be far more sensible to sync every couple of days, do an "emerge -up --deep", pick the packages you really want to update and make them work (which sometimes is necessary) with etc-update and beyond. Also, if, say, apache has a security problem that is fixed in a newer version, it won't be fixed on your machine until you restart apache. Believe me, automated updates in gentoo are pointless. Do your updates manually, in a controlled, responsible and sensible fashion.
Back to top
View user's profile Send private message
nacs
n00b
n00b


Joined: 21 Feb 2003
Posts: 41

PostPosted: Thu Jul 10, 2003 6:36 am    Post subject: Reply with quote

It really is selfish to sync more than once a day in my opinion. I'm all for automated updates but syncing multiple times a day is just plain unthoughtful.
Back to top
View user's profile Send private message
cschwede
n00b
n00b


Joined: 06 Sep 2002
Posts: 19
Location: Hamburg, Germany

PostPosted: Sat Aug 23, 2003 7:24 pm    Post subject: Reply with quote

I use this skript from /etc/cron.daily:

Code:
#!/bin/bash
/usr/bin/emerge sync > /dev/null
for i in ` /usr/bin/emerge -pu --deep world | grep ebuild | cut -f 2- -d "/" | cut -f 1 -d " " ` ;
do
        if [ -f /usr/portage/packages/All/$i.tbz2 ]
        then echo "Binary package $i existing, but it's not installed."
        else echo "Binary package $i doen't exist, emerging... "; /usr/bin/nice -n +19 /usr/bin/emerge -B =$i > /var/log/update.log
fi
done


It's simple, but it works really good for updating a whole bunch of different servers. It runs every night, syncs portage, checks for needed updates and tests if their is a binary already and if not it creats the binary package. So i just have to do a
Code:
emerge -K packagename
and my system is up2date.
Back to top
View user's profile Send private message
ib
n00b
n00b


Joined: 14 Mar 2003
Posts: 25

PostPosted: Fri Sep 12, 2003 3:27 pm    Post subject: great script Reply with quote

cschwede

Just wanted to say: Great script, best of both worlds.


ib
_________________
2nd Law of Thermodynamics: Chaos will Reign.
Back to top
View user's profile Send private message
rwallace
Tux's lil' helper
Tux's lil' helper


Joined: 22 May 2003
Posts: 107
Location: Phoenix, AZ US

PostPosted: Fri Sep 12, 2003 7:41 pm    Post subject: Reply with quote

I agree, cschwede seems to have the best solution so far. Right now all I have is 'emerge sync && emerge -UDp world' but I'll be replacing that as soon as I get home. It's one of those "Why didn't I think of that?" kinda things. Nice job.
Back to top
View user's profile Send private message
lars_msh
n00b
n00b


Joined: 25 Aug 2002
Posts: 50
Location: Airstrip One

PostPosted: Sat Sep 13, 2003 1:53 pm    Post subject: Reply with quote

I think even an automated daily emerge -up world is pointless...

I did consider it myself, but once the novelty wore off I probably wouldn't even look at it every day, so it would just be wasting bandwidth.

Instead I find a manual update every 2-7 days, depending on what I'm doing and when I think of it, does the trick fine.
Back to top
View user's profile Send private message
pr0teus
n00b
n00b


Joined: 07 Nov 2002
Posts: 18

PostPosted: Sun Sep 14, 2003 5:40 am    Post subject: Reply with quote

I use a script as well to do automatic syncs but god it only does it every 5th week I'm not that completely obsessed over updates considering that 99% of the packages I have installed don't get updates but maybe every 8 to 16 weeks
_________________
:: pr0teus ::
Back to top
View user's profile Send private message
Hysteric
n00b
n00b


Joined: 31 Jan 2003
Posts: 46
Location: Québec

PostPosted: Thu Sep 25, 2003 3:35 am    Post subject: Reply with quote

Maybe an rsync in cron.daily can be good if you want to stay really up to date, but more than that is just being stupid. Portage sometimes want to update packages you don't want, downgrade some, etc. I prefer to be in control and verify what is merged everytime I update... (sometimes I am a little bit obsessed... like this week... one update a day!)
Back to top
View user's profile Send private message
cschwede
n00b
n00b


Joined: 06 Sep 2002
Posts: 19
Location: Hamburg, Germany

PostPosted: Thu Sep 25, 2003 1:18 pm    Post subject: Reply with quote

Hysteric wrote:
Maybe an rsync in cron.daily can be good if you want to stay really up to date, but more than that is just being stupid.

Yes and no. Sure, it doesnt' make sense to merge all available updated packages - especially on servers- , but take for example openssh this/last week: In the morning I now already that there is an updated binary package on my server (provided by the little script some posts earlier), so i just have to do an
Code:
emerke -uK openssh
(after reading the apropriate security mailing list). And everything is fine again :)
Of course you could run the script only on weekends, month 1st. etc., but that depends on your needs.
Hysteric wrote:
I prefer to be in control and verify what is merged

You are still in control of the updates ;-)
Back to top
View user's profile Send private message
Squinky86
Retired Dev
Retired Dev


Joined: 25 Mar 2003
Posts: 309
Location: Alabama, USA

PostPosted: Thu Sep 25, 2003 10:28 pm    Post subject: Reply with quote

Just to throw in my thoughts:
I love these kinds of scripts that sync and send an email to me whenever there are updates. I have my main desktop sync every 2 days and send me an email about what needs to be updated, and my minimalistic server syncs every 7 days and sends me a nice little email :). My server has ssh on it, so I can update it immediately or shut it down if some bad worm is going around.
As for GLSA, whenever I see an announcement, it's faster to go to the online cvs repository at http://www.gentoo.org/cgi-bin/viewcvs.cgi/ and manually download the ebuild and patches, digest the file, and begin the install. It normally takes no more than a minute or two, and it keeps me from straining the rsync servers. However, it's important to note that in the year I've been running Gentoo, I can recall very few advisories that affected me, and it's mostly because I keep most of my ports blocked.

Moral: syncing more than once a day is unnecessary. There are faster, more efficient ways of keeping an up-to-date system. I can't be hard on anyone though- for more than 3 weeks, I was one of the 3-times-a-day syncers, but I have repented of my error, and hopefully the Gentoo community has accepted back its prodigal brother :).
_________________
Me
Back to top
View user's profile Send private message
cbreaker
Tux's lil' helper
Tux's lil' helper


Joined: 05 Feb 2003
Posts: 105
Location: Pawtucket, RI

PostPosted: Sat Sep 27, 2003 2:36 am    Post subject: Reply with quote

Good thing rsync tries to keep the bandwidth usage at a minimum, although I'd imagine that the CPU and disk load on these boxes can get quite high too.

I think it becomes a bad habit; always having the latest and greatest software installed. Get over it.. play chess.. fly a kite.. learn to do something with your linux box other then emerge.

I only sync when I need to install a few new packages. I see absolutely no need to sync at any other time.

The rsync servers are there as a free service for us to use. Don't be so selfish..

I have four Gentoo boxes at home, and I use one as an rsync server for the rest of them. I only have to do a sync once in awhile, and the other machine sync to that. It's faster and it doesn't waste bandwidth.

For emerging world nightly, well, I think it's a terrible idea, especially if you're on unstable. If you actually use your machine for something other then compilations, you wouldn't want to break something just because you're compulsive about having everything new.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum