Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Updating in small stages
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
wjb
Guru
Guru


Joined: 10 Jul 2005
Posts: 400
Location: Fife, Scotland

PostPosted: Mon Jun 29, 2009 10:27 pm    Post subject: Updating in small stages Reply with quote

After one gcc upgrade, I didn't do a rebuild of everything but I checked and saw some very very old dates in /usr/bin. Ended up writing this script to rebuild the oldest 10 packages at a time - it does the biz so nicely, that, well here it is ...
Code:
#!/usr/bin/python

# This program checks whether the files in /usr/bin are old (for a
# given value of old), and generates an emerge command to build the
# oldest ones. By default it finds the oldest 10 apps and runs the emerge
# with --ask.
#
# --help for optiions
#

from optparse import OptionParser
from glob import glob
import os.path
import os
import time
import re
from subprocess import Popen, PIPE

# The maximum number of packages to identify (it takes time to check
# each one)
HOW_MANY_PACKAGES = 10

# A list of absolute filenames to be ignored.
ignorefiles = [r'/usr/bin/coral2c',
               r'/usr/bin/pango-querymodules32',
               r'/usr/bin/gtk-query-immodules-2.0-32',
               r'/usr/bin/gdk-pixbuf-query-loaders32']

def sortfilesbytime(a,b):
    """ Used to sort file list in order of mtime """
    if a[1]<b[1]:
        return 1
    elif a[1]>b[1]:
        return -1
    return 0


def gettime(t):
    """ Return formatted time string for given time """
    return time.strftime("%Y-%m-%d %H:%M",time.gmtime(t))


def getpackage(filename):
    """ Return the name of the package owning the given file.
        Note that equery only returns the package name when run
        like this - if the same command is run from the command
        line, you also get the filename in brackets.
    """
    wot = Popen("equery -q belongs -e "+filename,shell=True,stdin=PIPE,stdout=PIPE,close_fds=True)
    packagenames = wot.stdout.readlines()
    result = ''
    for p in packagenames:
        if result<>'':
            result += ', '
        pname = p.strip()
        result += pname
    return result


def findexe(filedir):
    """ Return a set containing the executable filenames in the given
        directory. This runs the "file" command on the whole directory,
        and checks the resulting  mime-type for each file found. The
        ones that are "application/x-executable" are added to the result.

        Each line returned by "file" has the format:
          <filename> <colon> <whitespace> <mime-type> [<crud>]

        Reduced to using "file" since I could not find an off-the-shelf
        method for this in python. Not happy with the crud that "file"
        adds to the end of the line; no sign of a --no-crud option.
    """
    exe = set()
    exe_re = re.compile(r"([^:]+):\s*application/x-executable",re.IGNORECASE)
    wot = Popen("file -i "+os.path.join(filedir,'*'),shell=True,stdin=PIPE,stdout=PIPE,close_fds=True)
    for filedefs in wot.stdout.readlines():
        look = exe_re.match(filedefs)
        if look:
            exe.add(os.path.join(filedir,look.group(1)))
    return exe

def findlibs(filedir):
    """ Return a set containing the filenames in the given
        directory. This runs the "file" command on the whole directory,
        and checks the resulting  mime-type for each file found. The
        ones that are "application/x-sharedlib|object|archive" are added
        to the result.

        Each line returned by "file" has the format:
          <filename> <colon> <whitespace> <mime-type> [<crud>]
    """
    lib = set()
    lib_re = re.compile(r"([^:]+):\s*application/x-(sharedlib|object|archive)",re.IGNORECASE)
    wot = Popen("file -i "+os.path.join(filedir,'*'),shell=True,stdin=PIPE,stdout=PIPE,close_fds=True)
    for filedefs in wot.stdout.readlines():
        look = lib_re.match(filedefs)
        if look:
            libname = os.path.join(filedir,look.group(1)).strip()
            lib.add(libname)
    return lib

# Parse the command line options

parser = OptionParser()

parser.add_option('-c','--count',
                  type="int",
                  dest="count",
                  default=HOW_MANY_PACKAGES,
                  help="the number of packages to find and build, default %d." % HOW_MANY_PACKAGES)

parser.add_option('-p','--pretend',
                  action="store_false",
                  dest="ask",
                  default=True,
                  help="whether to skip the emerge")

(options,args) = parser.parse_args()


# Turn the list ignored files into a set.
ignorelist = set(ignorefiles)

# Make a list of all the directories containing executable files
pathlist = os.environ['PATH'].split(':')

# Reset the list of executable files
filelist = []

allfiles = set()
for binpath in pathlist:
    # Scan each directory in turn for a list of executable files
    allexes = findexe(binpath)
    allfiles.update(allexes)

# There is no point going any deeper - there are a lot of subdirectories
# but these are almost all for packages that will be found in the other
# directories.
libspath = '/usr/lib'
if os.path.islink(libspath):
    # Its a link - amd64? - try and find where its actually pointing
    libsloc = os.readlink(libspath)
    if os.path.isabs(libsloc):
        libspath = libsloc
    else:
        libspath = os.path.join(os.path.dirname(libspath),libsloc)
alllibs = findlibs(libspath)
allfiles.update(alllibs)

# The list only has executable files in it, so there is no
# need to check for directories or links.
for filename in allfiles:
    if filename not in ignorelist:
        details = os.stat(filename)
        lastmod = details.st_mtime
        filelist.append((filename,lastmod))

# Display the file list in time order, newest first
filelist.sort(sortfilesbytime)
for (filename,lastmod) in filelist:
    print gettime(lastmod),filename

# Search the list of files, oldest first, looking
# for the required number of unique packages. This
# can take a while, so display progress onscreen.
print "\nLooking for oldest packages:\n"
filelist.reverse()
sofar = set()
lasttime = ''
for (filename,lastmod) in filelist:
    thistime = gettime(lastmod)
    if thistime <> lasttime:
        # If the file has the same time as the last one, it is
        # almost certainly the same package, so don't waste
        # time checking - it'll get caught on the next run
        # if it is a different package.
        pkgname = getpackage(filename)
        if pkgname:
            print "  ",pkgname," ("+filename+")"
            sofar.add(pkgname)
            if len(sofar)>=options.count:
                break
        lasttime = thistime
   
if len(sofar)>0:

    # Build an emerge command line if there are any
    # packages selected.
    suggested = 'emerge -1'
    if options.ask:
        suggested += ' --ask'
    for pkgname in sofar:
        suggested += ' =' + pkgname

    # Write the command to the display so it can be run
    # manually using copy/paste.
    print
    print suggested

    # Run the emerge command if it has been selected
    if options.ask:
        args = suggested.split(' ')
        os.execvp('emerge',args)



EDIT1: removed a stray blank line at the start

EDIT2: (finally) sorted the deprecated warnings, and now also checks /usr/lib (which I somehow forgot about before, and is where the biggest cobwebs gather - on stable, anyway)


Last edited by wjb on Sat May 29, 2010 10:49 pm; edited 2 times in total
Back to top
View user's profile Send private message
rtomek
Apprentice
Apprentice


Joined: 05 Jan 2007
Posts: 210
Location: Chicago

PostPosted: Tue Jun 30, 2009 2:27 am    Post subject: Reply with quote

I get a depracated module, but it still works great! Would also prefer to add the --verbose switch to emerge, but I just threw that in there myself. It does it's job and I'm happy :D
Back to top
View user's profile Send private message
cyrillic
Watchman
Watchman


Joined: 19 Feb 2003
Posts: 7313
Location: Groton, Massachusetts USA

PostPosted: Tue Jun 30, 2009 12:41 pm    Post subject: Reply with quote

This is a pretty cool idea -- updating packages based on how much dust has collected on them. :)
Back to top
View user's profile Send private message
MaximeG
l33t
l33t


Joined: 15 Apr 2008
Posts: 722
Location: Belgium

PostPosted: Tue Jun 30, 2009 12:43 pm    Post subject: Reply with quote

Hi,

I may be dumb but ... why not keep the system up to date ?

Regards,
Maxime
_________________
Future is wide open.
Back to top
View user's profile Send private message
Hupf
Tux's lil' helper
Tux's lil' helper


Joined: 11 Sep 2005
Posts: 110
Location: Germany

PostPosted: Tue Jun 30, 2009 5:13 pm    Post subject: Reply with quote

I think this is about re-building packages that have not been updated in portage *themselves* with the new gcc version. So emerge --update --deep won't help here and I believe the OP does not want to do an emerge --emptytree world.
Back to top
View user's profile Send private message
wjb
Guru
Guru


Joined: 10 Jul 2005
Posts: 400
Location: Fife, Scotland

PostPosted: Tue Jun 30, 2009 6:27 pm    Post subject: Reply with quote

Yes, it takes about 24-28 hours to do a full rebuild on my main PC which can be inconvenient, so this was mostly about doing it in small bites.

Gentoo's been on the PC since October 2004. I settled on "--update --tree --newuse --deep" fairly early on and don't think I've used --emptytree.
Back to top
View user's profile Send private message
MaximeG
l33t
l33t


Joined: 15 Apr 2008
Posts: 722
Location: Belgium

PostPosted: Wed Jul 01, 2009 8:31 am    Post subject: Reply with quote

Oh,

OK, makes sense. I completely missed that GCC part.
/learns to read for next time.

Regards,
Maxime
_________________
Future is wide open.
Back to top
View user's profile Send private message
ronmon
Veteran
Veteran


Joined: 15 Apr 2002
Posts: 1043
Location: Key West, FL

PostPosted: Wed Jul 01, 2009 11:28 am    Post subject: Reply with quote

Nice one, wjb. That's a handy little tool.

One thing I noticed is that it works just like revdep-rebuild, in that it specifies exact package versions. Just for kicks, I ran emerge without the versions and have found lots of stuff that needed updates. For some reason emerge misses lots of updates in spite of regularly running "emerge -uDNtva world".

Is there any way to get results like:
Code:

dev-libs/foo-lib

instead of
Code:

=dev-libs/foo-lib-1.0.0

_________________
Ask Questions the Smart Way - by ESR
Back to top
View user's profile Send private message
wjb
Guru
Guru


Joined: 10 Jul 2005
Posts: 400
Location: Fife, Scotland

PostPosted: Wed Jul 01, 2009 11:35 pm    Post subject: Reply with quote

It is do-able but there are probably better ways - are the packages you're seeing actually used by anything (i.e. are they just junk that would be removed by a --depclean?)
Back to top
View user's profile Send private message
rmh3093
Advocate
Advocate


Joined: 06 Aug 2003
Posts: 2138
Location: Albany, NY

PostPosted: Wed Jul 01, 2009 11:46 pm    Post subject: Reply with quote

why would you waste time rebuilding apps... if you run ~arch just about ever single app you have will be updated within 1 months time... if your box is so slow that you cant rebuild your whole system in <48hours dont even bother, just run "emerge --sync; emerge -u world" with cron at night, or just update shit when you NEED a newer version, you arent going to notice a difference regardless
_________________
Do not meddle in the affairs of wizards, for they are subtle and quick to anger.
Back to top
View user's profile Send private message
ronmon
Veteran
Veteran


Joined: 15 Apr 2002
Posts: 1043
Location: Key West, FL

PostPosted: Thu Jul 02, 2009 9:27 am    Post subject: Reply with quote

They are mainly lib packages with some obscure apps, but a few were pretty main stream stuff. The apps that I no longer use, I got rid of, and I tried the same with some libraries that I thought were not needed. However, most of the libs got pulled back in as dependencies with an update.

No big deal. Thanks again for the script.
_________________
Ask Questions the Smart Way - by ESR
Back to top
View user's profile Send private message
MaximeG
l33t
l33t


Joined: 15 Apr 2008
Posts: 722
Location: Belgium

PostPosted: Thu Jul 02, 2009 10:15 am    Post subject: Reply with quote

rmh3093 wrote:
why would you waste time rebuilding apps... if you run ~arch just about ever single app you have will be updated within 1 months time... if your box is so slow that you cant rebuild your whole system in <48hours dont even bother, just run "emerge --sync; emerge -u world" with cron at night, or just update shit when you NEED a newer version, you arent going to notice a difference regardless


Same mistake as me : it's not to update the system, it to recompile the system.

Regards,
Maxime
_________________
Future is wide open.
Back to top
View user's profile Send private message
rmh3093
Advocate
Advocate


Joined: 06 Aug 2003
Posts: 2138
Location: Albany, NY

PostPosted: Thu Jul 02, 2009 2:19 pm    Post subject: Reply with quote

MaximeG wrote:
rmh3093 wrote:
why would you waste time rebuilding apps... if you run ~arch just about ever single app you have will be updated within 1 months time... if your box is so slow that you cant rebuild your whole system in <48hours dont even bother, just run "emerge --sync; emerge -u world" with cron at night, or just update shit when you NEED a newer version, you arent going to notice a difference regardless


Same mistake as me : it's not to update the system, it to recompile the system.

Regards,
Maxime


Yes I realize that, im saying its a waste of time!
_________________
Do not meddle in the affairs of wizards, for they are subtle and quick to anger.
Back to top
View user's profile Send private message
ronmon
Veteran
Veteran


Joined: 15 Apr 2002
Posts: 1043
Location: Key West, FL

PostPosted: Thu Jul 02, 2009 3:40 pm    Post subject: Reply with quote

The GCC Upgrade Gude requires rebuilding of system and world with the emptytree setting. This script allows you to do it in bite sized chunks. Do you understand now, or are you just trolling?
_________________
Ask Questions the Smart Way - by ESR
Back to top
View user's profile Send private message
loftwyr
l33t
l33t


Joined: 29 Dec 2004
Posts: 970
Location: 43°38'23.62"N 79°27'8.60"W

PostPosted: Thu Jul 02, 2009 4:42 pm    Post subject: Reply with quote

That upgrade guide says that only for the 3.3 to 3.4 versions where the the libs changed version from libc.so.5 to .6
_________________
My emerge --info
Have you run revdep-rebuild lately? It's in gentoolkit and it's worth a shot if things don't work well.
Celebrating 5 years of Gentoo-ing.
Back to top
View user's profile Send private message
NotQuiteSane
Guru
Guru


Joined: 30 Jan 2005
Posts: 481
Location: Klamath Falls, Jefferson, USA, North America, Midgarth

PostPosted: Tue Jul 07, 2009 10:25 am    Post subject: Reply with quote

this is what I see:

Code:
ifruit nqs # ./update.old
./update.old: line 10: from: command not found
./update.old: line 11: from: command not found
./update.old: line 12: import: command not found
./update.old: line 13: import: command not found
./update.old: line 14: import: command not found
./update.old: line 15: import: command not found
./update.old: line 19: HOW_MANY_PACKAGES: command not found
./update.old: line 22: ignorefiles: command not found
./update.old: line 25: syntax error near unexpected token `('
./update.old: line 25: `def sortfilesbytime(a,b):'
ifruit nqs #


nqs
_________________
These opinions are mine, mine I say! Piss off and get your own.

As I see it -- An irregular blog, Improved with new location

To delete French language packs from system use 'sudo rm -fr /'
Back to top
View user's profile Send private message
ronmon
Veteran
Veteran


Joined: 15 Apr 2002
Posts: 1043
Location: Key West, FL

PostPosted: Tue Jul 07, 2009 5:17 pm    Post subject: Reply with quote

I would guess that you missed picking up the final ')' , close parenthesis.

You might want to try copying it again.
_________________
Ask Questions the Smart Way - by ESR
Back to top
View user's profile Send private message
timeBandit
Bodhisattva
Bodhisattva


Joined: 31 Dec 2004
Posts: 2674
Location: here, there or in transit

PostPosted: Tue Jul 07, 2009 5:33 pm    Post subject: Re: Updating in small stages Reply with quote

wjb wrote:
After the last gcc upgrade, I didn't do a rebuild of everything but last week I looked and saw some very very old dates in /usr/bin. Ended up writing this script to rebuild the oldest 10 apps at a time ...
Clever. It's interesting to see the many different ways people wrap their heads around Portage and system maintenance.

PS: Love the avatar, wjb, despite my baseless preference for odd parity. :D
_________________
Plants are pithy, brooks tend to babble--I'm content to lie between them.
Super-short f.g.o checklist: Search first, strip comments, mark solved, help others.
Back to top
View user's profile Send private message
wjb
Guru
Guru


Joined: 10 Jul 2005
Posts: 400
Location: Fife, Scotland

PostPosted: Tue Jul 07, 2009 7:11 pm    Post subject: Reply with quote

NotQuiteSane: I think you have a couple of blank lines at the start of the file, the #!/usr/bin/python should be the first line in the file.

timeBandit: the parity was easy, I couldn't remember which side the sprockets were on.
Back to top
View user's profile Send private message
szczerb
Veteran
Veteran


Joined: 24 Feb 2007
Posts: 1700
Location: Poland => Lodz

PostPosted: Tue Jul 07, 2009 7:24 pm    Post subject: Reply with quote

What's the use of it? (other then completing an interesting idea?) Why not just let it all go and when you need your system CTRL+C and then --resume?
Back to top
View user's profile Send private message
NotQuiteSane
Guru
Guru


Joined: 30 Jan 2005
Posts: 481
Location: Klamath Falls, Jefferson, USA, North America, Midgarth

PostPosted: Wed Jul 08, 2009 2:26 pm    Post subject: Reply with quote

wjb wrote:
NotQuiteSane: I think you have a couple of blank lines at the start of the file, the #!/usr/bin/python should be the first line in the file.


Yep, that was it

NQS
_________________
These opinions are mine, mine I say! Piss off and get your own.

As I see it -- An irregular blog, Improved with new location

To delete French language packs from system use 'sudo rm -fr /'
Back to top
View user's profile Send private message
Floppe
n00b
n00b


Joined: 27 Feb 2003
Posts: 50
Location: Finland

PostPosted: Fri Jul 17, 2009 6:47 am    Post subject: Reply with quote

Really great tool as emerge -e world and system will not pull in every package.

Thanks
Back to top
View user's profile Send private message
NotQuiteSane
Guru
Guru


Joined: 30 Jan 2005
Posts: 481
Location: Klamath Falls, Jefferson, USA, North America, Midgarth

PostPosted: Fri Jul 17, 2009 6:54 am    Post subject: Reply with quote

Floppe wrote:
Really great tool as emerge -e world and system will not pull in every package.

Thanks


um, if emerge -e world doesn't pull in every package, then ther eis a problem

NQS
_________________
These opinions are mine, mine I say! Piss off and get your own.

As I see it -- An irregular blog, Improved with new location

To delete French language packs from system use 'sudo rm -fr /'
Back to top
View user's profile Send private message
Floppe
n00b
n00b


Joined: 27 Feb 2003
Posts: 50
Location: Finland

PostPosted: Fri Jul 17, 2009 6:59 am    Post subject: Reply with quote

So i think too, but don't know what else to do.

It's a really old installation from when Gentoo still had 2.4 as kernel. Have upgraded kernel and gcc and followed the gcc upgrade guide. But still this script finds packages that is not upgraded.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum