Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Checking for stale /etc files
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
TheCoop
Veteran
Veteran


Joined: 15 Jun 2002
Posts: 1814
Location: Where you least expect it

PostPosted: Sat Dec 20, 2003 9:19 pm    Post subject: Checking for stale /etc files Reply with quote

Due to the portage /etc protection, and package that installs files in /etc and is then unmerged leaves the files there. Over time /etc can get very cluttered. Therefore here is a very simple one-liner that will print out the paths of all the files in /etc that do not belong to any package. However, beware of deleting them, as deleting some files in /etc may seriously damage your health as you try to find a livecd to boot from

Code:
for n in /etc/*; do if [[ `qpkg -f $n` = "" ]]; then echo $n; fi; done

You can also change it to run under /etc/init.d or /etc/conf.d, cos atm it doesn't do recursive checks

ps. I know most of you probably would have got this by now, but there are a fair few, including me as of 10 mins ago, dont know the first thing about bash scripting and therefore would not be able to do this. Just bear with me, will you? :D
_________________
95% of all computer errors occur between chair and keyboard (TM)

"One World, One web, One program" - Microsoft Promo ad.
"Ein Volk, Ein Reich, Ein Führer" - Adolf Hitler

Change the world - move a rock


Last edited by TheCoop on Sat Dec 20, 2003 11:38 pm; edited 1 time in total
Back to top
View user's profile Send private message
frekiR
n00b
n00b


Joined: 15 Dec 2003
Posts: 42
Location: Sweden

PostPosted: Sat Dec 20, 2003 11:26 pm    Post subject: Reply with quote

Very nice, good job :)
Back to top
View user's profile Send private message
neenee
Veteran
Veteran


Joined: 20 Jul 2003
Posts: 1786

PostPosted: Sat Dec 20, 2003 11:42 pm    Post subject: Reply with quote

thanks for sharing this. i too do not know enough
about many things in linux myself :wink:
Back to top
View user's profile Send private message
Tazmanian
Apprentice
Apprentice


Joined: 01 Jul 2003
Posts: 222

PostPosted: Sun Dec 28, 2003 12:12 am    Post subject: Re: Checking for stale /etc files Reply with quote

TheCoop wrote:
Code:
for n in /etc/*; do if [[ `qpkg -f $n` = "" ]]; then echo $n; fi; done

You can also change it to run under /etc/init.d or /etc/conf.d, cos atm it doesn't do recursive checks


A modified version that checks /etc recursively and only queries packages that you have installed:
Code:
for n in `find /etc -type f` ; do
    if [[ `qpkg -I -f $n` = "" ]] ; then
        echo $n
    fi
done
Back to top
View user's profile Send private message
BlindSpy
Apprentice
Apprentice


Joined: 20 Mar 2004
Posts: 263

PostPosted: Tue Apr 26, 2005 5:05 pm    Post subject: Reply with quote

Awesome work guys! I've been looking for something like this since ecatmur's cruft script has stopped working for me =).
_________________
Symlinks to:
xorg.conf
Back to top
View user's profile Send private message
Hack_Benjamin
Apprentice
Apprentice


Joined: 12 Nov 2004
Posts: 158

PostPosted: Wed Jul 13, 2005 11:36 am    Post subject: Reply with quote

if i wanted to print what the output was to a file instead of just showing it to me in the term how would i do it?
Back to top
View user's profile Send private message
extragedy
n00b
n00b


Joined: 19 May 2005
Posts: 10

PostPosted: Thu Jul 14, 2005 12:43 am    Post subject: Reply with quote

you can always put ">> your file" without the quotes at the end of the echo file
Back to top
View user's profile Send private message
Hack_Benjamin
Apprentice
Apprentice


Joined: 12 Nov 2004
Posts: 158

PostPosted: Thu Jul 14, 2005 11:16 am    Post subject: Reply with quote

sorry, i thought it would be obvious :P

my plan is to hav that script, then save its output and then open an editor to remove what you wish to keep and then do something like:

su -c
for i in cat `file` rm $i


ace.


great script
Back to top
View user's profile Send private message
Grilo
Tux's lil' helper
Tux's lil' helper


Joined: 22 Apr 2003
Posts: 114
Location: Canada

PostPosted: Fri Jul 15, 2005 5:15 pm    Post subject: Reply with quote

Thanks for the script. I thought you should be able to tell it how to skip the files you knew for sure you shouldnt delete. So Luke MacGregor (dpl) modified the script. here it is

Code:

#!/bin/bash
# this is the list of files that you don't want to trash, make sure they're
# in quotations.

KEEP_FILES=("/etc/make.conf" "/etc/X11/Sessions/e17" "/etc/mtab")

NUM_FILES=${#KEEP_FILES[*]}

BOOL=1

OUTPUT="etc-check.output"

# first remove the output file if it is there.
if [ -e $OUTPUT ]; then
       rm $OUTPUT
fi

for n in `find /etc -type f` ; do
   BOOL=1
   if [[ `qpkg -I -f $n` = "" ]] ; then
       for i in ${KEEP_FILES[@]} ; do
               if [[ $n == ${i} ]] ; then
                       BOOL=0
                       echo not inserting $n
               fi
       done
       if [[ $BOOL == "1" ]] ; then
               echo $n >> $OUTPUT
       fi
   fi
done

in the keep files just list the ones to skip.

this is new and improved. it will check to see if you have an output file then delete it so that you have a new clean list after being run


good luck
Grilo
_________________
Knowledge is power but the drive to learn is harnessing wizdom
Back to top
View user's profile Send private message
rickj
Guru
Guru


Joined: 06 Feb 2003
Posts: 366
Location: Calgary, Alberta, Canada

PostPosted: Sun Apr 30, 2006 6:40 pm    Post subject: Reply with quote

Could some knowledgeable person please update these scripts, in the light of the fact that qpkg has gone the way of the KSR33 Teletype?
Back to top
View user's profile Send private message
TheCoop
Veteran
Veteran


Joined: 15 Jun 2002
Posts: 1814
Location: Where you least expect it

PostPosted: Sun Apr 30, 2006 10:53 pm    Post subject: Reply with quote

Code:
#!/bin/bash
# this is the list of files that you don't want to trash, make sure they're
# in quotations.

KEEP_FILES=("/etc/make.conf" "/etc/X11/Sessions/e17" "/etc/mtab")

NUM_FILES=${#KEEP_FILES[*]}

BOOL=1

OUTPUT="/tmp/etc-check.output"

# first remove the output file if it is there.
if [ -e $OUTPUT ]; then
       rm $OUTPUT
fi

for n in `find /etc -type f` ; do
   BOOL=1
   if [[ $(equery -q -C b $n) = "" ]] ; then
       for i in ${KEEP_FILES[@]} ; do
               if [[ $n == ${i} ]] ; then
                       BOOL=0
                       echo not inserting $n
               fi
       done
       if [[ $BOOL == "1" ]] ; then
               echo $n >> $OUTPUT
       fi
   fi
done

It takes quite a while though. Best to leave it running in a detached screen or something (sh etc-check.sh && beep)
_________________
95% of all computer errors occur between chair and keyboard (TM)

"One World, One web, One program" - Microsoft Promo ad.
"Ein Volk, Ein Reich, Ein Führer" - Adolf Hitler

Change the world - move a rock
Back to top
View user's profile Send private message
think4urs11
Bodhisattva
Bodhisattva


Joined: 25 Jun 2003
Posts: 6659
Location: above the cloud

PostPosted: Mon May 01, 2006 12:45 pm    Post subject: Reply with quote

Tested it on two machines and must say it is slow even when comparing to trees getting older ;)

times for TheCoop-script
a) P4-2400: 120:11 min
b) Via Eden 600: stopped after 656 minutes (approx. 50% done)

my script
a) P4-2400: 4:11 minutes
b) Via Eden 600: 37:57 minutes

all whats needed to speed it up is
replace
Code:
   if [[ $(equery -q -C b $n) = "" ]] ; then

with
Code:
if [[ $(qfile $n) = "" ]] ; then


qfile is part of app-portage/portage-utils

HTH
_________________
Nothing is secure / Security is always a trade-off with usability / Do not assume anything / Trust no-one, nothing / Paranoia is your friend / Think for yourself
Back to top
View user's profile Send private message
dmartinsca
Guru
Guru


Joined: 09 Dec 2005
Posts: 303
Location: Ontario, Canada

PostPosted: Mon May 01, 2006 5:28 pm    Post subject: Reply with quote

Think4UrS11 wrote:
Tested it on two machines and must say it is slow even when comparing to trees getting older ;)

times for TheCoop-script
a) P4-2400: 120:11 min
b) Via Eden 600: stopped after 656 minutes (approx. 50% done)

my script
a) P4-2400: 4:11 minutes
b) Via Eden 600: 37:57 minutes

all whats needed to speed it up is
replace
Code:
   if [[ $(equery -q -C b $n) = "" ]] ; then

with
Code:
if [[ $(qfile $n) = "" ]] ; then


qfile is part of app-portage/portage-utils

HTH


How many files in /etc and how many packages do you have installed?
find /etc -type f | wc -l
find /var/db/pkg -name "CONTENTS" | wc -l

I'm working on a script that just uses grep on the CONTENTS files in /var/db/pkg, doesn't look like it's going to be much faster though.
Back to top
View user's profile Send private message
think4urs11
Bodhisattva
Bodhisattva


Joined: 25 Jun 2003
Posts: 6659
Location: above the cloud

PostPosted: Mon May 01, 2006 5:48 pm    Post subject: Reply with quote

dmartinsca wrote:
How many files in /etc and how many packages do you have installed?


(values for Via Eden)

find /etc -type f | wc -l
1097
find /var/db/pkg -name "CONTENTS" | wc -l
298
output of my script:
337
_________________
Nothing is secure / Security is always a trade-off with usability / Do not assume anything / Trust no-one, nothing / Paranoia is your friend / Think for yourself
Back to top
View user's profile Send private message
BitJam
Advocate
Advocate


Joined: 12 Aug 2003
Posts: 2486
Location: Silver City, NM

PostPosted: Mon May 01, 2006 10:44 pm    Post subject: Reply with quote

Here is a very fast Perl program that will find cruft in /etc/ and its subdirectories. It is very fast because it reads the CONTENTS files just once and puts possible target files (from /etc) in a hash.

It works fine as is, but it could use a bit more work. The list of IGNORE files could probably be larger and I want to add a feature so it can read in additional ignore files both from the command line and from a file so each person can customize which files to ignore on their own system without modifying the code.

It has two limitations, both of which are easy to fix. The first is that it will fail for filenames with spaces in them in the CONTENTS files. Second, it only searches for beginning matches for files from the IGNORE list which makes it very easy to exclude directories. There should probably be an IGNORE_DIR list that still behaves this way and an IGNORE_FILE list that only ignores exact matches.

I am posting it now because it works and is very fast but I'm tired now and may not get around to adding those other features right away.
Code:
#!/usr/bin/perl -w

use strict;
use Getopt::Long;

my $TARGET = "/etc";
my $DB_DIR = "/var/db/pkg";
my @IGNORE = ("/etc/OLD_ETC",
              "/etc/splash",
              "/etc/runlevels",
              "/etc/ssl/certs",
              "/etc/X11/xorg.conf",
              "/etc/group",
              "/etc/shadow",
              "/etc/gshadow-",
              "/etc/fstab",
              "/etc/localtime",
              "/etc/make.conf",
              "/etc/passwd",
              "/etc/resolv.conf",
              "/etc/hosts",
              );

my $ME = $0; $ME =~ s{.*/}{};
my $USAGE = <<USAGE;
Usage: $ME [options]
Options:
  --target DIR   Look for cruft in DIR (default: /etc)
  --db_dir DIR   Look for package files in DIR (default: /var/db/package)
  --pretend      Does nothing yet.
  --verbose      Prints a few summary lines.
  --help         Print this usage.
USAGE

my $PRETEND;
my $VERBOSE;

GetOptions(
    "help"    => sub { print $USAGE; exit},
    "pretend" => \$PRETEND,
    "verbose" => \$VERBOSE,
    "target"  => \$TARGET,
    "db_dir"  => \$DB_DIR,
) or die $USAGE;

#--- Grab valid files from all contents files ---
my %VALID;

for my $dir (<$DB_DIR/*>) {
    for my $package (<$dir/*>) {
        open(CONTENTS, "$package/CONTENTS") or next;
        while (<CONTENTS>) {
            m/^(?:obj|dir) ($TARGET\S*)/o and $VALID{$1}++;
        }
        close CONTENTS;
    }
}

$VERBOSE and print "Found @{[scalar keys %VALID]} valid file names.\n";

my @FILES = `find $TARGET -print`;
chomp(@FILES);

$VERBOSE and print "Found @{[scalar @FILES]} target files.\n";

my @CRUFT;
FILE:
for my $file (@FILES) {
    $VALID{$file} and next FILE;
    for my $re (@IGNORE) {
        $file =~ /^$re/ and next FILE;
    }
    push @CRUFT, $file;
}

print join "\n", @CRUFT, "\n";

$VERBOSE and print "Found @{[scalar @CRUFT]} cruft files.\n";
Back to top
View user's profile Send private message
mikemcquaid
Tux's lil' helper
Tux's lil' helper


Joined: 11 May 2004
Posts: 108
Location: Edinburgh, Scotland

PostPosted: Sun May 07, 2006 11:35 pm    Post subject: Reply with quote

BitJam:
Can't find string terminator "USAGE" anywhere before EOF at ./etccheck line 25.
Back to top
View user's profile Send private message
BitJam
Advocate
Advocate


Joined: 12 Aug 2003
Posts: 2486
Location: Silver City, NM

PostPosted: Mon May 08, 2006 12:05 am    Post subject: Reply with quote

Make sure that line 34 (that just contains the string "USAGE") has no trailing whitespace.
Back to top
View user's profile Send private message
electrofreak
l33t
l33t


Joined: 30 Jun 2004
Posts: 713
Location: Ohio, USA

PostPosted: Sat Jun 10, 2006 9:07 pm    Post subject: Reply with quote

Code:
bash: qpkg: command not found


The perl script works just fine, but seems to spit out WAY to many files.
_________________
Desktop: ABit AN8, Athlon64 X2 4400+ 939 2.75GHz, 2x1GB Corsair XMS DDR400, 2x160GB SATA RAID-0, 2x20"W, Vista Ultimate x64
Laptop: 15.4" MacBook Pro 2.4Ghz, 2x1GB RAM, 160GB, Mac OS X 10.5.1
Server: PIII 550Mhz, 3x128MB RAM, 160GB, Ubuntu Server 7.10
Back to top
View user's profile Send private message
ttuttle
Tux's lil' helper
Tux's lil' helper


Joined: 03 Oct 2004
Posts: 131

PostPosted: Tue Dec 26, 2006 5:32 pm    Post subject: Reply with quote

Try this:

Code:
find /etc | sort > ~/etc-actual
find /var/db/pkg/ -name CONTENTS | xargs -n 1 cat | cut -d" " -f 2 | grep "^/etc" | sort | uniq > ~/etc-intended
diff ~/etc-actual ~/etc-intended | less


Files marked "< file" are files you have but aren't listed in any package; files marked "> file" are files that are listed in a package but no longer on your system (perhaps example config files you renamed or removed).

It will, like most of these scripts, flag files like /etc/X11/xorg.conf that are obviously important but don't actually come with any package. As the script does not remove anything, it's up to you to remove what you don't want and keep what you do.
_________________
Visit my website.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum