Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
File list creation for upgrading networkless Gentoo systems
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
Vulpes_
Tux's lil' helper
Tux's lil' helper


Joined: 24 Apr 2004
Posts: 119
Location: Hungary

PostPosted: Wed Aug 25, 2004 12:53 pm    Post subject: File list creation for upgrading networkless Gentoo systems Reply with quote

Hi all!

I have written a small Bash script that generates the list of files necessary to upgrade a networkless Gentoo system, or to install an ebuild on it (it lists files, unlike to 'emerge -pv'), and I would like to know what do you think about it. Any comments are welcomed! The usage instructions are in the code.

<Update #1>

Here's a redesigned new script that fixes everything that was mentioned below. It still creates one Bash executable, but temporary files are now created in /tmp and with a random name (actually derived from the script Process ID) and deleted afterwards, plus all the mirrors provided by Portage are included and used if necessary.

Code:
#!/bin/sh
if [ ! $1 ]
then
echo Getfiles creates another script called getfiles-\<ebuild\>.sh,
echo that uses wget to download all the files necessary to merge
echo an ebuild on a networkless Gentoo system. The files already
echo present in /usr/portage/distfiles/ are excluded, but all files
echo are listed in getfiles-\<ebuild\>.sh. Portage error messages
echo \(if any\) are logged in getfiles-\<ebuild\>.error.
echo
echo USAGE \(without the \<\> signs\):
echo
echo getfiles \<name of ebuild\>
echo getfiles \<name of ebuild\> \<USE=\"...\"\>
echo getfiles \<name of ebuild\> \<ACCEPT_KEYWORDS=\"...\"\>
echo getfiles \<name of ebuild\> \<ACCEPT_KEYWORDS=\"...\"\> \<USE=\"...\"\>
else
 echo Running emerge...
 `echo $2 $3` emerge -pf $1 > /tmp/$$-1.txt 2>&1 #Redirect both stdout and stderr to file.
 if [ `echo $?` -ne 0 ] #In case of any errors...
 then
  echo
  EBUILD=`echo $1 | cut -f2 -d\/`
  echo Creation of script failed!!!
  echo Check getfiles-`echo $1 | cut -f2 -d\/`.error for details!
  mv /tmp/$$-1.txt getfiles-`echo $1 | cut -f2 -d\/`.error
 else
  EBUILD=`echo $1 | cut -f2 -d\/`
  grep tp\:\/\/ /tmp/$$-1.txt | cut -f1 -d\  > /tmp/$$-2.txt #Extracting URLs.
  sort /tmp/$$-2.txt > /tmp/$$-3.txt
  uniq /tmp/$$-3.txt > /tmp/$$-4.txt
  echo \#\!\/bin\/sh > getfiles-`echo $EBUILD`.sh
  echo \#All necessary files: >> getfiles-`echo $EBUILD`.sh
  echo \# >> getfiles-`echo $EBUILD`.sh
  while read line
  do
   a=4
   while [ 1 ]
   do
    b=`echo $line | cut -f$a -d\/`
    test $b
    if [ $? -eq 1 ]
    then
     a=$(($a-1))
     x=`echo $line | cut -f$a -d\/`
     echo \#\#\# $x >> getfiles-`echo $EBUILD`.sh #The list of all necessary files, regardless of whether or not they are present.
     break
    else
     a=$(($a+1))
    fi
   done
  done < /tmp/$$-4.txt
  echo Checking /usr/portage/distfiles/...
  for i in /usr/portage/distfiles/*
  do
   grep `echo $i | cut -f5 -d\/` /tmp/$$-4.txt >> /tmp/$$-5.txt #List of local files already present in /usr/portage/distfiles.
  done
  echo Excluding files already present in /usr/portage/distfiles/...
  cat /tmp/$$-4.txt /tmp/$$-5.txt > /tmp/$$-6.txt
  sort /tmp/$$-6.txt > /tmp/$$-7.txt
  uniq -u /tmp/$$-7.txt > /tmp/$$-8.txt #List of files not present locally in /usr/portage/distfiles.
  echo \# >> getfiles-`echo $EBUILD`.sh
  echo \#All files that need to be downloaded: >> getfiles-`echo $EBUILD`.sh
  echo \# >> getfiles-`echo $EBUILD`.sh
  while read line
  do
  a=4
   while [ 1 ]
   do
    b=`echo $line | cut -f$a -d\/`
    test $b
    if [ $? -eq 1 ]
    then
     a=$(($a-1))
     x=`echo $line | cut -f$a -d\/`
     echo \#\_\# $x >> getfiles-`echo $EBUILD`.sh #List of all files that need to be downloaded.
     break
    else
     a=$(($a+1))
    fi
   done
  done < /tmp/$$-8.txt
  x=`head -n 1 /tmp/$$-8.txt`
  test $x #Test if there is any files to download.
  if [ $? -eq 1 ]
  then
   echo Nothing needs to be downloaded.
   rm getfiles-`echo $EBUILD`.sh
  else
   echo Creating getfiles-`echo $EBUILD`.sh script...
   while read line
   do
    grep $line /tmp/$$-1.txt | head -n 1 | sort | uniq >> /tmp/$$-9.txt #List of the to-be-downloaded files and all of their mirrors.
   done < /tmp/$$-8.txt
   cat -v /tmp/$$-9.txt > /tmp/$$-10.txt #Convert everything to ASCII.
   while read line
   do
    echo $line | cut -f1 -d\^ >> /tmp/$$-11.txt #Remove everything but the URLs.
   done < /tmp/$$-10.txt
   echo \# >> getfiles-`echo $EBUILD`.sh
   echo \#List of all mirrors of all files: >> getfiles-`echo $EBUILD`.sh
   echo \# >> getfiles-`echo $EBUILD`.sh
   while read line
   do
    a=1
    echo $line > /tmp/$$-12.txt #Step-by-step on the mirrorlist of files...
    b=`wc -w /tmp/$$-12.txt | cut -f1 -d\ `
    while [ $a -le $b ]
    do
     c=`cut -f$a -d\  /tmp/$$-12.txt` #One URL at a time.
     echo \# wget -nc $c >> getfiles-`echo $EBUILD`.sh #Adding mirrors to the download script.
     a=$(($a+1))
    done
   done < /tmp/$$-11.txt
   x=`wc -l /tmp/$$-11.txt | cut -f1 -d\ `
   echo \# >> getfiles-`echo $EBUILD`.sh
   echo FILE\=getfiles-`echo $EBUILD`.sh \#Name of this script. >> getfiles-`echo $EBUILD`.sh
   echo echo Download script generated by getfiles\! \> \`echo \$FILE\`\.log >> getfiles-`echo $EBUILD`.sh
   echo echo Downloading `echo $x` files now! \| tee -a \`echo \$FILE\`\.log >> getfiles-`echo $EBUILD`.sh #Creating a header for the download script.
   echo grep wget \$FILE \| cut -f2- -d\\\  \| grep -v \"\$FILE\" \> \/tmp\/\$\$-1.txt \#Total number of mirrors. >> getfiles-`echo $EBUILD`.sh
   echo TIMES\=\`wc -l \/tmp\/\$\$-1.txt \| cut -f1 -d\\\ \` >> getfiles-`echo $EBUILD`.sh
   echo grep \\\#\\\#\\\# \$FILE \| cut -f2 -d\\\  \> \/tmp\/\$\$-2.txt >> getfiles-`echo $EBUILD`.sh
   echo ALLFILES\=\`wc -l \/tmp\/\$\$-2.txt \| cut -f1 -d\\\ \` \#Total number of the files necessary for the merge. >> getfiles-`echo $EBUILD`.sh
   echo grep \\\#\\\_\\\# \$FILE \| cut -f2 -d\\\  \> \/tmp\/\$\$-3.txt >> getfiles-`echo $EBUILD`.sh
   echo FILES\=\`wc -l \/tmp\/\$\$-3.txt \| cut -f1 -d\\\ \` \#Total number of files that need to be downloaded >> getfiles-`echo $EBUILD`.sh
   echo COUNTER\=\$FILES >> getfiles-`echo $EBUILD`.sh
   echo echo Alltogether \$FILES files from \$TIMES mirrors are to be downloaded. >> getfiles-`echo $EBUILD`.sh
   echo while read line >> getfiles-`echo $EBUILD`.sh
   echo do >> getfiles-`echo $EBUILD`.sh
   echo \ grep \$line \/tmp\/\$\$-1.txt \> \/tmp\/\$\$-4.txt \#Mirrors of a certain file. >> getfiles-`echo $EBUILD`.sh
   echo \ STATUS\=1 >> getfiles-`echo $EBUILD`.sh
   echo \ while read LINE >> getfiles-`echo $EBUILD`.sh
   echo \ do >> getfiles-`echo $EBUILD`.sh
   echo \ \ \$LINE \2\>\&\1 \| tee -a \$FILE\.log \#Downloading file... >> getfiles-`echo $EBUILD`.sh
   echo \ \ if \[ \$\? -eq 0 \] >> getfiles-`echo $EBUILD`.sh
   echo \ \ then >> getfiles-`echo $EBUILD`.sh
   echo \ \ \ STATUS\=0 \#Chech whether or not the file is succefully downloaded. >> getfiles-`echo $EBUILD`.sh
   echo \ \ fi >> getfiles-`echo $EBUILD`.sh
   echo \ done \< \/tmp\/\$\$-4.txt \#Mirrors of a certain file, one-by-one. >> getfiles-`echo $EBUILD`.sh
   echo \ if \[ \$STATUS -eq 1 \] >> getfiles-`echo $EBUILD`.sh
   echo \ then >> getfiles-`echo $EBUILD`.sh
   echo \ \ echo Unsuccesful download\: \$line \>\> \$FILE\.log \#Log files that could not be downloaded. >> getfiles-`echo $EBUILD`.sh
   echo \ fi >> getfiles-`echo $EBUILD`.sh
   echo \ COUNTER\=\$\(\(\$COUNTER\-1\)\) >> getfiles-`echo $EBUILD`.sh
   echo \ echo \$COUNTER from \$FILES files are left. \>\> \$FILE\.log >> getfiles-`echo $EBUILD`.sh
   echo done \< \/tmp\/\$\$-3.txt \#Files to be downloaded, one-by-one. >> getfiles-`echo $EBUILD`.sh
   echo grep -q Unsuccesful \$FILE\.log >> getfiles-`echo $EBUILD`.sh
   echo if \[ \$\? -eq 0 \] >> getfiles-`echo $EBUILD`.sh
   echo then >> getfiles-`echo $EBUILD`.sh
   echo \ grep Unsuccesful \$FILE\.log >> getfiles-`echo $EBUILD`.sh
   echo fi >> getfiles-`echo $EBUILD`.sh
   echo echo Done\! >> getfiles-`echo $EBUILD`.sh
   echo rm \/tmp\/\$\$-\*.txt >> getfiles-`echo $EBUILD`.sh
   echo echo Logfile\: \`echo \$FILE\`\.log >> getfiles-`echo $EBUILD`.sh
   chmod +x getfiles-`echo $EBUILD`.sh #Changing the attributes of the download script to executable.
  echo Done!
  fi
  rm /tmp/$$-*.txt #Removing all temporary files from /tmp.
 fi
fi


Last edited by Vulpes_ on Fri Oct 15, 2004 9:35 am; edited 1 time in total
Back to top
View user's profile Send private message
ashtong
n00b
n00b


Joined: 23 Jan 2003
Posts: 44

PostPosted: Sat Aug 28, 2004 3:31 pm    Post subject: Reply with quote

Seeing as my post announcing an alternative approach has been moved to the duplicate threads forum (and I therefore can't reply to your recent message) I'm going to reply here.

Thanks for pointing your script out, I hadn't seen it. I like the idea of making a one off executable that gets what you need; neat. I hope you won't mind a couple of suggestions.

It would be safer not to clobber (potentially) existing temporary files in the current directory. You could avoid this by creating temporary files with the tempfile or mktemp programs. For example:

Code:
temp1=$(tempfile); emerge -pf $1 > $temp1


They'll still need deleting afterwards though, obviously.

I also find it quite useful to make use of emerge's ability to try and download each file from multiple mirrors (that's why there's a loop in my download_files() function). I suspect it's particularly useful to me as I download most of my stuff from an on site mirror at work, and it's not as current as some of the upstream mirrors. If a file isn't available on it my script will then pull it from the next mirror, and then the next, etc.
Back to top
View user's profile Send private message
Vulpes_
Tux's lil' helper
Tux's lil' helper


Joined: 24 Apr 2004
Posts: 119
Location: Hungary

PostPosted: Sat Aug 28, 2004 9:30 pm    Post subject: Reply with quote

Thank you for your suggestions, ashtong.

Regarding the tempfiles, using mktemp seems to be a good thing, the only problem seems to be that neither suggestions (tempfile or mktemp, or any similar apps) are officially part of the Gentoo system. Not a real problem anyway, everyone can install it from source, but it would make the script "mktemp-dependant", for example. I think, it is easier to modify the script to alter temp file names, or to run it in a separate directory.

Your other suggestion is a tough one. Indeed, my script only tries on one mirror, but it is the default mirror, and everyone can and should specify it precisely! On the other hand, if that particular mirror is down, any other mirror can be chosen from the public list to replace the original one, only one sed substitution command will do the trick. True, not as elegant as using a loop, like you... :wink:
Back to top
View user's profile Send private message
Vulpes_
Tux's lil' helper
Tux's lil' helper


Joined: 24 Apr 2004
Posts: 119
Location: Hungary

PostPosted: Sat Oct 02, 2004 11:43 am    Post subject: Reply with quote

Redesigned new script that fixes everything ashtong mentioned above. It still creates one Bash executable, but temporary files are now created in /tmp and with a random name (actually derived from the script Process ID) and deleted afterwards, plus all the mirrors provided by Portage are included and used if necessary.

Last edited by Vulpes_ on Fri Oct 15, 2004 9:36 am; edited 1 time in total
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Sat Oct 09, 2004 6:52 am    Post subject: Reply with quote

The script in this howto and the wiki both seem to accomplish the same thing in two different ways. In the linux tradition, there are even more ways to do the same thing.

Here is one way that seems to have gotten lost in the recent Great Gentoo Forum Reorganization. I post it in hopes someone else remembers the other versions.

On the computer without the network:

Code:

emerge -uDpf world 2>&1 | grep '://' | sed "s: .*::" > download_list


The file download_list now contains the list of packages portage needs...
Then on the computer with the network:

Code:

wget -i download_list



Seems simple enough to me...
Back to top
View user's profile Send private message
ashtong
n00b
n00b


Joined: 23 Jan 2003
Posts: 44

PostPosted: Mon Oct 11, 2004 6:32 am    Post subject: Reply with quote

I really like the simplicity, and I'd use it instead of my script, if it wasn't for the issue of only attempting to download stuff from the first mirror.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Mon Oct 11, 2004 9:17 am    Post subject: Reply with quote

ashtong wrote:
I really like the simplicity, and I'd use it instead of my script, if it wasn't for the issue of only attempting to download stuff from the first mirror.


Ok, Ok, You made me dig through my files to find this:

Without network:
Code:

for url in $(emerge -uDpf world 2>&1); do echo "$url" | egrep 'http://|ftp://'; done > download_list


With network:
Code:

wget -nc -nd -i download_list



This will download only the files not present on the box without the network, and will try all mirrors portage knows about till it finds the file....

A simple script and still meets all your needs?

[EDIT] This script downloads ALL packages :-( see fixed version below =>


Last edited by flybynite on Thu Oct 14, 2004 8:12 pm; edited 1 time in total
Back to top
View user's profile Send private message
ashtong
n00b
n00b


Joined: 23 Jan 2003
Posts: 44

PostPosted: Mon Oct 11, 2004 10:05 pm    Post subject: Reply with quote

Cunning and simple indeed. Thanks for digging.
Back to top
View user's profile Send private message
Vulpes_
Tux's lil' helper
Tux's lil' helper


Joined: 24 Apr 2004
Posts: 119
Location: Hungary

PostPosted: Thu Oct 14, 2004 7:14 am    Post subject: Reply with quote

flybynite: Your solution downloads the files necessary for ebuilds that are not installed, but it does not exclude the files that are already present in /usr/portage/distfiles/. So if you perform an "emerge -f world" (on a networked system), it would still create a huge download_list file afterwards.

Apart from this, indeed simple and elegant, congratulations!
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Thu Oct 14, 2004 8:16 pm    Post subject: Fetching without network Reply with quote

Vulpes_ wrote:
flybynite: Your solution downloads the files necessary for ebuilds that are not installed, but it does not exclude the files that are already present in /usr/portage/distfiles/.

Apart from this, indeed simple and elegant, congratulations!


Aarrgg! You're correct vulpes!! The first version did indeed download everything including files already present on the system. I won't go into how I think it's a portage bug that '-fetch -pretend' isn't the same as '-fetch', but it doesn't matter.....

If you liked the original, you'll love this simple fix :-)


Without network: Must be run as root
Code:

for url in $(http_proxy=null ftp_proxy=null emerge -uDf world 2>&1); do echo "$url" | egrep 'http://|ftp://'; done > download_list


With network:
Code:

wget -nc -nd -i download_list



This will really only download the files not present on the box without the network, and will try all mirrors portage knows about till it finds the file....

Is this still a simple script and meets all your needs?
Back to top
View user's profile Send private message
Vulpes_
Tux's lil' helper
Tux's lil' helper


Joined: 24 Apr 2004
Posts: 119
Location: Hungary

PostPosted: Fri Oct 15, 2004 8:33 am    Post subject: Reply with quote

Yepp, seems to work correctly for me. There is only one question that came into my mind: Will portage try to download everything on a computer that has no active (or any) network interfaces? Currently I am at work, and can not disable the NIC of my PC for a test...
Back to top
View user's profile Send private message
Vulpes_
Tux's lil' helper
Tux's lil' helper


Joined: 24 Apr 2004
Posts: 119
Location: Hungary

PostPosted: Fri Oct 15, 2004 8:39 am    Post subject: Reply with quote

Oops, one problem: If you execute this script on a networked system, it will not only create the download_files file, but download the files themselves too! Modem users won't be too happy about this...
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Sat Oct 16, 2004 12:33 am    Post subject: Reply with quote

Vulpes_ wrote:
Yepp, seems to work correctly for me. There is only one question that came into my mind: Will portage try to download everything on a computer that has no active (or any) network interfaces? Currently I am at work, and can not disable the NIC of my PC for a test...


Actually the script disables the ability to download anything with the http/ftp_proxy=null statement. So it will do the right thing whether or not the box has an active network. I did my testing on a box with an active network connection. Here is a test first with and then without a network...

Code:

gate1 junk # for url in $(http_proxy=null ftp_proxy=null emerge -uDf netcat 2>&1); do echo "$url" | egrep 'http://|ftp://';done > download_list
gate1 junk # cat ./download_list
http://gentoo.osuosl.org/distfiles/libnet-1.1.2.1.tar.gz
http://gentoo.osuosl.org/distfiles/libnet-1.1.2.1.tar.gz
http://gentoo.osuosl.org/distfiles/libnet-1.1.2.1.tar.gz
http://gentoo.osuosl.org/distfiles/libnet-1.1.2.1.tar.gz
http://www.gtlib.cc.gatech.edu/pub/gentoo/distfiles/libnet-1.1.2.1.tar.gz
http://www.gtlib.cc.gatech.edu/pub/gentoo/distfiles/libnet-1.1.2.1.tar.gz
http://www.packetfactory.net/libnet/dist/libnet-1.1.2.1.tar.gz
http://www.packetfactory.net/libnet/dist/libnet-1.1.2.1.tar.gz
http://gentoo.osuosl.org/distfiles/libmix-205.tgz
http://gentoo.osuosl.org/distfiles/libmix-205.tgz
http://gentoo.osuosl.org/distfiles/libmix-205.tgz
http://gentoo.osuosl.org/distfiles/libmix-205.tgz
http://www.gtlib.cc.gatech.edu/pub/gentoo/distfiles/libmix-205.tgz
http://www.gtlib.cc.gatech.edu/pub/gentoo/distfiles/libmix-205.tgz
http://mixter.void.ru/libmix-205.tgz
http://mixter.void.ru/libmix-205.tgz
http://gentoo.osuosl.org/distfiles/nc-v6-20000918.patch.gz
http://gentoo.osuosl.org/distfiles/nc-v6-20000918.patch.gz
http://gentoo.osuosl.org/distfiles/nc-v6-20000918.patch.gz
http://gentoo.osuosl.org/distfiles/nc-v6-20000918.patch.gz
http://www.gtlib.cc.gatech.edu/pub/gentoo/distfiles/nc-v6-20000918.patch.gz
http://www.gtlib.cc.gatech.edu/pub/gentoo/distfiles/nc-v6-20000918.patch.gz
ftp://sith.mimuw.edu.pl/pub/users/baggins/IPv6/nc-v6-20000918.patch.gz
ftp://sith.mimuw.edu.pl/pub/users/baggins/IPv6/nc-v6-20000918.patch.gz
gate1 junk # dhcpcd -k
gate1 junk # ifconfig
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:39634 errors:0 dropped:0 overruns:0 frame:0
          TX packets:39634 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:423043887 (403.4 Mb)  TX bytes:423043887 (403.4 Mb)

gate1 junk # for url in $(http_proxy=null ftp_proxy=null emerge -uDf netcat 2>&1); do echo "$url" | egrep 'http://|ftp://';done > download_list
gate1 junk # cat ./download_list
http://gentoo.osuosl.org/distfiles/libnet-1.1.2.1.tar.gz
http://gentoo.osuosl.org/distfiles/libnet-1.1.2.1.tar.gz
http://gentoo.osuosl.org/distfiles/libnet-1.1.2.1.tar.gz
http://gentoo.osuosl.org/distfiles/libnet-1.1.2.1.tar.gz
http://www.gtlib.cc.gatech.edu/pub/gentoo/distfiles/libnet-1.1.2.1.tar.gz
http://www.gtlib.cc.gatech.edu/pub/gentoo/distfiles/libnet-1.1.2.1.tar.gz
http://www.packetfactory.net/libnet/dist/libnet-1.1.2.1.tar.gz
http://www.packetfactory.net/libnet/dist/libnet-1.1.2.1.tar.gz
http://gentoo.osuosl.org/distfiles/libmix-205.tgz
http://gentoo.osuosl.org/distfiles/libmix-205.tgz
http://gentoo.osuosl.org/distfiles/libmix-205.tgz
http://gentoo.osuosl.org/distfiles/libmix-205.tgz
http://www.gtlib.cc.gatech.edu/pub/gentoo/distfiles/libmix-205.tgz
http://www.gtlib.cc.gatech.edu/pub/gentoo/distfiles/libmix-205.tgz
http://mixter.void.ru/libmix-205.tgz
http://mixter.void.ru/libmix-205.tgz
http://gentoo.osuosl.org/distfiles/nc-v6-20000918.patch.gz
http://gentoo.osuosl.org/distfiles/nc-v6-20000918.patch.gz
http://gentoo.osuosl.org/distfiles/nc-v6-20000918.patch.gz
http://gentoo.osuosl.org/distfiles/nc-v6-20000918.patch.gz
http://www.gtlib.cc.gatech.edu/pub/gentoo/distfiles/nc-v6-20000918.patch.gz
http://www.gtlib.cc.gatech.edu/pub/gentoo/distfiles/nc-v6-20000918.patch.gz
ftp://sith.mimuw.edu.pl/pub/users/baggins/IPv6/nc-v6-20000918.patch.gz
ftp://sith.mimuw.edu.pl/pub/users/baggins/IPv6/nc-v6-20000918.patch.gz
gate1 junk #                                                               



Seems to work fine... Exact same results with or without any network interface...


Last edited by flybynite on Sat Oct 16, 2004 12:48 am; edited 1 time in total
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Sat Oct 16, 2004 12:46 am    Post subject: Reply with quote

Vulpes_ wrote:
Oops, one problem: If you execute this script on a networked system, it will not only create the download_files file, but download the files themselves too! Modem users won't be too happy about this...



Ok, I don't see that behavior. See the above test both with an active network interface and without any network interface AT ALL. Either way it works fine and no files are downloaded.

The "http_proxy=null ftp_proxy=null" will cause any and all attemps to download a file to fail. That is by design and the trick that makes this script work. This is actually not needed for the stated purpose of the script, a box without a network, but I included it for just the situation where you have a slow or expensive connection active but don't want to use it.

Check again and make sure you copied the script correctly.
Back to top
View user's profile Send private message
Vulpes_
Tux's lil' helper
Tux's lil' helper


Joined: 24 Apr 2004
Posts: 119
Location: Hungary

PostPosted: Mon Oct 18, 2004 7:42 am    Post subject: Reply with quote

True, I did some experiment, probably I left the proxy switches out.
Back to top
View user's profile Send private message
Vulpes_
Tux's lil' helper
Tux's lil' helper


Joined: 24 Apr 2004
Posts: 119
Location: Hungary

PostPosted: Mon Oct 18, 2004 3:38 pm    Post subject: Reply with quote

OK, I made alterations utilizing flybynite's solution. The script is now shorter and simpler, but still has error handling, random temp file names and USE FLAGS support. Before anyone would ask why I don't use the one-liner instead, the answer is because it is nice and fancy this way, and because I am a masochist and I like to code just for myself! :wink:

Code:
#!/bin/sh
if [ ! $1 ]
then
 echo Getfiles creates another script called getfiles-\<ebuild\>.sh,
 echo that uses wget to download all the files necessary to merge
 echo an ebuild on a networkless Gentoo system. The files already
 echo present in /usr/portage/distfiles/ are excluded, but all files
 echo are listed in getfiles-\<ebuild\>.sh. Error messages
 echo \(if any\) are logged in getfiles-\<ebuild\>.log.
 echo
 echo USAGE \(without the \<\> signs\):
 echo
 echo getfiles \<name of ebuild\>
 echo getfiles \<name of ebuild\> \<USE=\"...\"\>
 echo getfiles \<name of ebuild\> \<ACCEPT_KEYWORDS=\"...\"\>
 echo getfiles \<name of ebuild\> \<ACCEPT_KEYWORDS=\"...\"\> \<USE=\"...\"\>
else
 EBUILD=`echo $1 | cut -f2 -d\/`
 echo Running emerge...
 http_proxy=null ftp_proxy=null `echo $2 $3` emerge -uDf $1 > /tmp/$$-1.txt 2>&1
 egrep 'http://|ftp://' /tmp/$$-1.txt | cut -f3 -d\  | egrep 'http://|ftp://' > /tmp/$$-2.txt #Mirrors with files that need to be downloaded.
 if [ `echo $?` -ne 0 ] #Check if there is anything to download...
 then
  echo Nothing to download, possible error!
  echo Check getfiles-`echo $EBUILD`.log for details!
  mv /tmp/$$-1.txt getfiles-`echo $EBUILD`.error
 else #There are files to download.
  echo Creating download script: getfiles-`echo $EBUILD`\.sh
  egrep md5 /tmp/$$-1.txt | cut -f5 -d\  > /tmp/$$-3.txt #Files already present in /usr/portage/distfiles.
  egrep Aborting /tmp/$$-1.txt | cut -f4 -d\  > /tmp/$$-4.txt #Files that need to be downloaded.
  echo \#\!\/bin\/sh > getfiles-`echo $EBUILD`.sh #Creating script...
  echo \# >> getfiles-`echo $EBUILD`.sh
  echo \#Files that are necessary, but already in \/usr\/portage\/distfiles. >> getfiles-`echo $EBUILD`.sh
  sed "s/^/\#\ /g" /tmp/$$-3.txt >> getfiles-`echo $EBUILD`.sh
  echo \# >> getfiles-`echo $EBUILD`.sh
  echo \#Files that are necessary, and need to be downloaded. >> getfiles-`echo $EBUILD`.sh
  echo \# >> getfiles-`echo $EBUILD`.sh
  sed "s/^/\#\ /g" /tmp/$$-4.txt >> getfiles-`echo $EBUILD`.sh
  echo \# >> getfiles-`echo $EBUILD`.sh
  echo \#List of download mirrors. >> getfiles-`echo $EBUILD`.sh
  echo \# >> getfiles-`echo $EBUILD`.sh
  sed "s/^/\#\ wget\ \-nc\ \-nd\ /g" /tmp/$$-2.txt >> getfiles-`echo $EBUILD`.sh
  echo FILE\=getfiles-`echo $EBUILD`.sh \#Name of this script. >> getfiles-`echo $EBUILD`.sh
  echo FILES\=`wc -l /tmp/$$-4.txt | cut -f1 -d\ ` \#Number of files that need to be downloaded. >> getfiles-`echo $EBUILD`.sh
  echo echo Alltogether \$FILES files are to be downloaded. >> getfiles-`echo $EBUILD`.sh
  echo grep wget \$FILE \| cut -f2- -d\\\  \| grep -v \"\$FILE\" \> \/tmp\/\$\$\-5\.txt \#Extracting the mirrors from getfiles-`echo $EBUILD`.sh...  >> getfiles-`echo $EBUILD`.sh
  echo while read line >> getfiles-`echo $EBUILD`.sh
  echo do >> getfiles-`echo $EBUILD`.sh
  echo \ \$line \| tee -a \`echo \$FILE\`\.log 2\>\&1 >> getfiles-`echo $EBUILD`.sh
  echo done \< \/tmp\/\$\$\-5\.txt >> getfiles-`echo $EBUILD`.sh
  echo grep Could \`echo \$FILE\`\.log \> \/tmp\/\$\$\-6.txt >> getfiles-`echo $EBUILD`.sh
  echo if \[ \$\? \-ne \0 \] >> getfiles-`echo $EBUILD`.sh
  echo then >> getfiles-`echo $EBUILD`.sh
  echo \ cat \/tmp\/\$\$\-6\.txt >> getfiles-`echo $EBUILD`.sh
  echo else  >> getfiles-`echo $EBUILD`.sh
  echo echo Done! >> getfiles-`echo $EBUILD`.sh
  echo fi >> getfiles-`echo $EBUILD`.sh
  echo echo See \`echo \$FILE\`\.log for details. >> getfiles-`echo $EBUILD`.sh
  echo rm -f \/tmp\/\$\$-\*.txt >> getfiles-`echo $EBUILD`.sh
  echo Done!
  chmod 744 getfiles-`echo $EBUILD`.sh
 fi
 rm -f /tmp/$$-*.txt
fi
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum