Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
POWER DOWNLOADING: 1 file from {$} mirrors, with speed eval.
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
Hackeron
Guru
Guru


Joined: 01 Nov 2002
Posts: 307

PostPosted: Sun Feb 29, 2004 5:35 pm    Post subject: POWER DOWNLOADING: 1 file from {$} mirrors, with speed eval. Reply with quote

Ever wanted 10 simultaneous connections to 10 mirrors using a great mirror search engine with ability to resume and have mirror speed testing on the fly, all integrated into portage to replace the crappy combinations of: wget + mirrorselect or prozilla which never finds mirrors and just gets you banned from providers?

Anyway, solution is here,

Code:

emerge axel


and add the following to lines to /etc/make.conf

Code:

FETCHCOMMAND="/usr/bin/axel -a -S10 \${URI} -o \${DISTDIR}"
RESUMECOMMAND="/usr/bin/axel -a -S10 \${URI} -o \${DISTDIR}"


This comes with everything you need, change 10 to the required number of simultaneous mirrors and enjoy :)

Please test and give me your opinion.
Back to top
View user's profile Send private message
snakattak3
Guru
Guru


Joined: 11 Dec 2002
Posts: 468
Location: Seattle

PostPosted: Mon Mar 01, 2004 5:23 am    Post subject: Reply with quote

If means putting less strain on the gentoo mirrors, then i'm all for it. But I really only think it would help out with really large files, like Americas Army, or Enemy Territory. The trouble with those, is that there aren't enough mirrors, as I believe you can only download those from the designated servers. I did a test with mozilla, wich is about 30megs. It also had download enigmail-0.83 and ipc-1.0.5 to satisfy dependancies. The problem lies with axel doing its searching and testing. Anyways, here's the results.
Code:
time emerge --fetchonly mozilla
real    1m52.879s
user    0m1.628s
sys     0m0.619s

With the axel method
Code:
time emerge --fetchonly mozilla
real    3m41.491s
user    0m2.839s
sys     0m1.123s

It took almost 2 full minutes longer using axel as the fetchcommand, and like I said, i think its because it has to search and test each connection for every file it downloads. So i'm thinking that this really is only useful for really huge files like ET or AA, and on the other hand, the downloading speeds were almost identical. About 300 - 350 K/s with a cable connection. If I am wrong in thinking this, let me know. But like I said earlier, I really like this idea, and if this will put less strain on the gentoo mirrors, i'll definately use it. Otherwise, the default fetching using wget is more than fine for me.
_________________
Ban Reality TV!
Adopt an Unanswered Post
Back to top
View user's profile Send private message
Hackeron
Guru
Guru


Joined: 01 Nov 2002
Posts: 307

PostPosted: Mon Mar 01, 2004 9:46 am    Post subject: Reply with quote

Well, the point is to take the load of mirrors while speeding up, again, take the 10 simultaneous mirrors number down for faster start, I set mine to 4 mirrors. That way the test for speed is very fast.

This is indeed extremely helpful for large files, like enemy territory was only downloading at 12KB/sec before, now it's at 60KB/sec. This is also incredibly useful when a new package comes out, like KDE3.2, but there are not many mirrors yet, I was getting KDE at 4KB/sec, now when KDE 3.3 comes out, I'll be getting it at 60KB/sec

But I don't find this any slower than using wget anyway. And you should read the comments on freshmeat.net about Axel, because your benchmarks are pretty rare.. even for smaller files like xmms.

Here is a test with a 52Kb file:

Code:
 WGET:
10:10:08 (19.25 KB/s) - `gentoo_ice-xmms-0.2.tar.bz2.1' saved [51839]


real    0m7.024s


Code:
AXEL -S4:
Downloaded 50.6 kilobytes in 1 second. (45.67 KB/s)

real    0m8.125s


So yes, it can be slower for small files, but this is for a 52Kb file, and Axel although overall slower, downloaded much quicker.

As for slightly larger, but common fize files, the following shows a test for a 2Mb file:

Code:
WGET:
10:16:38 (28.26 KB/s) - `xmms-1.2.8.tar.bz2.1' saved [2322808/2322808]

real    1m21.372s


Code:
AXEL -S4:
Downloaded 1821.2 kilobytes in 38 seconds. (46.77 KB/s)

real    0m51.354s


So there you have it. Overall in my opinion this is worth any potential slow down, this takes load off mirrors, makes sure you never end up with a slow mirror, and speeds things up significantly for larger files.
Back to top
View user's profile Send private message
DArtagnan
l33t
l33t


Joined: 30 Apr 2002
Posts: 942
Location: Israel, Jerusalem

PostPosted: Mon Mar 01, 2004 12:38 pm    Post subject: Re: POWER DOWNLOADING: 1 file from {$} mirrors, with speed e Reply with quote

Hackeron wrote:
Ever wanted 10 simultaneous connections to 10 mirrors using a great mirror search engine with ability to resume and have mirror speed testing on the fly, all integrated into portage to replace the crappy combinations of: wget + mirrorselect or prozilla which never finds mirrors and just gets you banned from providers?

Anyway, solution is here,

Code:

emerge axel


and add the following to lines to /etc/make.conf

Code:

FETCHCOMMAND="/usr/bin/axel -a -S10 \${URI} -o \${DISTDIR}"
RESUMECOMMAND="/usr/bin/axel -a -S10 \${URI} -o \${DISTDIR}"


This comes with everything you need, change 10 to the required number of simultaneous mirrors and enjoy :)

Please test and give me your opinion.


Try prozilla and then see what speed is... :-)
_________________
All for one and one for All
--

MACPRO machine...
Back to top
View user's profile Send private message
Hackeron
Guru
Guru


Joined: 01 Nov 2002
Posts: 307

PostPosted: Mon Mar 01, 2004 1:16 pm    Post subject: Re: POWER DOWNLOADING: 1 file from {$} mirrors, with speed e Reply with quote

DArtagnan wrote:

Try prozilla and then see what speed is... :-)


Prozilla downloads from only !ONE! mirror, if you get a slow mirror, say <5KB/sec, it MAY squeeze a few more KB/sec, but you normally just get banned.

All prozilla does is open multiple connections to the !SAME! mirror.

Axel will download from as many mirrors as you wish, so you can have 5 x 5Kb/sec mirrors, and have fast downloads, putting a much lower load on mirrors overall.
Back to top
View user's profile Send private message
ed0n
l33t
l33t


Joined: 23 Apr 2003
Posts: 638
Location: Prishtine/Kosove

PostPosted: Mon Mar 01, 2004 1:19 pm    Post subject: Re: POWER DOWNLOADING: 1 file from {$} mirrors, with speed e Reply with quote

Quote:


Try prozilla and then see what speed is... :-)


Yeah that's true :) . I use prozilla
Back to top
View user's profile Send private message
ikaro
Advocate
Advocate


Joined: 14 Jul 2003
Posts: 2526
Location: Denmark

PostPosted: Mon Mar 01, 2004 2:38 pm    Post subject: Reply with quote

works fine here, thx for the tip.


another thing i use the in FETCHCOMMAND is :
FETCHCOMMAND='efetch ${URI} ${DISTDIR} ftp://sunsite.dk/projects/deltup/patchfiles'

when you want to upgrade a package, it searches for a patch first instead of downloading the whole package, saves some bw.
_________________
linux: #232767
Back to top
View user's profile Send private message
DArtagnan
l33t
l33t


Joined: 30 Apr 2002
Posts: 942
Location: Israel, Jerusalem

PostPosted: Mon Mar 01, 2004 3:52 pm    Post subject: Re: POWER DOWNLOADING: 1 file from {$} mirrors, with speed e Reply with quote

ed0n wrote:
Quote:


Try prozilla and then see what speed is... :-)


Yeah that's true :) . I use prozilla


The only problem with prozilla is prozilla does not know to work if you are behind proxy! :-((
_________________
All for one and one for All
--

MACPRO machine...
Back to top
View user's profile Send private message
aaronjb
Tux's lil' helper
Tux's lil' helper


Joined: 25 May 2003
Posts: 106
Location: Berkshire, UK

PostPosted: Tue Mar 02, 2004 9:45 pm    Post subject: Reply with quote

I was having problems getting wget to work properly - FTP connections were a no-go, and http connections would only transfer a few hundred Kb at a time before restarting.. (No idea why, wget always used to work fine!)

Popped Axel in instead, and everything is working just fine again, and quickly too, although my wDSL connection doesn't really like the simultaneous downloads (or my router doesn't, one or the other).

One question though - am I right in thinking that it works best if you don't have a GENTOO_MIRRORS line in /etc/make.conf, so that it searches based on gentoo.oregonstate.edu ?

Cheers and thanks for the excellent pointer to axel :)
Aaron
_________________
#my gentoo farm
Back to top
View user's profile Send private message
Doctor
n00b
n00b


Joined: 11 Feb 2003
Posts: 26

PostPosted: Wed Mar 03, 2004 2:40 am    Post subject: Reply with quote

Uhm, axel doesn't seem gentoo specific so how does it know where else to download from? No mention of a mirrors list in the ebuild either.
Back to top
View user's profile Send private message
antenore
n00b
n00b


Joined: 15 Jul 2003
Posts: 46

PostPosted: Wed Mar 03, 2004 12:31 pm    Post subject: Reply with quote

Very Good!!!!

For download via proxy You must set

HTTP_PROXY=http://xxx.xxx.xxx.xxx:xxxx
FTP_PROXY=http://xxx.xxx.xxx.xxx:xxxx

export HTTP_PROXY FTP_PROXY

Tanks again...
Back to top
View user's profile Send private message
antenore
n00b
n00b


Joined: 15 Jul 2003
Posts: 46

PostPosted: Wed Mar 03, 2004 12:37 pm    Post subject: Reply with quote

:oops: Thanks again... :oops:
Back to top
View user's profile Send private message
Hackeron
Guru
Guru


Joined: 01 Nov 2002
Posts: 307

PostPosted: Wed Mar 03, 2004 2:19 pm    Post subject: Reply with quote

Doctor wrote:
Uhm, axel doesn't seem gentoo specific so how does it know where else to download from? No mention of a mirrors list in the ebuild either.


Axel uses filesearching.com , basically, since gentoo is a popular distro the filesearching looks at all the regular ftps for all sorts of distros, you get a nice number of mirrors found.

All it does is look at the first mirror in "GENTOO_MIRRORS" to get information about file, then searches on filesearching.com.
Back to top
View user's profile Send private message
Balthasar
n00b
n00b


Joined: 25 May 2003
Posts: 62
Location: Florida

PostPosted: Wed Mar 03, 2004 6:27 pm    Post subject: even with ten... Reply with quote

Well I tried it while also downloading a LiveCD (which is going at 150k)...
This is running with 10 mirrors

Results of emerge -f mozilla :

With Axel :

real 6m7.015s
user 0m3.664s
sys 0m2.786s

Without Axel - Pure wget

real 8m14.067s
user 0m2.473s
sys 0m1.887s


Thanks.
_________________
"Trust the computer industry to shorten ‘Year 2000’ to ‘Y2K’. It was this kind of thinking that caused the problem in the first place."

Just because it's common sense, doesn't mean it's common practice. - Will Rogers
Back to top
View user's profile Send private message
aaronjb
Tux's lil' helper
Tux's lil' helper


Joined: 25 May 2003
Posts: 106
Location: Berkshire, UK

PostPosted: Thu Mar 04, 2004 9:08 pm    Post subject: Reply with quote

Hmm.. I've noticed an annoying problem..

Sometimes, occasionally, axel will just... stop... like below:

Code:
>>> emerge (1 of 3) sys-libs/libcap-1.10 to /
>>> Downloading ftp://ftp.belnet.be/mirror/rsync.gentoo.org/gentoo/distfiles/libcap-1.10.tar.bz2
Initializing download: ftp://ftp.belnet.be/mirror/rsync.gentoo.org/gentoo/distfiles/libcap-1.10.tar.bz2
Doing search...
Testing speeds, this can take a while...
10 usable servers found, will use these URLs:
URL                                                                    Speed
ftp://ftp.freenet.de/pub/.disk0/ftp.snt.utwente.nl/pub/os/linux/gentoo   504
ftp://ftp.freenet.de/pub/.disk0/ftp.de.kernel.org/pub/linux/libs/secur   551
ftp://ftp.rfc822.org/.disk2/ftp.kernel.org/pub/linux/libs/security/lin   622


And that's all you ever get - it never actually downloads the file, and as far as I can tell, isn't actually utilising the network at all (I've not checked with iptraf or similar)..

Very annoying, and I'm not sure why :(
_________________
#my gentoo farm
Back to top
View user's profile Send private message
ali3nx
l33t
l33t


Joined: 21 Sep 2003
Posts: 621
Location: Winnipeg, Canada

PostPosted: Fri Mar 05, 2004 5:15 am    Post subject: Reply with quote

i've never been dissapointed with wget :D

Code:
esfu root # emerge -f apache
Calculating dependencies ...done!
>>> emerge (1 of 1) net-www/apache-2.0.48-r4 to /
>>> Downloading http://66.90.81.114/gentoo/distfiles/httpd-2.0.48.tar.gz
--05:17:14--  http://66.90.81.114/gentoo/distfiles/httpd-2.0.48.tar.gz
           => `/usr/portage/distfiles/httpd-2.0.48.tar.gz'
Connecting to 66.90.81.114:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6,258,490 [application/x-tar]
 
100%[====================================================================================>] 6,258,490     25.28M/s
 
05:17:14 (25.22 MB/s) - `/usr/portage/distfiles/httpd-2.0.48.tar.gz' saved [6258490/6258490]
 
>>> httpd-2.0.48.tar.gz MD5 ;-)
>>> md5 src_uri ;-) httpd-2.0.48.tar.gz
esfu root #


_________________
Compiling Gentoo since version 1.4
Thousands of Gentoo Installs Completed
Emerged on every continent but Antarctica
Compile long and Prosper!
Back to top
View user's profile Send private message
Hackeron
Guru
Guru


Joined: 01 Nov 2002
Posts: 307

PostPosted: Fri Mar 05, 2004 5:15 pm    Post subject: Reply with quote

aaronjb wrote:
Hmm.. I've noticed an annoying problem..

Sometimes, occasionally, axel will just... stop... like below:

Code:
>>> emerge (1 of 3) sys-libs/libcap-1.10 to /
>>> Downloading ftp://ftp.belnet.be/mirror/rsync.gentoo.org/gentoo/distfiles/libcap-1.10.tar.bz2
Initializing download: ftp://ftp.belnet.be/mirror/rsync.gentoo.org/gentoo/distfiles/libcap-1.10.tar.bz2
Doing search...
Testing speeds, this can take a while...
10 usable servers found, will use these URLs:
URL                                                                    Speed
ftp://ftp.freenet.de/pub/.disk0/ftp.snt.utwente.nl/pub/os/linux/gentoo   504
ftp://ftp.freenet.de/pub/.disk0/ftp.de.kernel.org/pub/linux/libs/secur   551
ftp://ftp.rfc822.org/.disk2/ftp.kernel.org/pub/linux/libs/security/lin   622


And that's all you ever get - it never actually downloads the file, and as far as I can tell, isn't actually utilising the network at all (I've not checked with iptraf or similar)..

Very annoying, and I'm not sure why :(


Yes, I noticed that happened to me as well recently, must be a bug in axel, I contacted the auther (who also wrote bitlbee), he said he doesn't care and he's not planning to fix bugs in a program he wrote 3 years ago. Maybe someone else can pick up where he started from?, this is really a great program if a few bugs are fixed.
Back to top
View user's profile Send private message
brassj41
n00b
n00b


Joined: 28 Mar 2003
Posts: 39

PostPosted: Wed Sep 22, 2004 2:09 pm    Post subject: Re: POWER DOWNLOADING: 1 file from {$} mirrors, with speed e Reply with quote

ed0n wrote:
Quote:


Try prozilla and then see what speed is... :-)


Yeah that's true :) . I use prozilla


um, how do you set up emerge to use prozilla?


[edit]nevermind.. I found out. [/edit]


Last edited by brassj41 on Mon Sep 27, 2004 3:09 am; edited 1 time in total
Back to top
View user's profile Send private message
neenee
Veteran
Veteran


Joined: 20 Jul 2003
Posts: 1786

PostPosted: Wed Sep 22, 2004 3:11 pm    Post subject: Reply with quote

thanks for sharing :wink:
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum