Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Emerge Faster
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
sigmalll
Guru
Guru


Joined: 24 Aug 2003
Posts: 332

PostPosted: Sun Nov 14, 2004 12:24 am    Post subject: Emerge Faster Reply with quote

Quite a simple trick to make emerge finish a little sooner, very useful when building something with a lot of dependencies from scatch (like kde)

First in one shell do something like the following

Code:
emerge -fD kde


Then in a second shell

Code:
emerge kde


You will have one shell thats downloading all the required packages one after another, and the second shell will be constantly building. Breaking the compile, wait for download, compile cycle.

Now, you may be wondering what happens when the second shell tries to build something thats not finished downloading, not a worry as it simply waits for the downloading file lock to be removed before starting. This may happen a few times to start with, but after a few packages the download thread pulls ahead.


Last edited by sigmalll on Sun Nov 14, 2004 10:33 am; edited 1 time in total
Back to top
View user's profile Send private message
Mr_Smiley
n00b
n00b


Joined: 29 Apr 2004
Posts: 72
Location: Adelaide, South Australia

PostPosted: Sun Nov 14, 2004 2:28 am    Post subject: Reply with quote

That's a good idea thanks for that :)
Back to top
View user's profile Send private message
hiroki
Guru
Guru


Joined: 29 Jul 2003
Posts: 327
Location: @home in germany

PostPosted: Sun Nov 14, 2004 3:40 pm    Post subject: Reply with quote

that's just what i always do... and i thought everyone does it.. but it seems.. that i was wrong.. not everyone uses this "strategy"...
Back to top
View user's profile Send private message
jack_mort
Apprentice
Apprentice


Joined: 30 Jan 2003
Posts: 167

PostPosted: Sun Nov 14, 2004 4:49 pm    Post subject: Reply with quote

Indeed, I often do like this, but be careful about "synchronisation" : if for whatever reason, the shell compiling tries to compile something which is being downloaded, it will fail and then stop all the emerge process... So launch the downloading shell a few moment before the compiling one ;)
Back to top
View user's profile Send private message
tscolari
l33t
l33t


Joined: 02 May 2004
Posts: 602
Location: curitiba - pr - Brazil

PostPosted: Mon Nov 15, 2004 3:14 am    Post subject: Reply with quote

jack_mort wrote:
Indeed, I often do like this, but be careful about "synchronisation" : if for whatever reason, the shell compiling tries to compile something which is being downloaded, it will fail and then stop all the emerge process... So launch the downloading shell a few moment before the compiling one ;)


all packages suffer md5sum check before extracting and compiling so thats very hard to happen

you could make all this in one big shell:
Code:

emerge -fD kde & emerge kde


:)
_________________
------------------------------------
Tiago Scolari
Back to top
View user's profile Send private message
RinkyDinks_RJ
n00b
n00b


Joined: 12 Aug 2003
Posts: 42

PostPosted: Wed Nov 17, 2004 4:46 am    Post subject: Reply with quote

Quote:
emerge -fD kde & emerge kde


Which doesn't do the same thing. Your code first downloads then compiles only after all files have been downloaded.

If you use two shells and let the download one run for a minute or two first, then start the compiling then you are downloading/compiling at the same time.

God I wish this could be implemented into portage somehow. Would make life soooo much easier.

Also, been doing this forever.
Back to top
View user's profile Send private message
BlindSpy
Apprentice
Apprentice


Joined: 20 Mar 2004
Posts: 263

PostPosted: Wed Nov 17, 2004 4:55 am    Post subject: Re: Emerge Faster Reply with quote

sigmalll wrote:
Now, you may be wondering what happens when the second shell tries to build something thats not finished downloading, not a worry as it simply waits for the downloading file lock to be removed before starting. This may happen a few times to start with, but after a few packages the download thread pulls ahead.


Portage is so smart :D This is an awesome idea and i'd do it except I'm on the Purdue University network and have 100MB/s access to a Gentoo package database :P
_________________
Symlinks to:
xorg.conf
Back to top
View user's profile Send private message
yogipsu
Tux's lil' helper
Tux's lil' helper


Joined: 30 Aug 2003
Posts: 98

PostPosted: Wed Nov 17, 2004 7:58 pm    Post subject: Reply with quote

Quote:
Which doesn't do the same thing. Your code first downloads then compiles only after all files have been downloaded.


& isn't the same as &&.

What you're describing is &&, which waits until the first process to complete and only begins the second upon successful completion of the first.
Back to top
View user's profile Send private message
castorilo
Apprentice
Apprentice


Joined: 25 Dec 2002
Posts: 157

PostPosted: Wed Nov 17, 2004 9:10 pm    Post subject: Reply with quote

jack_mort wrote:
Indeed, I often do like this, but be careful about "synchronisation" : if for whatever reason, the shell compiling tries to compile something which is being downloaded, it will fail and then stop all the emerge process... So launch the downloading shell a few moment before the compiling one ;)


Since portage 2.0.51, they added locks and that problem is no longer a concern. If the second emerge tries to compile a package that is being downloaded, it just waits until the download finishes, and then proceeds to compile the package.
Back to top
View user's profile Send private message
etnoy
Apprentice
Apprentice


Joined: 29 Aug 2003
Posts: 255
Location: Västerås, Sweden

PostPosted: Thu Nov 18, 2004 9:04 pm    Post subject: Reply with quote

paulpach wrote:
jack_mort wrote:
Indeed, I often do like this, but be careful about "synchronisation" : if for whatever reason, the shell compiling tries to compile something which is being downloaded, it will fail and then stop all the emerge process... So launch the downloading shell a few moment before the compiling one ;)


Since portage 2.0.51, they added locks and that problem is no longer a concern. If the second emerge tries to compile a package that is being downloaded, it just waits until the download finishes, and then proceeds to compile the package.


Ugh, I have always been a little against this. What if the computer gets a power failure in the midde of a d/l, and the "lock" still is there? The build will wait and wait...
_________________
The md5sum of the above post is 06280ccd85ef9deb49c336e7945f4b5c

God is dead! - Nietzsche
Nietzsche is dead! -God
Back to top
View user's profile Send private message
busfahrer
n00b
n00b


Joined: 18 Sep 2004
Posts: 57
Location: Germany

PostPosted: Fri Nov 19, 2004 12:17 am    Post subject: Reply with quote

Someone suggest this to the portage developers, it would make a nice addition if implemented properly.
Sidenote: I also want emerge --foo bar which does the same thing as emerge -f bar && emerge bar ;)
Back to top
View user's profile Send private message
thecrazyperson_ws
Tux's lil' helper
Tux's lil' helper


Joined: 06 Apr 2004
Posts: 89

PostPosted: Sun Nov 21, 2004 10:11 pm    Post subject: Reply with quote

This would be a great thing to implement into portage. It would be especially useful for us 56K users out there because it would allow the download shell to get a little ways ahead before the first compile finishes. I will start using this so called strategery as soon as my bootstrap finishes.
_________________
Bevelle: Dual Athlon MP 2000+, 1.25G ram, 120G HD, Gentoo 2005.1-r1, Kernel 2.6.15-gentoo-r5

Gagazet: Pentium 2-266, 288M ram, 240G HD, currently installing gentoo

Besaid: p-150, 80M ram, win2k pro
Back to top
View user's profile Send private message
Twink
Apprentice
Apprentice


Joined: 06 Dec 2002
Posts: 178
Location: New Zealand

PostPosted: Sun Nov 21, 2004 10:28 pm    Post subject: Reply with quote

etnoy wrote:
paulpach wrote:
jack_mort wrote:
Indeed, I often do like this, but be careful about "synchronisation" : if for whatever reason, the shell compiling tries to compile something which is being downloaded, it will fail and then stop all the emerge process... So launch the downloading shell a few moment before the compiling one ;)


Since portage 2.0.51, they added locks and that problem is no longer a concern. If the second emerge tries to compile a package that is being downloaded, it just waits until the download finishes, and then proceeds to compile the package.


Ugh, I have always been a little against this. What if the computer gets a power failure in the midde of a d/l, and the "lock" still is there? The build will wait and wait...


i guess locks could be removed on system boot or something, although if you're cheating like me and have several machines using one nfs share to download to only the machine that actually owns the drive should be allowed to remove them. Other then that I love the new locking system, helps fix a number of problems i had.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum