Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Is portage capable of emerging 2 diff programs?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
MrVahn
Apprentice
Apprentice


Joined: 30 Dec 2006
Posts: 227
Location: Makati

PostPosted: Sun Dec 31, 2006 12:28 pm    Post subject: Is portage capable of emerging 2 diff programs? Reply with quote

Hi there. I'm new to the gentoo community. I did my assignment (read about the docs) but upon reading through one question about portage left unanswered.

I remember the time when I first installed gentoo. I broke it. Maybe because I ran 2 different terminals each emerging different programs. I did not even reach the stage of experiencing a WM on gentoo lol. Now I'm afraid to run multiple emerges.
The question is, is running multiple emerges at the same time allowed? Or is it harmful for the system?

Thanks
Vahn
Back to top
View user's profile Send private message
Dieter@be
Guru
Guru


Joined: 03 Apr 2005
Posts: 314
Location: Wetteren, Belgium

PostPosted: Sun Dec 31, 2006 1:00 pm    Post subject: Reply with quote

does it matter? why simply not use "emerge <package1> <package2> <package3>" for example? it will emerge the packages after each other.

if it's performance you're after, i don't think running to emerges at the same time will improve performance (because the more you run at the same time, the less cpu power is available per emerge process)
_________________
Nothing beats a ride on the Gentoo learning curve.
2 Gentoo Linux boxes - 2 Arch Linux boxes - Love all four
Say no to i386 binaries! -- Adopt an unanswered topic
Back to top
View user's profile Send private message
MrVahn
Apprentice
Apprentice


Joined: 30 Dec 2006
Posts: 227
Location: Makati

PostPosted: Sun Dec 31, 2006 1:11 pm    Post subject: Reply with quote

I see. Thanks for the quick reply. AFAIK when I did 2 emerges it caused a circular error. Im lost after that. BTW what happens to the downloaded sources? Are they deleted after being compiled? I heard from someone that he does not like gentoo because it eats too much space.
Back to top
View user's profile Send private message
PaulBredbury
Watchman
Watchman


Joined: 14 Jul 2005
Posts: 7310

PostPosted: Sun Dec 31, 2006 1:50 pm    Post subject: Reply with quote

MrVahn wrote:
AFAIK when I did 2 emerges it caused a circular error.

Don't do simultaneous compilations, unless you're sure they don't affect each other. Even better, don't do them at all, in case Portage has a problem with it (including subtle race conditions).

The downloaded sources are in /usr/portage/distfiles. Compilation occurs in /var/tmp/portage.

Quote:
I heard from someone that he does not like gentoo because it eats too much space.

There are much better reasons for disliking Gentoo. It fills a niche for those who see the benefits, and are prepared to put up with the annoyance, of compiling everything from source.
Back to top
View user's profile Send private message
MrVahn
Apprentice
Apprentice


Joined: 30 Dec 2006
Posts: 227
Location: Makati

PostPosted: Sun Dec 31, 2006 2:19 pm    Post subject: Reply with quote

Wow. I'm really amazed of the quick replies! Thanks guys. It's a great deal of help. I'm actually enjoying my time with Gentoo. It actually gives me time to do more things while compiling a ton of stuff LOL. Things out of the computer world like cooking, social life, and giving my aching back a stretch. LOL.

One last thing. Is it safe to delete the source files? What will happen if the compiling is terminated in some way (ie internet disconnection, energy fluctuation etc)? Will it still work afterwards or will the program being emerged malfunction?

Thanks
Vahn
Back to top
View user's profile Send private message
PaulBredbury
Watchman
Watchman


Joined: 14 Jul 2005
Posts: 7310

PostPosted: Sun Dec 31, 2006 2:38 pm    Post subject: Reply with quote

MrVahn wrote:
Is it safe to delete the source files?

/usr/portage/distfiles/* :?: Yes, it's just a cache. Don't do it during an emerge, in case you remove a file which is just about to be used.

However, if space is a big concern, then take a look at your options.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43751
Location: 56N 3W

PostPosted: Sun Dec 31, 2006 3:16 pm    Post subject: Reply with quote

MrVahn,

Multiple emerges at the same time are a bad idea. It doesn't save any time and can prolong the entire process because of CPU cache thrashing, disk extra I/Oand so on. Worse, several users have reported their world file being truncated, although thats supposed to be fixed now.

To optimise emerge speed, play with
Code:
MAKEOPTS="-jX"
in your /etc/make.conf. X is the number of files you want to allow make to process concurrently. A good starting point for X is number of CPUs/Cores +1.
Higher numbers work well as long as you can use the -pipe CFLAG without using disk.

My Athlon XP 3200+ with 1G RAM seems to like -j5. You need to do your own experiments
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
MrVahn
Apprentice
Apprentice


Joined: 30 Dec 2006
Posts: 227
Location: Makati

PostPosted: Sun Dec 31, 2006 5:20 pm    Post subject: Reply with quote

That's surely a heap of info. Thanks for sharing it to me guys. It helped a lot. I'll try to experiment with the MAKEOPTS variable. I only have an Athlon XP 2GHz + 384 mb of ram so maybe it can only handle j2 - j3.
Back to top
View user's profile Send private message
sonicbhoc
Veteran
Veteran


Joined: 24 Oct 2005
Posts: 1805
Location: In front of the computer screen

PostPosted: Sun Dec 31, 2006 6:46 pm    Post subject: Reply with quote

MAKEOPTS="-jx" should be the number of processes you have +1. Mine is -j2 because mine is a single core processor.

Also, after you are done emerging stuff it might be a good idea to:

rm -rf /usr/portage/distfiles/* /var/tmp/portage/*

so that it doesn't take up too much space.
_________________
I'm too lazy to keep this stupid signature up to date, so here's something more interesting:
My friend Hetdegon can draw if you ask me.
Now using PClinuxOS on my laptop and Gentoo on my desktop and new laptop.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43751
Location: 56N 3W

PostPosted: Sun Dec 31, 2006 8:06 pm    Post subject: Reply with quote

sonicbhoc,

Thats old rule of thumb thats a good starting point for further tuning.
It comes from a time when CFLAGS -pipe wasn't useful because RAM was too small and emacs meant 'Eight Megs And Constantly Swapping"
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
timeBandit
Bodhisattva
Bodhisattva


Joined: 31 Dec 2004
Posts: 2674
Location: here, there or in transit

PostPosted: Sun Dec 31, 2006 9:34 pm    Post subject: Reply with quote

NeddySeagoon wrote:
It comes from a time when CFLAGS -pipe wasn't useful because RAM was too small and emacs meant 'Eight Megs And Constantly Swapping"

:D
First and only time emacs has made me laugh, even indirectly.

Thanks for the tip on -jX, I'm just about to update a boatload of packages and will try a little tuning first.

EDIT: Based on a random sample of one package :) and tests at -j2 through -j5, my system (P4 2.4GHz w/1.5GB) peaks with -j3, completing the compile phase about 20% faster than with -j2. Well worth the 20 minutes it took to determine. 8)
_________________
Plants are pithy, brooks tend to babble--I'm content to lie between them.
Super-short f.g.o checklist: Search first, strip comments, mark solved, help others.
Back to top
View user's profile Send private message
Dieter@be
Guru
Guru


Joined: 03 Apr 2005
Posts: 314
Location: Wetteren, Belgium

PostPosted: Mon Jan 01, 2007 5:07 pm    Post subject: Reply with quote

sorry for going offtopic (the problem is solved anway ;))
but how do you guys test your new opts settings? time an emerge -e <some big package> ?
_________________
Nothing beats a ride on the Gentoo learning curve.
2 Gentoo Linux boxes - 2 Arch Linux boxes - Love all four
Say no to i386 binaries! -- Adopt an unanswered topic
Back to top
View user's profile Send private message
spb
Retired Dev
Retired Dev


Joined: 02 Jan 2004
Posts: 2135
Location: Cambridge, UK

PostPosted: Mon Jan 01, 2007 5:29 pm    Post subject: Re: Is portage capable of emerging 2 diff programs? Reply with quote

MrVahn wrote:
The question is, is running multiple emerges at the same time allowed? Or is it harmful for the system?
Since noone seems to have answered outright yet: If the two dependency graphs have no packages in common, then it should be safe. Otherwise it's likely to break in new and exciting ways.
Back to top
View user's profile Send private message
valkyrite
Apprentice
Apprentice


Joined: 19 Sep 2002
Posts: 241

PostPosted: Mon Jan 01, 2007 7:09 pm    Post subject: Re: Is portage capable of emerging 2 diff programs? Reply with quote

spb wrote:
MrVahn wrote:
The question is, is running multiple emerges at the same time allowed? Or is it harmful for the system?
Since noone seems to have answered outright yet: If the two dependency graphs have no packages in common, then it should be safe. Otherwise it's likely to break in new and exciting ways.


That's why it's a safe bet not to emerge multiple packages at the same time (unless you are a l33t and you know what you are doing). :evil:
Back to top
View user's profile Send private message
LaintalAy
n00b
n00b


Joined: 30 Dec 2003
Posts: 10

PostPosted: Mon Jan 01, 2007 7:44 pm    Post subject: Reply with quote

Multiple emerges are sometimes useful if one compilation takes too much time and you want to emerge a simple little program to do something quickly. Specially during the installation of gentoo, when you need a little util or script and you're compiling, for example, Xorg and KDE.

I've desired this feature sometimes in Debian's apt... because of long downloads instead of compile times, of course :)
_________________
Sorry for my english.
Back to top
View user's profile Send private message
valkyrite
Apprentice
Apprentice


Joined: 19 Sep 2002
Posts: 241

PostPosted: Mon Jan 01, 2007 8:44 pm    Post subject: Reply with quote

I run the following command in one terminal.
Code:

emerge -f blah


and regular emerge in a different terminal.
Back to top
View user's profile Send private message
mark_alec
Bodhisattva
Bodhisattva


Joined: 11 Sep 2004
Posts: 6066
Location: Melbourne, Australia

PostPosted: Mon Jan 01, 2007 9:09 pm    Post subject: Reply with quote

valkyrite wrote:
I run the following command in one terminal.
Code:

emerge -f blah


and regular emerge in a different terminal.

FEATURES="parallel-fetch" will enable portage to do that automatically (fetching in the background, while compiling programs.)
_________________
www.gentoo.org.au || #gentoo-au
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43751
Location: 56N 3W

PostPosted: Mon Jan 01, 2007 10:50 pm    Post subject: Reply with quote

spb,

Has the rumored issue with truncated world files been addressed.
Some years ago now, when both emerges wanted to update world at the same time, there were reports that it got truncated.
As you know, this is AVeryBadThing.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
spb
Retired Dev
Retired Dev


Joined: 02 Jan 2004
Posts: 2135
Location: Cambridge, UK

PostPosted: Mon Jan 01, 2007 10:58 pm    Post subject: Reply with quote

No idea. I don't follow Portage development much these days.
Back to top
View user's profile Send private message
sonicbhoc
Veteran
Veteran


Joined: 24 Oct 2005
Posts: 1805
Location: In front of the computer screen

PostPosted: Tue Jan 02, 2007 12:03 am    Post subject: Reply with quote

NeddySeagoon wrote:
spb,

Has the rumored issue with truncated world files been addressed.
Some years ago now, when both emerges wanted to update world at the same time, there were reports that it got truncated.
As you know, this is AVeryBadThing.


I don't think that happens anymore - there is a lock on changing the world file while one running version of portage is doing it at the same time as far as I can tell. Well, at least that's what it looks like over here on my computer.
_________________
I'm too lazy to keep this stupid signature up to date, so here's something more interesting:
My friend Hetdegon can draw if you ask me.
Now using PClinuxOS on my laptop and Gentoo on my desktop and new laptop.
Back to top
View user's profile Send private message
valkyrite
Apprentice
Apprentice


Joined: 19 Sep 2002
Posts: 241

PostPosted: Tue Jan 02, 2007 6:04 am    Post subject: Reply with quote

mark_alec wrote:


and regular emerge in a different terminal.

FEATURES="parallel-fetch" will enable portage to do that automatically (fetching in the background, while compiling programs.)[/quote]

Thanks. :D
Back to top
View user's profile Send private message
sonicbhoc
Veteran
Veteran


Joined: 24 Oct 2005
Posts: 1805
Location: In front of the computer screen

PostPosted: Tue Jan 02, 2007 12:56 pm    Post subject: Reply with quote

I wondered why it wasn't working... I spelled parallel wrong. :lol:
_________________
I'm too lazy to keep this stupid signature up to date, so here's something more interesting:
My friend Hetdegon can draw if you ask me.
Now using PClinuxOS on my laptop and Gentoo on my desktop and new laptop.
Back to top
View user's profile Send private message
timeBandit
Bodhisattva
Bodhisattva


Joined: 31 Dec 2004
Posts: 2674
Location: here, there or in transit

PostPosted: Tue Jan 02, 2007 2:48 pm    Post subject: Reply with quote

Dieter@be wrote:
sorry for going offtopic (the problem is solved anway ;))
but how do you guys test your new opts settings? time an emerge -e <some big package> ?

I timed just the compile phase, like so (using ntp as my test) package:
Code:
ebuild /usr/portage/net-misc/ntp/ntp-4.2.2_p3.ebuild unpack
time ebuild /usr/portage/net-misc/ntp/ntp-4.2.2_p3.ebuild compile
ebuild /usr/portage/net-misc/ntp/ntp-4.2.2_p3.ebuild clean

I need to re-test with a bigger package, something in the 10-20 minute range (ntp compiles in <2:30 for me). Over the weekend I updated about 200 packages and many builds were significantly (as much as 50%) slower than the last merge. I reverted to -j2 when I noticed this, but that may have been premature: for one, I didn't bother to detach the screen session from the console, which adds a ton of I/O time as the framebuffer scrolls. Stupid me. :oops:
_________________
Plants are pithy, brooks tend to babble--I'm content to lie between them.
Super-short f.g.o checklist: Search first, strip comments, mark solved, help others.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum