Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[solved] paludis and distcc in "pump" mode
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
Moderato
n00b
n00b


Joined: 02 Jun 2006
Posts: 26
Location: Karlsruhe, Baden-Württemberg, Germany

PostPosted: Sat May 22, 2010 7:14 pm    Post subject: [solved] paludis and distcc in "pump" mode Reply with quote

Hi,

is there any way to let paludis use the (hopyfully faster and) new distcc pump mode?

I haven't found any way as of yet.

The "normal" distcc mode is ofc working fine, but my PIII-S (1.4GHz) box is just gaining like 150% speedup chained with a Q9550 (4x2.88Ghz) and a PM (2.0GHz), which I think is still too low, despite having even tried letting make do 30 parallel compilations etc.

Any help would be appreciated.

Best regards,


Mihai


Last edited by Moderato on Mon May 24, 2010 6:54 pm; edited 1 time in total
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 7056

PostPosted: Sat May 22, 2010 7:52 pm    Post subject: Reply with quote

higher jobs priority per cpu type
in distcc.conf you
q4:8 pm: 2 piii 1

this way the q4 will get 8 jobs while distcc will assign just 1 to the pIII and 2 to the pm.
the order they appears also affect who will get a task from higher to lower priority (in case you wish to 1:1 ratio pm & pIII)

i know nothing about paludis (except the name)
Back to top
View user's profile Send private message
tomk
Bodhisattva
Bodhisattva


Joined: 23 Sep 2003
Posts: 7221
Location: Sat in front of my computer

PostPosted: Mon May 24, 2010 7:53 am    Post subject: Reply with quote

Moved from Other Things Gentoo to Unsupported Software, paludis is not an officially supported package manager.
_________________
Search | Read | Answer | Report | Strip
Back to top
View user's profile Send private message
Moderato
n00b
n00b


Joined: 02 Jun 2006
Posts: 26
Location: Karlsruhe, Baden-Württemberg, Germany

PostPosted: Mon May 24, 2010 10:55 am    Post subject: Reply with quote

Hey,

basically, that's what I already have got.

The QC box is the first in list with 8 jobs max, the PM second in list with 2 jobs max.

Make is set up to use 10 jobs in total.

localhost is not comprised in the distcc list (for it being too slow.)

So, everything is cool on configuration's side.

I guess it's impossible to use pump mode with paludis anyway, as you cannot hijack the make call (or, even if, only by editing vital files which will be overwritten anyways.)

Best regards,


Mihai


P.S.: thanks for moving the thread, I was unsure about the category.
Back to top
View user's profile Send private message
sera
Retired Dev
Retired Dev


Joined: 29 Feb 2008
Posts: 1017
Location: CET

PostPosted: Mon May 24, 2010 11:16 am    Post subject: Reply with quote

Moderato wrote:
Hey,

basically, that's what I already have got.

The QC box is the first in list with 8 jobs max, the PM second in list with 2 jobs max.

Make is set up to use 10 jobs in total.

localhost is not comprised in the distcc list (for it being too slow.)

So, everything is cool on configuration's side.

I guess it's impossible to use pump mode with paludis anyway, as you cannot hijack the make call (or, even if, only by editing vital files which will be overwritten anyways.)

Best regards,


Mihai


P.S.: thanks for moving the thread, I was unsure about the category.


Use masquerading as described in man distcc OR set CC=distcc-pump in bashrc. CXX you might want to set as well.
Back to top
View user's profile Send private message
Moderato
n00b
n00b


Joined: 02 Jun 2006
Posts: 26
Location: Karlsruhe, Baden-Württemberg, Germany

PostPosted: Mon May 24, 2010 12:40 pm    Post subject: Reply with quote

Hi,

basically, this is a nice idea, but unfortunately technically unusable.

Based on the information I could gather, distcc-pump mode works only when either wrapping the make call in the included pump script/binary or when starting up pump by using pump --startup, then running make and shutting down pump by running pump --shutdown.

Wrapping the CC/CPP calls into a pump script does not work, I guess.

Maybe hooking the build process will do, I just got that idea now, I'll check that and report back later.

Best regards,


Mihai
Back to top
View user's profile Send private message
Moderato
n00b
n00b


Joined: 02 Jun 2006
Posts: 26
Location: Karlsruhe, Baden-Württemberg, Germany

PostPosted: Mon May 24, 2010 6:53 pm    Post subject: Reply with quote

Hi again,

finally managed it.

Comparison between plain localhost compile (-j2), distcc and distcc-pump (same settings besides using pump mode: -j10; QC/8(,lzo,cpp) PM/2(,lzo,cpp)):

Code:

     Tue May 18 03:27:24 2010 >>> sys-apps/paludis-0.46.0:0
       merge time: 2 hours, 35 minutes and 49 seconds.

     Thu May 20 20:39:11 2010 >>> sys-apps/paludis-0.46.0:0
       merge time: 29 minutes and 44 seconds.

     Mon May 24 16:31:47 2010 >>> sys-apps/paludis-0.46.0:0
       merge time: 14 minutes and 5 seconds.


The difference speaks for itself.

At least with C++.

A compilation of WINE was only 10 minutes faster than previously (50 minutes), but an error occured during compilation quite early and distcc reverted back to using non-pump mode for the rest of it. No idea though, why, as of now.

Anyways, the solution to this problem was rather trivial.

Define the following stuff in /etc/paludis/bashrc:

Code:

EMAKE_WRAPPER="/usr/bin/pump"
DISTCC_HOSTS="host1,cpp host2,cpp, host3,cpp ..."


Hope this helps someone. :)

Best regards,


Mihai
Back to top
View user's profile Send private message
harpette
n00b
n00b


Joined: 09 Jan 2005
Posts: 4
Location: Austin, Tx. USA

PostPosted: Tue Jan 11, 2011 10:22 am    Post subject: Reply with quote

Moderato wrote:

Define the following stuff in /etc/paludis/bashrc:
Code:

EMAKE_WRAPPER="/usr/bin/pump"

Hope this helps someone. :)

It just did. Thanks a million.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum