Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
distcc - it works (quick but dirty...)
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
de4d
Apprentice
Apprentice


Joined: 12 Sep 2002
Posts: 181
Location: fr. i. br. (ger)

PostPosted: Thu Sep 26, 2002 7:36 pm    Post subject: distcc - it works (quick but dirty...) Reply with quote

if anybody wondered howto use distcc during an emerge process:
(warning: this method might be ridiculous - better ideas appreciated :)
Code:

emerge distcc #(if !done)
mv /usr/bin/make /usr/bin/made

vi/nano/whatever /usr/bin/make add:
Code:

#!/bin/bash
JINT=5#or whatever u like
export DISTCC_HOSTS="hostA hostB [localhost]"
export DISTCC_VERBOSE=1 #if u wanna c it working
/usr/bin/made CC=distcc -j$JINT $1 $2 $3 $4 $5 $6 $7 $8 $9#no idea how to pass all (n) args... :P

now
Code:

emerge <whatever>

could use distcc
have phun

edit:
ok now, $@ passes all arguments
but theres another problem:
the portage ebuild (and several others) use
make CFLAGS="-march=i386 -O3"
when catching this up to pass it to the real make binary, those double quotes are stripped. this is nothing strange - but make complains about a syntax error (as expected). does anybody know a workaround for that?
_________________
void main(){fork();main();}
Back to top
View user's profile Send private message
phong
Bodhisattva
Bodhisattva


Joined: 16 Jul 2002
Posts: 778
Location: Michigan - 15 & Ryan

PostPosted: Fri Sep 27, 2002 4:06 am    Post subject: Reply with quote

Ok, I've been doing a lot of playing with distcc over the past week or two, and I think I can give some tips that will be slightly quicker and a bit less dirty. I planed on writing up a whole HOWTO when I determine if/how distcc can be used for the bootstrap (I've already gotten it to work with emerge system, the kernel compile, most of kde and several misc. things). In the mean time, here's a quickie.

First off, distcc has no built in security mechanisms. Before running the distcc daemon on any volunteer machines, it's critical that you block off tcp port 4200 incoming at your firewall otherwise anybody can run whatever they want with the permissions of the user running distccd (which need not and should not be root BTW).

Second step, take a quick look at the distcc web page and emerge distcc on all the involved machines.

Add the following lines to your /etc/make.conf
Code:
CC="distcc"
CXX="distcc g++"        # may not be neccesary, and may serve only to break things - not sure yet
MAKEOPTS="-j3"          # replace 3 with a number slightly higher than the number of CPUs that will be working
DISTCC_HOSTS="volunteer1 volunteer2 ... localhost"
#DISTCC_VERBOSE=1       # if you want distcc to spew extra output
#DISTCC_LOG="whatever"  # to log distcc instead of using stderr, BUT it's limited to the sandbox

Now go ahead and run distccd on the volunteer machines. Do "distccd --help" to get some options for verbose output and whatnot (it'll work fine with no options though). Before going any further, huff and puff at port 4200 from outside your network to be sure your house is made of bricks.

Now anything you emerge on the original machine should use distcc. Some packages disable the -j option because their makefiles (or whatever) are written in such a way as they cannot be compiled in a parallel fashion. Also, some packages just don't compile with distcc for whatever reason. If one does, just comment out the lines you added to make.conf. If the primary machine is quite a bit slower than the volunteers, you can just comment out the MAKEOPTS line and it may still be able to compile, just passing off the compile job to the volunteers - just not in parallel, and you may still get a significant speedup. So far, I've had arts fail (just taking out the MAKEOPTS let it through, and it took advantage of compiling on a faster volunteer machine). I also had kdelibs fail. After taking out all the options from make.conf, I tried it again, and it failed right away. I thought I'd borked it up good, but then I tired it a third time and it went through no problem. So, the moral is, if a package fails with distcc, it may be a good idea to blow out the corresponding /var/tmp/portage directory manually before trying it again.

Your kernel is one thing that CAN take advantage of distcc for sure, and it gets a really nice boost. It won't use your make.conf settings though:
Code:
# make menuconfig
# make dep && make clean
# export DISTCC_HOSTS="....."
# make CC="distcc" -j3 bzImage
# make CC="distcc" -j3 modules
# make modules_install
# [copy your kernel, etc. etc.]
Good luck!
_________________
"An empty head is not really empty; it is stuffed with rubbish. Hence the difficulty of forcing anything into an empty head."
-- Eric Hoffer
Back to top
View user's profile Send private message
de4d
Apprentice
Apprentice


Joined: 12 Sep 2002
Posts: 181
Location: fr. i. br. (ger)

PostPosted: Fri Sep 27, 2002 7:02 am    Post subject: Reply with quote

tanx a lot!
ive been trying similar stuff before i ended up replacing make by a script - just didnt get it to work ...
should have read some more documentation about make ;))
_________________
void main(){fork();main();}
Back to top
View user's profile Send private message
zen_guerrilla
Guru
Guru


Joined: 18 Apr 2002
Posts: 343
Location: Greece

PostPosted: Mon Sep 30, 2002 9:43 pm    Post subject: Reply with quote

phong wrote:
Now go ahead and run distccd on the volunteer machines.

I followed your instructions on my desktop (volunteer) & my notebook ("emerger"). However when running distccd I get no message and "ps ax | grep distccd" returns null. Also when emerging something on the notebook it returns that connection is not allowed on port 4200 on desktop.
I tried distccd --verbose --daemon but still no message and no ps. Am I doing something wrong here ?

.:: zen ::.
Back to top
View user's profile Send private message
phong
Bodhisattva
Bodhisattva


Joined: 16 Jul 2002
Posts: 778
Location: Michigan - 15 & Ryan

PostPosted: Mon Sep 30, 2002 9:46 pm    Post subject: Reply with quote

Hmmm, does distccd --help give any output? It should give some output to stderr if run with --verbose though... Can you try running it with --log-file=test.log or something to see if anything appears in the log file?
_________________
"An empty head is not really empty; it is stuffed with rubbish. Hence the difficulty of forcing anything into an empty head."
-- Eric Hoffer
Back to top
View user's profile Send private message
zen_guerrilla
Guru
Guru


Joined: 18 Apr 2002
Posts: 343
Location: Greece

PostPosted: Mon Sep 30, 2002 9:59 pm    Post subject: Reply with quote

The log simply told me not to run distcc as root, oops 8) Now everything works gr8. I' m finally emerging gnome2 :)
phong : Thanx for the great tip on distcc. Now I'm gonna finish my gentoo installation at work.

.:: zen ::.
Back to top
View user's profile Send private message
verwilst
n00b
n00b


Joined: 16 Apr 2002
Posts: 30
Location: Belgium, Gent

PostPosted: Mon Oct 07, 2002 5:07 pm    Post subject: Reply with quote

I have the following in /etc/make.conf:


CC="distcc"
CXX="distcc g++"
MAKEOPTS="-j2"
DISTCC_HOSTS="gateway localhost"
DISTCC_VERBOSE=1


and i get this when trying to compile an app with emerge:

checking whether make sets ${MAKE}... yes
checking for i586-pc-linux-gnu-gcc... distcc
checking for C compiler default output... configure: error: C compiler cannot create executables

!!! ERROR: The ebuild did not complete successfully.
!!! Function src_compile, Line 3, Exitcode 77
!!! (no error message)

Any suggestions?
_________________
Bart Verwilst
Gentoo Linux Developer
Gent, Belgium
Back to top
View user's profile Send private message
rac
Bodhisattva
Bodhisattva


Joined: 30 May 2002
Posts: 6553
Location: Japanifornia

PostPosted: Mon Oct 07, 2002 5:46 pm    Post subject: Reply with quote

Anything in config.log that might give us more detail?
_________________
For every higher wall, there is a taller ladder
Back to top
View user's profile Send private message
phong
Bodhisattva
Bodhisattva


Joined: 16 Jul 2002
Posts: 778
Location: Michigan - 15 & Ryan

PostPosted: Mon Oct 07, 2002 10:51 pm    Post subject: Reply with quote

I assume you built distcc before putting that stuff in your make.conf... Can you try just compiling a single .c file with distcc, i.e. "distcc hello_world.c"? Before you do that, export DISTCC_VERBOSE="1" so it will give you more info.
_________________
"An empty head is not really empty; it is stuffed with rubbish. Hence the difficulty of forcing anything into an empty head."
-- Eric Hoffer
Back to top
View user's profile Send private message
gentuse
Tux's lil' helper
Tux's lil' helper


Joined: 15 Aug 2002
Posts: 75
Location: Seattle

PostPosted: Thu Oct 31, 2002 5:35 pm    Post subject: Reply with quote

I would recommend setting CC="distcc gcc" and CXX="distcc g++" to overcome the bug described here: http://lists.samba.org/pipermail/distcc/2002q4/000308.html
Back to top
View user's profile Send private message
kerframil
l33t
l33t


Joined: 19 Apr 2002
Posts: 712

PostPosted: Mon Nov 04, 2002 7:25 pm    Post subject: Thanks so much, but .... Reply with quote

Thanks so much for this thread, in particular the recommendations for make.conf. It works like a charm, although some builds fail or don't take advantage of distcc in the desired manner (I'm keeping a log of these because I'm setting up a new system).

However, I think I just discovered something rather worrying. I was proceeding to emerge app-misc/endeavour on a pretty clean system. I noticed this message when compilation of the tiff library ebuild dependency began:
Code:
Reading site-wide parameters from ./config.site.
Oh no, not another i686-pc-linux-gnu system...
Using /usr/bin/distcc for a C compiler (use -with-CC=compilername to override).
Looks like /usr/bin/distcc supports the -g option.
... but not together with the -march=pentium3 -O3 -pipe option, not using it.
Using /usr/bin/make to configure the software.
Defaulting MACHDEPLIBS to -lm.

Hmm, a quick check of the gcc manpage explains that the "-g" option produces debugging info in a fashion specific for the host's arch. This message seems to imply that use of that gcc flag prevents distcc from using any optimisations! This is quite distressing, because I have emerged a few things now, and I'm not sure if this has happened on any other builds.

If I've got the above right, then I'm wondering whether it would make sense to report them and whether an ebuild can somehow "override" the use of distcc in this fashion, or whether that would be too kludgy. In any case, I think I'll shunt this along to distcc's author because just maybe this issue can be resolved in a future release (I hope).

PS: I tried going through a few different permutations with the C compiler flags "-march=i686", no "-pipe" but it seems to make no difference
Back to top
View user's profile Send private message
rac
Bodhisattva
Bodhisattva


Joined: 30 May 2002
Posts: 6553
Location: Japanifornia

PostPosted: Tue Nov 26, 2002 7:40 am    Post subject: Reply with quote

pahud's question has gone to distcc problem. Please don't post support questions to threads in Documentation, Tips & Tricks.
_________________
For every higher wall, there is a taller ladder
Back to top
View user's profile Send private message
yokem55
Guru
Guru


Joined: 18 Apr 2002
Posts: 360
Location: Oregon

PostPosted: Tue Nov 26, 2002 9:58 pm    Post subject: Reply with quote

Just how scaleable is distcc? Could you say have a 100 machines on a network all helping make a single build that could then be applied to all the systems? I mean, if this could cut a KDE compile time down to minutes when distributed so broadly, I wouldn't be suprised to see corporate sysadmins embracing Gentoo since it would be able to produce full binary packages just as fast as they could download them from the net.
Back to top
View user's profile Send private message
puddpunk
l33t
l33t


Joined: 20 Jul 2002
Posts: 681
Location: New Zealand

PostPosted: Tue Nov 26, 2002 10:28 pm    Post subject: Reply with quote

yep, sure!

just on your emerger be sure to set the config up correctly with your 100 slaves, then emerge with the --buildpkg command, and it will build a package which can be distributed over all 100 machines (you'll have to do that yourself though)
Back to top
View user's profile Send private message
zojas
Veteran
Veteran


Joined: 22 Apr 2002
Posts: 1138
Location: Phoenix, AZ

PostPosted: Fri Nov 29, 2002 10:26 pm    Post subject: distcc +ccache Reply with quote

I use distcc in combination with ccache. that way if the ebuild fails, once you fix it there's not as much work to do.

make sure you have distcc and ccache installed. I put /usr/bin/ccache as the first directory in root's path.

then in your make.conf file: (almost the same as the code above):

Code:
CC="ccache distcc gcc"
CXX="ccache distcc g++"
MAKEOPTS="-j3"          # replace 3 with a number slightly higher than the number of CPUs that will be working
DISTCC_HOSTS="volunteer1 localhost"
#DISTCC_VERBOSE=1       # if you want distcc to spew extra output


then on all the volunteers you need to change the /etc/init.d/distccd script slightly. just before the "start-stop-daemon" line, add this line:
Code:
export PATH="/bin:/usr/bin"


otherwise the distccd server tries to run ccache too.

With this setup, the machine you run the "emerge" on will cache all the compiling. any compiling that gets done will be distributed.
Back to top
View user's profile Send private message
mglauche
Retired Dev
Retired Dev


Joined: 25 Apr 2002
Posts: 564
Location: Germany

PostPosted: Sat Nov 30, 2002 9:08 pm    Post subject: Reply with quote

KDE does not seem to like distcc, so i doubt the 100 machines will help compiling it ..

I think for compiling on such many machines distcc would have to have finer controll over the make process, in order to be still efficient.

I usually get only 30-40% speed improvement by compiling on 3-4 machines, with the distant once being quite a bit faster than the local machine ...
Back to top
View user's profile Send private message
zojas
Veteran
Veteran


Joined: 22 Apr 2002
Posts: 1138
Location: Phoenix, AZ

PostPosted: Sun Dec 01, 2002 3:50 pm    Post subject: Reply with quote

I posted some timings using ccache+distcc on emerging mozilla. There are three cases: with distcc+cold ccache, distcc+warm ccache, and with no distcc/no ccache. the times come out as you expect. :)

http://desertsol.com/~kevin/ccache_distcc.html
Back to top
View user's profile Send private message
fyerk
Apprentice
Apprentice


Joined: 17 Sep 2002
Posts: 212
Location: Atlanta, GA

PostPosted: Fri Dec 13, 2002 6:28 pm    Post subject: Reply with quote

phong wrote:

First off, distcc has no built in security mechanisms. Before running the distcc daemon on any volunteer machines, it's critical that you block off tcp port 4200 incoming at your firewall otherwise anybody can run whatever they want with the permissions of the user running distccd (which need not and should not be root BTW).


One thing to note here is that distcc can be run from within [x]inetd. Permissions can then be controlled at the system level via native xinetd access controls or /etc/hosts.(allow|deny).
_________________
-David
Back to top
View user's profile Send private message
proxy
Apprentice
Apprentice


Joined: 20 Apr 2002
Posts: 260
Location: Chantilly, VA

PostPosted: Tue Dec 17, 2002 11:45 pm    Post subject: Reply with quote

I have noticed an issue which was preventing distcc from working at all:

in the newer ebuilds for gcc, there are two new lines in /etc/env.d/05gcc

they are:

Code:

CC="gcc"
CXX="g++"


these seem top override the values we have been putting in make.conf, therefore making it NOT call "distcc gcc"

the solution is simple, it worked without this before, so comment them out :)

then just do a env-update

I wish there were a more elegnant solution, such as it conditionally setting the values, perhaps the script which reads the /etc/env.d/ files should skip environment vairables which have already been set? or be able to specify ones which will not get overwritten? I dunno, what do you guys think?

proxy
Back to top
View user's profile Send private message
nitro322
Guru
Guru


Joined: 24 Jul 2002
Posts: 562
Location: USA

PostPosted: Thu Jan 16, 2003 5:48 pm    Post subject: Reply with quote

proxy wrote:
I have noticed an issue which was preventing distcc from working at all:

in the newer ebuilds for gcc, there are two new lines in /etc/env.d/05gcc

they are:

Code:

CC="gcc"
CXX="g++"


these seem top override the values we have been putting in make.conf, therefore making it NOT call "distcc gcc"

the solution is simple, it worked without this before, so comment them out :)

then just do a env-update

Thanks, proxy. I couldn't seem to get distcc to work at all, and this turned out to be my problem. Works great now.

I do have a question that I'm concerned about, though. My plan is to run the distcc client on my firewall/router box, a P1 133, and run distccd on my desktop, an Athlon 2100+ MP. Needless to say, I have quite different CFLAGS on the two machines. I want my desktop to do as much of the work as possible, so I have my make.conf (on the firewall) set to:
Code:
DISTCC_HOSTS="desktop"
DISTCC_VERBOSE=1
MAKEOPTS="-j2"
CC="distcc"
CXX="distcc g++"

This will compile on my desktop, but I'm not sure what CFLAGS it's using. I have the desktop set with fairly aggressive optimizations that the firewall definitely can't handle. So, which CFLAGS options will packages on my firewall be compiled with? Will this configuration produce stable binaries? Thanks.
Back to top
View user's profile Send private message
zojas
Veteran
Veteran


Joined: 22 Apr 2002
Posts: 1138
Location: Phoenix, AZ

PostPosted: Thu Jan 16, 2003 8:17 pm    Post subject: Reply with quote

the client tells the distccd what flags to use. I use distcc routinely with a pentium II as a client and an athlon as a server and it works fine. I've also used a 90MHz pentium as a client and that worked too.
Back to top
View user's profile Send private message
mbp
n00b
n00b


Joined: 03 Jul 2002
Posts: 4
Location: Canberra

PostPosted: Fri Jan 17, 2003 1:10 am    Post subject: 100 machines? Reply with quote

I doubt if distcc would scale to 100 machines, for a few reasons:

- Many Makefiles are written in such a way that they can't schedule 100 jobs at once. For example, they may not parallelize across directories, and each directory may have <100 files.

- The client may run out of steam trying to run the preprocessor and distcc client for 100 machines. The client-side overhead is a few percent of the total work, which suggests an upper limit of a few tens of machines.

- The network may saturate trying to pass source and objects around.

Aside from that there is no in-principle reason why distcc couldn't scale up to that number. It would however be interesting to try. Unfortunately I don't have 100 Linux machines to try on. :(
Back to top
View user's profile Send private message
Decibels
Veteran
Veteran


Joined: 16 Aug 2002
Posts: 1598
Location: U.S.A.

PostPosted: Wed Feb 26, 2003 1:49 am    Post subject: Reply with quote

I am having a bit of confusion on the CC & CXX flages. I see most people include them in their make.conf.

Then you also see stuff like this:
Quote:
Regardless of whether you are using portage or not, do not set CC or CXX; the simple default values provided by gcc-config in /etc/env.d/05gcc are used. Setting these to anything that is not a single C program will cause problems with some builds that use incompatible versions of libtool.


from: http://gentoo.superlucidity.net/www/distcc.html

I had just taken them out of make.conf when I was pointed to this page. Only tried to compile the new gcc so far and it failed. Maybe it doesn't like distcc, but still not clear.

Since /etc/env.d/05gcc contains the same info seem like you wouldn't have to include it anywhere else.
Back to top
View user's profile Send private message
nephros
Advocate
Advocate


Joined: 07 Feb 2003
Posts: 2139
Location: Graz, Austria (Europe - no kangaroos.)

PostPosted: Wed Feb 26, 2003 5:22 pm    Post subject: another distcc script Reply with quote

I have four machines at home, all running distccd, but not all are up all the time.

I use the following script therefore, like this:
Code:
[root@scourge]# distcc.sh emerge foo/bar

Code:

CC="distcc gcc"
CXX="distcc c++"
DISTCC_HOSTS="localhost"
MCNT=3 # I have a SMP system, set to 2 for single-cpu

for TRY_HOST in 192.168.0.{1,2,13,23}
do
  echo trying $TRY_HOST
  if ping -q -c 3 -w 5 $TRY_HOST > /dev/null 2>&1 ; then
    echo "$TRY_HOST is up!"
    DISTCC_HOSTS="$DISTCC_HOSTS $TRY_HOST"
    let MCNT++
  fi
done

MAKE="make -j $MCNT"
echo "set: \$CC=\"$CC\" \$CXX=\"$CXX\" \$DISTCC_HOSTS=\"$DISTCC_HOSTS\" \$MAKE=\"$MAKE\""

$@

_________________
Please put [SOLVED] in your topic if you are a moron.
Back to top
View user's profile Send private message
nacs
n00b
n00b


Joined: 21 Feb 2003
Posts: 41

PostPosted: Sat Mar 15, 2003 6:06 pm    Post subject: Reply with quote

After following the instructions in the posts above, I found that distcc wasn't being used. However adding the following line to make.conf fixed it immediately:

Code:
FEATURES="distcc"
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum