Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
How-To Bootstrap and 'emerge system' with distcc
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
securiteaze
Tux's lil' helper
Tux's lil' helper


Joined: 24 Oct 2002
Posts: 77
Location: Tulsa,Oklahoma

PostPosted: Thu Oct 24, 2002 6:26 pm    Post subject: How-To Bootstrap and 'emerge system' with distcc Reply with quote

How-To Bootstrap with distcc
I have successfully bootstrapped a gentoo box from stage1-x86-1.4_rc1.
The following outlines the process used.
Disclaimer: This may break your system! There may be more graceful ways of doing this ... but it worked for me :D

At the point during an install when you would normally run
Code:
scripts/bootstrap.sh

Don't.
To use distcc while bootstrapping, do the following:

  1. Emerge distcc to your system
    Code:
    emerge distcc

  2. Backup bootstrap.sh
    Code:
    cp /usr/portage/scripts/bootstrap.sh /usr/portage/scripts/bootstrap.sh.orig

  3. Edit bootstrap.sh to make it use distcc.
    Code:
    nano -w /usr/portage/scripts/bootstrap.sh

    Find the following section in bootstrap.sh.
    Code:
    cp /etc/make.conf /etc/make.conf.build
    export CFLAGS="`$PYTHON -c 'import portage; print portage.settings["CFLAGS"];'`"
    export CHOST="`$PYTHON -c 'import portage; print portage.settings["CHOST"];'`"
    export CXXFLAGS="`$PYTHON -c 'import portage; print portage.settings["CXXFLAGS"];'`"
    export MAKEOPTS="`$PYTHON -c 'import portage; print portage.settings["MAKEOPTS"];'`"

  4. Comment out the line :
    Code:

    export MAKEOPTS="`$PYTHON -c 'import portage; print portage.settings["MAKEOPTS"];'`"

    Should now look like this :
    Code:
    cp /etc/make.conf /etc/make.conf.build
    export CFLAGS="`$PYTHON -c 'import portage; print portage.settings["CFLAGS"];'`"
    export CHOST="`$PYTHON -c 'import portage; print portage.settings["CHOST"];'`"
    export CXXFLAGS="`$PYTHON -c 'import portage; print portage.settings["CXXFLAGS"];'`"
    #export MAKEOPTS="`$PYTHON -c 'import portage; print portage.settings["MAKEOPTS"];'`"

  5. Add a new line above the one you just commented out.
    Code:
    export MAKEOPTS="-j12 CC='distcc' CXX='distcc'"

    The -j12 specifies the number of threads to use; you may want to tweak this. (ie -j8 or -j32).
    The rest tells make to use distcc for compiling c and c++.

    Should now look like this :
    Code:
    cp /etc/make.conf /etc/make.conf.build
    export CFLAGS="`$PYTHON -c 'import portage; print portage.settings["CFLAGS"];'`"
    export CHOST="`$PYTHON -c 'import portage; print portage.settings["CHOST"];'`"
    export CXXFLAGS="`$PYTHON -c 'import portage; print portage.settings["CXXFLAGS"];'`"
    export MAKEOPTS="-j12 CC='distcc' CXX='distcc'"
    #export MAKEOPTS="`$PYTHON -c 'import portage; print portage.settings["MAKEOPTS"];'`"

  6. Add one more line :
    Code:
    export DISTCC_HOSTS="localhost <distccd-srv> <distccd-srv> <distccd-srv>"

    Replace <distccd-srv> with the hostname/IP of the boxes running distccd.
    This line tells distcc which hosts to send work and in which order.
    On a really slow box you may wish to leave out localhost or move it to last.

    Should now look similar to this:
    Code:
    cp /etc/make.conf /etc/make.conf.build
    export CFLAGS="`$PYTHON -c 'import portage; print portage.settings["CFLAGS"];'`"
    export CHOST="`$PYTHON -c 'import portage; print portage.settings["CHOST"];'`"
    export CXXFLAGS="`$PYTHON -c 'import portage; print portage.settings["CXXFLAGS"];'`"
    export DISTCC_HOSTS="localhost 10.1.1.25 10.1.1.50"
    export MAKEOPTS="-j12 CC='distcc' CXX='distcc'"
    #export MAKEOPTS="`$PYTHON -c 'import portage; print portage.settings["MAKEOPTS"];'`"

  7. Now, scroll the end of the script.*
    Code:

    #make.conf has been overwritten, so we explicitly export our original settings
    export USE="$ORIGUSE bootstrap"
    emerge $myGLIBC $myBASELAYOUT $myTEXINFO $myGETTEXT $myBINUTILS $myGCC || cleanup 1
    #restore original make.conf
    cleanup 0

  8. Find the following line and comment it out.*
    Code:
    emerge $myGLIBC $myBASELAYOUT $myTEXINFO $myGETTEXT $myBINUTILS $myGCC || cleanup 1

    It should now look like
    Code:

    #make.conf has been overwritten, so we explicitly export our original settings
    export USE="$ORIGUSE bootstrap"
    #emerge $myGLIBC $myBASELAYOUT $myTEXINFO $myGETTEXT $myBINUTILS $myGCC || cleanup 1
    #restore original make.conf
    cleanup 0

  9. Now add the following lines above the one you commented out.*
    Code:

    export MAKEOPTS="-j12 CC='distcc gcc' CXX='distcc g++'"
    emerge $myGLIBC || cleanup 1
    export MAKEOPTS="-j12 CC='distcc' CXX='distcc'"
    emerge  $myBASELAYOUT $myTEXINFO $myGETTEXT $myBINUTILS $myGCC || cleanup 1

    Should finally look like
    Code:

    #make.conf has been overwritten, so we explicitly export our original settings
    export USE="$ORIGUSE bootstrap"
    export MAKEOPTS="-j12 CC='distcc gcc' CXX='distcc g++'"
    emerge $myGLIBC || cleanup 1
    export MAKEOPTS="-j12 CC='distcc' CXX='distcc'"
    emerge  $myBASELAYOUT $myTEXINFO $myGETTEXT $myBINUTILS $myGCC || cleanup 1
    #emerge $myGLIBC $myBASELAYOUT $myTEXINFO $myGETTEXT $myBINUTILS $myGCC || cleanup 1
    #restore original make.conf
    cleanup 0

  10. Save bootstrap.sh then
    Code:

    cd /usr/portage && scripts/bootstrap.sh

    Enjoy!

*Edited to fix a bug in emerging glibc 10/25
*Edited to change subject 10/27

Comments? Suggestions?
_________________
Blah..


Last edited by securiteaze on Sun Oct 27, 2002 7:32 pm; edited 2 times in total
Back to top
View user's profile Send private message
securiteaze
Tux's lil' helper
Tux's lil' helper


Joined: 24 Oct 2002
Posts: 77
Location: Tulsa,Oklahoma

PostPosted: Sun Oct 27, 2002 7:25 pm    Post subject: How-To emerge system with distcc Reply with quote

How-To emerge system with distcc
Disclaimer: This may break your system! There may be more graceful ways of doing this ... but it worked for me :D

This method allows you to 'emerge system' using distcc.
A few packages have issues, but the workaround is included.

If you are continuing from a distcc bootstrap skip to step 2

  1. Install distcc
    Code:
    emerge sync ; emerge distcc

  2. Tweak /etc/make.conf for distcc

    1. Backup make.conf
      Code:
      cp /etc/make.conf /etc/make.conf.orig

    2. Edit /etc/make.conf to use distcc

      The following could be the entire content of make.conf
      Code:

      #DISTCC_HOSTS tells distcc which hosts to send work
      DISTCC_HOSTS="localhost <distccd-srv> <distccd-srv>"
      #Make sure to replace <distccd-srv> with the hostname/IP of the boxes running distccd

      #MAKEOPTS specifies commandline options when portage call the make command
      MAKEOPTS="-j16 CC='distcc' CXX='distcc'"
      #the -j16 tell make to spawn 16 threads, you may want to tweak this

      #If you don't know what the next lines do then see note below
      CHOST="i686-pc-linux-gnu"
      CFLAGS="-march=pentium4 -O3 -pipe"
      CXXFLAGS="${CFLAGS}"

      Note: All about C flags

    3. Save make.conf

  3. Now start the build
    Code:
    emerge system

  4. There will be some packages which fail to build(util-linux, bin86, procps, psmisc, netkit-base).
    For some reason some packages don't like the CC='distcc', CC='distcc gcc', CXX='distcc', or CXX='distcc g++'.
    Other packages (procps) don't like parallel makes (-j2).

    Edit /etc/make.conf and comment out the MAKEOPTS line.
    It should look like this
    Code:

    #MAKEOPTS specifies commandline options when portage call the make command
    #MAKEOPTS="-j16 CC='distcc' CXX='distcc'"
    #the -j16 tell make to spawn 16 threads, you may want to tweak this

  5. Now build the finnicky packages
    Code:
    emerge netkit-base procps psmisc

  6. Edit /etc/make.conf and uncomment the MAKEOPTS line.
    Code:

    #MAKEOPTS specifies commandline options when portage call the make command
    MAKEOPTS="-j16 CC='distcc' CXX='distcc'"
    #the -j16 tell make to spawn 16 threads, you may want to tweak this

  7. Continue the build
    Code:
    emerge system

  8. When 'emerge system' fails again repeat step 4

  9. Build the last of the finnicky packages.
    Code:
    emerge util-linux bin86

  10. Repeat step 5 and 6. There shouldn't be any more build failures.

    Enjoy!

Comments? Suggestions?
_________________
Blah..
Back to top
View user's profile Send private message
gentuse
Tux's lil' helper
Tux's lil' helper


Joined: 15 Aug 2002
Posts: 75
Location: Seattle

PostPosted: Thu Oct 31, 2002 5:32 pm    Post subject: Reply with quote

Good tip. I would recommend setting CC="distcc gcc" and CXX="distcc g++" to overcome the bug described here: http://lists.samba.org/pipermail/distcc/2002q4/000308.html
Back to top
View user's profile Send private message
phong
Bodhisattva
Bodhisattva


Joined: 16 Jul 2002
Posts: 778
Location: Michigan - 15 & Ryan

PostPosted: Thu Oct 31, 2002 5:51 pm    Post subject: Reply with quote

Thanks for the glibc workaround! I was going to write up this exact HOWTO but couldn't figure out how to get past glibc so I didn't bother (too bad there are other things that break unless you set CC="distcc" because they don't like a compiler that's two words). It's also nice to know it's an actual bug (non-recognition of the .S extension) and not just a general shortcoming that can't be fixed.
_________________
"An empty head is not really empty; it is stuffed with rubbish. Hence the difficulty of forcing anything into an empty head."
-- Eric Hoffer
Back to top
View user's profile Send private message
BradN
Advocate
Advocate


Joined: 19 Apr 2002
Posts: 2391
Location: Wisconsin (USA)

PostPosted: Mon Nov 11, 2002 6:23 am    Post subject: Reply with quote

maybe you could make it run a script which then ran "distcc gcc" or something?
Back to top
View user's profile Send private message
BradB
Apprentice
Apprentice


Joined: 18 Jun 2002
Posts: 190
Location: Christchurch NZ

PostPosted: Sun Dec 15, 2002 7:50 pm    Post subject: Reply with quote

I followed this guide and had no problems installing. My workstation is an Athlon800 and the client is a Compaq Armada M300 (P2-300Mhz).
My only tips are
- to emerge packages without distcc and without changing /etc/make.conf you can
Code:
MAKEOPTS="" emerge blah

- the distccd daemon will not run as root - took me ages to figure that one out!!

Cheers
Brad
Back to top
View user's profile Send private message
zojas
Veteran
Veteran


Joined: 22 Apr 2002
Posts: 1138
Location: Phoenix, AZ

PostPosted: Tue Jan 07, 2003 12:32 am    Post subject: Reply with quote

ok, this is fun! but I've run into something strange. I'm bootstrapping on a 90 MHz pentium (this will be my iptables firewall and DNS server; until last night it was running redhat 7.2). my distcc_host contains only "raven", the hostname of my other machine, a 700 MHz athlon tbird.

Started the bootstrap around 9pm last night. At around 9:30 this morning was the last time the distccd on raven was connected to. So I'm thinking it's either done or it failed. when I got home just now I see that not only is it still running but it's invoking gcj to compile java code!! what package is that? this is the bootstrap.sh step still. so apparently it's been putzing with java for 8 hours, which of course all has to compile locally.
Back to top
View user's profile Send private message
zhenlin
Veteran
Veteran


Joined: 09 Nov 2002
Posts: 1361

PostPosted: Tue Jan 07, 2003 8:36 am    Post subject: Reply with quote

gcj is java, and is probably from gcc compiling it's own class files. USE="-java"
Back to top
View user's profile Send private message
Jeld
Tux's lil' helper
Tux's lil' helper


Joined: 28 Jun 2002
Posts: 84
Location: NYC, US

PostPosted: Fri Feb 07, 2003 8:40 pm    Post subject: Reply with quote

I have followed this guide, to the letter, and some of the stuff has compiled very fast ( Celeron 533 as client Athlon 1800 XP server ). Gcc though doesn't seem to use distcc. At some point into the compilation it switches to something it calles xgcc and compiles using that.
_________________
package JAPH;sub x{$/='$';@1=map{$_=ord;$_--;chr}
split//,<DATA>;@2=map{$_=ord;$_++;chr}split//
,<DATA>;$_=sub{$.++%2?shift@2:shift@1};bless$_;}
1;$x=JAPH->x;for(1..25){print&$x,;}__DATA__
Kt!ouf!fmIdf"$ts@ngqOq`jq
Back to top
View user's profile Send private message
phong
Bodhisattva
Bodhisattva


Joined: 16 Jul 2002
Posts: 778
Location: Michigan - 15 & Ryan

PostPosted: Fri Feb 07, 2003 9:00 pm    Post subject: Reply with quote

Unfortunately, that is a fact of life when compiling gcc. First it compiles itself using the host compiler (in this case, gcc via. distcc) using very conservative options. Then using that compiled version, it compiles itself again using optimizations. This part can't be done with distcc because it needs to use its freshly compiled copy. Then it compiles itself for a third and final time using the new optimized version. Again, this can't be done with distcc. Finally, the two final versions are compared. If they're identical, it considers itself to be working and installs. If not, it knows something bad happened along the way. Unfortunately, compling three times is slow, and more noticably when using distcc since that only works for the first one.
_________________
"An empty head is not really empty; it is stuffed with rubbish. Hence the difficulty of forcing anything into an empty head."
-- Eric Hoffer
Back to top
View user's profile Send private message
Jeld
Tux's lil' helper
Tux's lil' helper


Joined: 28 Jun 2002
Posts: 84
Location: NYC, US

PostPosted: Fri Feb 07, 2003 10:23 pm    Post subject: Reply with quote

Sorry for my last comment, I already found this information in a different thread. currently I am doing emerge system, and some packages seem to be not distcc friendly. So far I found that bash will use gcc no matter what MAKEOPT says.

Edit: I will start a collection of packages that do not use MAKEOPTS

bash-2.05b-r3
groff-1.18.1
perl-5.8.0-r9
db-3.2.9
expat-1.95.5-r1
ttmkfdir-3.0.4
pam-0.75-r11

XFree is compiling now, so my collection will probably have to stop here ( will let emerge run overnight ). A couple of things worth mentioning abuot speeding up the compile.

1. If you are trying to install on a slower system and have one fast system available you can try to partition the drive, mount partitions in /mnt/gentoo export /mnt/gentoo over nfs and mount on the fast system. Then you chroot and build. I cuoldn't get the nfsd running off the live CD, but then again, my CD is a bit outdated ( 1.4_rc1 I believe ).
2. On most systems ( even the slower ones ) hard drive access times are much slower then compile times. Therefore setting MAKEOPTS to -j8 or something similar may speed up the compile even on a single CPU system due to the fact that several source files are read at once and then compiled according to the kernel schedule, instead of having to wait for multiple disk accesses.
3. In the same spirit, during compiles gcc writes temp files into /tmp doing
mount -t tmpfs tmpfs /mnt/gentoo/tmp
imprives compile speeds dramatically on both my Athlon 1800 XP and Celeron 533.

Good luck.
_________________
package JAPH;sub x{$/='$';@1=map{$_=ord;$_--;chr}
split//,<DATA>;@2=map{$_=ord;$_++;chr}split//
,<DATA>;$_=sub{$.++%2?shift@2:shift@1};bless$_;}
1;$x=JAPH->x;for(1..25){print&$x,;}__DATA__
Kt!ouf!fmIdf"$ts@ngqOq`jq
Back to top
View user's profile Send private message
AlterEgo
Veteran
Veteran


Joined: 25 Apr 2002
Posts: 1619

PostPosted: Sat Feb 08, 2003 10:35 am    Post subject: Reply with quote

Jeld wrote:

3. In the same spirit, during compiles gcc writes temp files into /tmp doing
mount -t tmpfs tmpfs /mnt/gentoo/tmp
imprives compile speeds dramatically on both my Athlon 1800 XP and Celeron 533.

Jeld, please explain this one. I don't understand the gain in this 8O
Back to top
View user's profile Send private message
Jeld
Tux's lil' helper
Tux's lil' helper


Joined: 28 Jun 2002
Posts: 84
Location: NYC, US

PostPosted: Sat Feb 08, 2003 11:09 am    Post subject: Reply with quote

I am not sure, but logicaly speaking, disk writes are time consuming, and since the temp object files are going to be needed by gcc right away teh write operation have to be completed then and there ( i.e. the write is not really buffered ). So instead of writing temp files to the disk you are writing them to a RAM disk ( tempfs in this case ). Writing to a tempfs disk is much faster then teh hard drive so the compile goes one read/write operation faster. At least that is how I explain this, I might be wrong. It does speed things up quite a bit even on my Celeron 533 128 MB RAM. Also some other operations require temp files such as gzip, groff etc.
_________________
package JAPH;sub x{$/='$';@1=map{$_=ord;$_--;chr}
split//,<DATA>;@2=map{$_=ord;$_++;chr}split//
,<DATA>;$_=sub{$.++%2?shift@2:shift@1};bless$_;}
1;$x=JAPH->x;for(1..25){print&$x,;}__DATA__
Kt!ouf!fmIdf"$ts@ngqOq`jq
Back to top
View user's profile Send private message
ekoontz
n00b
n00b


Joined: 18 Apr 2002
Posts: 67
Location: San Francisco, California

PostPosted: Mon Feb 10, 2003 4:55 am    Post subject: Reply with quote

Just to follow up on the suggestion that BradN made above.
(also mentioned by Martin Pool here : http://lists.samba.org/pipermail/distcc/2002q4/000308.html


I tried having :

Code:

MAKEOPTS="-j3 CC=distcc CXX=distcc g++"


But this did not work for some packages that seemed to want CXX to be one token. So, I wrote a wrapper script which I installed as /usr/bin/distcc-c++ :

Code:

#!/bin/sh
distcc c++ $*


Then I put in my make.conf :

Code:

MAKEOPTS="-j3 CC=distcc CXX=distcc-c++"


Works well now!
Back to top
View user's profile Send private message
Donovan
Tux's lil' helper
Tux's lil' helper


Joined: 08 Feb 2003
Posts: 97
Location: Halifax, NS, Canada

PostPosted: Tue Feb 18, 2003 7:08 pm    Post subject: tempfs rocks Reply with quote

Jeld wrote:
Writing to a tempfs disk is much faster then teh hard drive so the compile goes one read/write operation faster.


Exactly! Using tmpfs means it's being held in RAM, and no matter how fast your hard drives are, they'll never be as fast as pure RAM. Since it's not an actual RAM disk, we don't have to worry about allocating RAM to something that won't be used. If tmp is blank, no memory is used, as it's allocated on the fly. Additionally, this frees up the HD from having to handle the temp files. It's great! :D (You need TMPFS support in your kernel of course)

The Amigas had a RAM disk like this back in the 80s. Of course, back then, the GUI and programs ran on a machine with 1/2MB of memory (and still do). What are the requirements for KDE again? :lol:
_________________
Gentoo Linux * DirecTivo * Who needs heaven?
Back to top
View user's profile Send private message
Normie
Tux's lil' helper
Tux's lil' helper


Joined: 03 Jul 2002
Posts: 83

PostPosted: Wed Feb 19, 2003 4:35 am    Post subject: Reply with quote

I've got a slow Gentoo system (400 mhz p2, 320mb ram), and a fairly speedy XP system (1.4ghz athlon, 512mb ram). I originally wanted to install gentoo on the XP system and boot it with a grub floppy for some good ole' distccd whoring (;)), but for really stupid reason's I can't (and won't bother to) explain here, I can't repartition. So I figure I'll just use vmware when I need distccd assistance. Does anyone know how much my performance can expect to be cut (as compared to a "true" gentoo install)? 50%? 60%? What?

(edit) this is OT, I know, but if anyone knows a better way than vmware to install a barebones, made-for-distccd gentoo system without repartitioning, please PM me.(/edit)
_________________
Achaea. Play it. Love it. Live it.
Back to top
View user's profile Send private message
elpierco
n00b
n00b


Joined: 28 Sep 2003
Posts: 59
Location: Lawrence Kansas

PostPosted: Wed Aug 11, 2004 3:47 pm    Post subject: Problems using distcc to emerge system Reply with quote

I started emerging the system and I see using netstat and top u distcc that it is distributing work to my other machine(an Athlon XP). I am trying to build the system on a P4 2.8C...the emerge runs for a while then I get this

checking for i686-pc-linux-gnu-g++... gcc
checking whether we are using the GNU C++ compiler... no
checking whether gcc accepts -g... no
checking dependency style of gcc... none
checking how to run the C++ preprocessor... /lib/cpp
configure: error: C++ preprocessor "/lib/cpp" fails sanity check
See `config.log' for more details.
configure: error: /bin/sh './configure' failed for autoconf-lib-link

!!! ERROR: sys-devel/gettext-0.12.1 failed.
!!! Function econf, Line 362, Exitcode 1
!!! econf failed

Should I try to emerge that without distcc? sys-devel/gettext-0.12.1?

here is what my make.conf looks like

# These settings were set by the catalyst build script that automatically built$
# Please consult /etc/make.conf.example for a more detailed example
CFLAGS="-O3 -march=i686 -pipe -fomit-frame-pointer"
CHOST="i686-pc-linux-gnu"
CXXFLAGS="${CFLAGS}"
USE="-qt -kde"
FEATURES="distcc"
DISTCC_DIR="${PORTAGE_TMPDIR}/.distcc"
MAKEOPTS="-j5"
CC="distcc gcc"
CXX="distcc g++"
Back to top
View user's profile Send private message
elpierco
n00b
n00b


Joined: 28 Sep 2003
Posts: 59
Location: Lawrence Kansas

PostPosted: Thu Aug 12, 2004 4:01 pm    Post subject: Another try Reply with quote

So I am going to use the minimal install disk and try this one more time...

Just to have some confirmation on the machine (localhost) that will be building the stage 1 and distributing jobs to others. Does it matter if CC="distcc" is set on the other boxes?

Does it matter if this is on a seperate line or should it be included with MAKEOPTS i.e. MAKEOPTS="-j7 CC=distcc"

Does anyone have any ideas what that error might be? This is a brand new system components wise. I will be using a P4 2.8GHz and an Athlon 1800XP.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum