Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[Solved] Portage with distcc and cross-compiling
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Installing Gentoo
View previous topic :: View next topic  
Author Message
solamour
l33t
l33t


Joined: 21 Dec 2004
Posts: 637
Location: San Diego, CA

PostPosted: Tue Jan 16, 2018 6:31 am    Post subject: [Solved] Portage with distcc and cross-compiling Reply with quote

[Moderator note: this post, and the first 3 responses, were originally attached to the extremely long dead thread Compile on fast machine, run on slow machine?, last active October 2003. This thread has similar subject matter, but can stand on its own. -Hu]

Sorry to wake up a really, really old thread.

My slow machine is 32-bit with CHOST="i486-pc-linux-gnu", and the fast machine is 64-bit with CHOST="x86_64-pc-linux-gnu". When I ran distcc, the slow one complained that it couldn't link 64-bit objects. Do I need to change the fast one's CHOST to "i486-pc-linux-gnu" and rebuild gcc? If so, do I need to change the fast one's CHOST back to "x86_64-pc-linux-gnu" when I update the fast one itself?
__
sol


Last edited by solamour on Tue Jun 19, 2018 12:10 am; edited 2 times in total
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 7071

PostPosted: Tue Jan 16, 2018 10:28 am    Post subject: Reply with quote

You need to cross compile.
As you use distcc refers to https://wiki.gentoo.org/wiki/Distcc/Cross-Compiling
But you should better not use distcc, and build binaries for the i486 host and provide them just ready made. See https://wiki.gentoo.org/wiki/Embedded_Handbook/General/Creating_a_cross-compiler and https://wiki.gentoo.org/wiki/Binary_package_guide
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43178
Location: 56N 3W

PostPosted: Tue Jan 16, 2018 12:48 pm    Post subject: Reply with quote

solamour,

What profile do you use on the 64 bit system?
What profile do you have on the 486?

What CFLAGS do you have on the 486?
You may need to use cross distcc, you may only need to fix the CFLAGS on the 486.

-- edit --

For something as old as a 486, I would create a 486 chroot on the 64 bit box and make a binary package server there.
Then have the 486 use the prebuilt binaries.

Heh. krinn said that already.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
solamour
l33t
l33t


Joined: 21 Dec 2004
Posts: 637
Location: San Diego, CA

PostPosted: Tue Jan 16, 2018 7:49 pm    Post subject: Reply with quote

The profile and CFLAGS for the slow the the fast are as the following.

Code:

[SLOW]
default/linux/x86/17.0

CHOST="i486-pc-linux-gnu"
CFLAGS="-march=geode -Os -mmmx -m3dnow -fomit-frame-pointer"
CXXFLAGS="${CFLAGS}"
CPU_FLAGS_X86="3dnow 3dnowext mmx mmxext"
MAKEOPTS="-j1"
USE=""


[FAST]
default/linux/amd64/17.0

CHOST="x86_64-pc-linux-gnu"
CFLAGS="-march=native -O2 -pipe"
CXXFLAGS="${CFLAGS}"
CPU_FLAGS_X86="aes avx mmx mmxext pclmul popcnt \
               sse sse2 sse3 sse4_1 sse4_2 ssse3"
MAKEOPTS="-j5"
USE="-cups"


The slow machine hasn't been a problem, but lately, I'm seeing build failure on large packages, such as llvm or gcc, due to running out of memory. I'll explore chroot environment and binary package.
__
sol


Last edited by solamour on Tue Jun 05, 2018 10:11 pm; edited 1 time in total
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 13830

PostPosted: Wed Jan 17, 2018 3:26 am    Post subject: Re: Compile on fast machine, run on slow machine? Reply with quote

solamour wrote:
My slow machine is 32-bit with CHOST="i486-pc-linux-gnu", and the fast machine is 64-bit with CHOST="x86_64-pc-linux-gnu". When I ran distcc, the slow one complained that it couldn't link 64-bit objects. Do I need to change the fast one's CHOST to "i486-pc-linux-gnu" and rebuild gcc? If so, do I need to change the fast one's CHOST back to "x86_64-pc-linux-gnu" when I update the fast one itself?
What exactly did you run that produced the complaint? Distributing a bare name gcc between different architectures is likely to end badly (as it did for you). Distributing a fully qualified name host-tuple-gcc-version should be fine. The chroot options discussed later in the thread are likely to be easier to maintain.
Back to top
View user's profile Send private message
solamour
l33t
l33t


Joined: 21 Dec 2004
Posts: 637
Location: San Diego, CA

PostPosted: Thu Jan 18, 2018 12:57 am    Post subject: Re: Compile on fast machine, run on slow machine? Reply with quote

Hu wrote:
What exactly did you run that produced the complaint? Distributing a bare name gcc between different architectures is likely to end badly (as it did for you). Distributing a fully qualified name host-tuple-gcc-version should be fine. The chroot options discussed later in the thread are likely to be easier to maintain.


I setup the systems as described in https://wiki.gentoo.org/wiki/Distcc, except that I left out the slow one from the available pool. I prefer not messing up the fast one if possible at all, so chroot seems like a good choice for me.

That said, I got the impression that if I setup the fast one as a cross-compiler (for 486), I need to revert everything back to the way it was when I want to update the fast one itself. If that is so, I'd say chroot it shall be.
__
sol
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43178
Location: 56N 3W

PostPosted: Thu Jan 18, 2018 11:48 am    Post subject: Reply with quote

solamour,

If you set up your cross compiler with
Code:
crossdev -t i486-pc-linux-gnu
, you get a complete target root in /usr/i486-pc-linux-gnu/ with its own /etc/portage/* which applies when you use cross emerge.
However, lots of things don't cross compile cleanly and gcc is one of the challenges.

If your 32 bit distcc client is getting 64 bit code back to link, there is something wrong with the CFLAGS its sending to helpers.

What is
Code:
emerge --info
from the 486?

What does gcc-config -l show on both systems?
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
solamour
l33t
l33t


Joined: 21 Dec 2004
Posts: 637
Location: San Diego, CA

PostPosted: Thu Jan 18, 2018 9:41 pm    Post subject: Reply with quote

NeddySeagoon wrote:
What is
Code:
emerge --info
from the 486?

What does gcc-config -l show on both systems?


As the following.
Code:
[slow]
# gcc-config -l
 [1] i486-pc-linux-gnu-6.4.0 *

[fast]
# gcc-config -l
 [1] x86_64-pc-linux-gnu-6.4.0 *


Code:
[slow]
# emerge --info
Portage 2.3.13 (python 2.7.14-final-0, default/linux/x86/17.0, gcc-6.4.0, glibc-2.25-r9, 4.9.34-gentoo i586)
=================================================================
System uname: Linux-4.9.34-gentoo-i586-Geode-TM-_Integrated_Processor_by_AMD_PCS-with-gentoo-2.4.1
KiB Mem:      229900 total,     21164 free
KiB Swap:    4193852 total,   4157268 free
Timestamp of repository gentoo: Wed, 17 Jan 2018 11:00:01 +0000
Head commit of repository gentoo: b569e1d7da8d6194f585bb90657c26b72e5cff13
sh bash 4.4_p12
ld GNU ld (Gentoo 2.29.1 p3) 2.29.1
distcc 3.2rc1 i486-pc-linux-gnu [enabled]
app-shells/bash:          4.4_p12::gentoo
dev-java/java-config:     2.2.0-r3::gentoo
dev-lang/perl:            5.24.3::gentoo
dev-lang/python:          2.7.14-r1::gentoo, 3.5.4-r1::gentoo, 3.6.3-r1::gentoo
dev-util/cmake:           3.9.6::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.4.1-r2::gentoo
sys-apps/openrc:          0.34.11::gentoo
sys-apps/sandbox:         2.12::gentoo
sys-devel/autoconf:       2.69-r4::gentoo
sys-devel/automake:       1.15.1-r1::gentoo
sys-devel/binutils:       2.29.1-r1::gentoo
sys-devel/gcc:            6.4.0::gentoo
sys-devel/gcc-config:     1.8-r1::gentoo
sys-devel/libtool:        2.4.6-r3::gentoo
sys-devel/make:           4.2.1::gentoo
sys-kernel/linux-headers: 4.13::gentoo (virtual/os-headers)
sys-libs/glibc:           2.25-r9::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000
    sync-rsync-extra-opts:

ACCEPT_KEYWORDS="x86"
ACCEPT_LICENSE="* -@EULA"
CBUILD="i486-pc-linux-gnu"
CFLAGS="-march=geode -Os -mmmx -m3dnow -fomit-frame-pointer"
CHOST="i486-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/easy-rsa /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=geode -Os -mmmx -m3dnow -fomit-frame-pointer"
DISTDIR="/usr/portage/distfiles"
FCFLAGS="-O2 -march=i686 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distcc distcc-pump distlocks ebuild-locks fixlafiles merge-sync multilib-strict news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -march=i686 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LANG="en_US.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j5 -l1"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="acl berkdb bzip2 cli crypt cxx dri fortran gdbm iconv ipv6 modules ncurses nls nptl openmp pam pcre readline seccomp ssl tcpd unicode x86 xattr zlib" ABI_X86="32" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1 emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="3dnow 3dnowext mmx mmxext" CURL_SSL="gnutls" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6 php7-0" POSTGRES_TARGETS="postgres9_5" PYTHON_SINGLE_TARGET="python3_5" PYTHON_TARGETS="python2_7 python3_5" RUBY_TARGETS="ruby22" USERLAND="GNU" VIDEO_CARDS="geode" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS


I think I might have missed a step (or two) when setting up cross compile, because I don't see "/usr/i486-pc-linux-gnu/" in the fast. Let me first check everything is where it's supposed to be.
__
sol
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43178
Location: 56N 3W

PostPosted: Thu Jan 18, 2018 9:54 pm    Post subject: Reply with quote

solamour,

Its a geode, not a i486. It might even be a i686. Please post /proc/cpuinfo from the slow system.

When you install cross compilers with crossdev, gcc-config lists them like
Code:
 [5] aarch64-unknown-linux-gnu-7.2.0 *

 [6] aarch64-unknown-linux-musl-6.4.0 *
 [7] aarch64-unknown-linux-musl-7.2.0

 [8] aarch64-unknown-linux-musleabi-7.2.0 *

You can use cross emerge to emerge into the target root and build binary packages or use cross distcc.
Both ways have their advantages.

If
Code:
[fast]
# gcc-config -l
 [1] x86_64-pc-linux-gnu-6.4.0 *
is using a multilib install, a cross toolchain is not required as your toolchain can produce both 32 bit and 64 bit code.

Passing
Code:
 CFLAGS="-march=geode -Os -mmmx -m3dnow -fomit-frame-pointer"
through distcc would be good enough as geode implies -m32
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
solamour
l33t
l33t


Joined: 21 Dec 2004
Posts: 637
Location: San Diego, CA

PostPosted: Fri Jan 26, 2018 1:36 am    Post subject: Reply with quote

Code:
/proc/cpuinfo

processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 5
model           : 10
model name      : Geode(TM) Integrated Processor by AMD PCS
stepping        : 2
cpu MHz         : 499.925
cache size      : 128 KB
fdiv_bug        : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu de pse tsc msr cx8 sep pge cmov clflush mmx mmxext 3dnowext 3dnow eagerfpu 3dnowprefetch vmmcall
bugs            : sysret_ss_attrs
bogomips        : 999.85
clflush size    : 32
cache_alignment : 32
address sizes   : 32 bits physical, 32 bits virtual
power management:


It's been a while, so I'm not sure why I thought it was i486, but most likely it's based on the following.

https://wiki.gentoo.org/wiki/Safe_CFLAGS
https://forums.gentoo.org/viewtopic-t-485271.html
https://www.twam.info/hardware/chost-i586-vs-i486-on-amd-geode-lx

Perhaps I should try a different CHOST, but frankly, I'm slightly reluctant to do so, because it might take a few weeks to build the important system tools with the new CHOST.
__
sol
Back to top
View user's profile Send private message
solamour
l33t
l33t


Joined: 21 Dec 2004
Posts: 637
Location: San Diego, CA

PostPosted: Thu Jun 14, 2018 9:51 pm    Post subject: Reply with quote

After much distraction by other tasks that had higher priorities, I finally got around setting up distcc with cross-compiling, so that I can start a build on a slow machine (x86) and send all the jobs to the fast machine (x64). The slow machine still does preprocessing and such, but at least compiling a non-trivial package doesn't take a few hours (or, more often, days).

Now the only remaining issue is, although I can build, say, vim from source code I manually downloaded from (https://github.com/vim/vim) and the final result runs on the slow machine just fine, portage (e.g. "emerge -v vim") always stays in the slow machine; the job is never sent to the fast machine. Do I need to issue a special command to distribute jobs?

Code:

[Settings in Slow Machine]


/etc/portage/make.conf
...
MAKEOPTS="-j6"
FEATURES="distcc"


# distcc-config --get-hosts
FAST_MACHINE_IP/5 localhost/1


# echo $PATH
/usr/i486-pc-linux-gnu/gcc-bin/6.4.0:/usr/lib/llvm/4/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin

__
sol


Last edited by solamour on Fri Jun 15, 2018 9:19 pm; edited 1 time in total
Back to top
View user's profile Send private message
gordonb3
Apprentice
Apprentice


Joined: 01 Jul 2015
Posts: 185

PostPosted: Fri Jun 15, 2018 5:59 pm    Post subject: Reply with quote

Nope. As the name already suggests, distcc distributes (g)cc jobs (and g++) to one or more other machines. Portage is based on python, so that will not be distributed.
Back to top
View user's profile Send private message
solamour
l33t
l33t


Joined: 21 Dec 2004
Posts: 637
Location: San Diego, CA

PostPosted: Fri Jun 15, 2018 9:23 pm    Post subject: Reply with quote

gordonb3 wrote:
Nope. As the name already suggests, distcc distributes (g)cc jobs (and g++) to one or more other machines. Portage is based on python, so that will not be distributed.


Then, am I reading the following incorrectly? The instruction does sound like portage can use distcc when setup properly.

https://wiki.gentoo.org/wiki/Distcc#With_Portage
__
sol


Last edited by solamour on Sat Jun 16, 2018 5:45 am; edited 1 time in total
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 13830

PostPosted: Sat Jun 16, 2018 1:16 am    Post subject: Reply with quote

Yes, Portage can cause C and C++ compilation jobs to be run remotely rather than locally. Portage's own code (dependency resolution, mainly) would be nice to distribute, but is completely outside the scope of distcc. I think gordonb3 thought you wanted all CPU heavy tasks distributed, regardless of the program that needs the CPU time.
Back to top
View user's profile Send private message
solamour
l33t
l33t


Joined: 21 Dec 2004
Posts: 637
Location: San Diego, CA

PostPosted: Sat Jun 16, 2018 5:54 am    Post subject: Reply with quote

Hu wrote:
Yes, Portage can cause C and C++ compilation jobs to be run remotely rather than locally. Portage's own code (dependency resolution, mainly) would be nice to distribute, but is completely outside the scope of distcc. I think gordonb3 thought you wanted all CPU heavy tasks distributed, regardless of the program that needs the CPU time.


I don't mind portage itself running on the slow machine, as long as C/C++ compilation is done on the fast machine. I think I followed the directions correctly, but portage is not sending the tasks to the fast machine. Other than "/etc/portage/make.conf" and "/etc/distcc/hosts", what else should I double-check?
__
sol
Back to top
View user's profile Send private message
gordonb3
Apprentice
Apprentice


Joined: 01 Jul 2015
Posts: 185

PostPosted: Sat Jun 16, 2018 7:00 am    Post subject: Reply with quote

Ah. Okay. Misinterpretation on my side. Since you want to helper machine to do cross-compiling, did you also create the wrapper script as described HERE?

As a personal note I find that building on the slow machine itself even with distcc never gets really quick. You get better results if you cross-build binary packages (FEATURES=buildpkg) on the fast machine. Shamefully this doesn't work with every package. e.g. a package can't do something like bootstrapping if the target code can't run on the host architecture. Package installers also often (or always) appear to ignore the ROOT environment setting and you end up with incorrect paths like 32 bit libraries in lib64
Back to top
View user's profile Send private message
solamour
l33t
l33t


Joined: 21 Dec 2004
Posts: 637
Location: San Diego, CA

PostPosted: Tue Jun 19, 2018 12:10 am    Post subject: Reply with quote

I finally got distcc and portage work correctly. Perhaps all the documentations are crystal clear to everyone else, but it certainly wasn't for me, so I'll summarize what I've found so far, as a future reference.

In short, I'd like to give all build jobs to the fast machine, because, well, the slow machine is just too slow.

* If, for whatever reasons, FAST (i.e. helper machine) is not available, all the jobs will remain in SLOW, so given enough time, I'd have the exactly same outcome; I just need to wait more. Way more.

* Even when FAST is performing all build jobs, SLOW still needs to do a lot of preprocessing and linking. In other words, simply using the "rule of thumb" for "J" value (i.e. "total cores * 2 + 1") might not be a good idea if SLOW is pitifully under-powered, because it would be too busy preparing the build jobs. Finding the optimal value is more of a trial-and-error process.

* When monitoring the status of SLOW, I need to set DISTCC_DIR.
Code:

DISTCC_DIR="/var/tmp/portage/.distcc/" distccmon-text 1


* The default port for distcc is 3632, so if there is a firewall, it needs to "allow outgoing traffic of 3632 to FAST".

* If SLOW is not providing distcc service to other machines, it doesn't need to start distccd.

* Pump never worked (at least for me); all jobs stayed in SLOW, and I haven't figured out why.
Code:

distcc[26754] (dcc_build_somewhere) Warning: remote compilation of 'digraph.c' failed, retrying locally
distcc[26754] Warning: failed to distribute digraph.c to fast/6,cpp,lzo, running locally instead

distcc[26390] (dcc_please_send_email_after_investigation) Warning: remote compilation of 'arabic.c' failed, retried locally and got a different result.
distcc[26390] (dcc_note_discrepancy) Warning: now using plain distcc, possibly due to inconsistent file system changes during build


* If CHOST is different between SLOW and FAST, it's considered as cross-compile, even if they are x86 and x64.

* After everything is said and done, in practice, it might make more sense to just build the packages in FAST and move them to SLOW, so that SLOW doesn't need to do any compiling.

* Speaking of building packages, if, let's say, FAST is x64 and SLOW is x86, simply using quickpkg on FAST's cross compiler won't work, because although FAST's cross compiler can generate the code for SLOW, the cross compiler itself is designed for FAST; it won't run on SLOW. Creating a chroot environment on FAST and using quickpkg inside the chroot is a better approach.

Thanks everyone for taking time to share your suggestions.

[Useful Links]
https://wiki.gentoo.org/wiki/Distcc
https://wiki.gentoo.org/wiki/Distcc/Cross-Compiling
https://wiki.gentoo.org/wiki/Raspberry_Pi/Cross_building#Raspberry_Pi_only
https://wiki.gentoo.org/wiki/Cross_build_environment
https://wiki.gentoo.org/wiki/Custom_repository#Crossdev
__
sol
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Installing Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum