Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Save 30-50% off Chromium compile time with one simple trick!
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 6033

PostPosted: Tue Jan 02, 2018 2:19 pm    Post subject: Save 30-50% off Chromium compile time with one simple trick! Reply with quote

</clickbait>

It's real though!

The build system in newer versions of chromium has support for source file concatenation, like what sqlite3 does (or to a lesser extent, the USE=pch flag). It's not enabled by default, but you can try it out yourself by doing this:
/etc/portage/package.env:
www-client/chromium chromium.conf

/etc/portage/env/chromium.conf:
EXTRA_GN="use_jumbo_build=true jumbo_file_merge_limit=25"

Update: the chromium ebuilds have USE=jumbo-build now, so you can leave the first variable out (or omit the entire thing and use the defaults, if you're brave).

Feel free to increase the number if you have the RAM to handle it - 25 is just a safe number I picked because it worked on my 2GB netbook. The default is 100, which has some gcc processes eating 2.5GB or more.

The improvements can range from several hours to half a day or more -
Code:
$ qlop -vgH chromium
chromium-63.0.3239.59: Tue Nov 28 11:35:50 2017: 7 hours, 29 minutes, 8 seconds
chromium-64.0.3282.24: Fri Dec 22 04:12:14 2017: 5 hours, 57 minutes, 19 seconds

$ lscpu | grep P
CPU op-mode(s):      32-bit, 64-bit
CPU(s):              4
On-line CPU(s) list: 0-3
CPU family:          16
Model name:          AMD Phenom(tm) II X4 20 Processor
CPU MHz:             800.000
CPU max MHz:         2800.0000
CPU min MHz:         800.0000
BogoMIPS:            5625.36


Code:
$ qlop -vgH chromium
chromium-60.0.3112.24: Mon Jun 19 20:48:02 2017: 1 day, 2 hours, 8 minutes, 37 seconds
chromium-60.0.3112.24: Mon Jun 19 20:48:02 2017: 1 day, 2 hours, 8 minutes, 37 seconds
chromium-62.0.3202.18: Sat Sep 30 00:34:10 2017: 1 day, 2 hours, 50 minutes, 43 seconds
chromium-64.0.3282.24: Fri Dec 22 19:10:06 2017: 21 hours, 18 minutes, 46 seconds
# (not pictured: a regular compile of 64.0 that OOMed at the linker stage after 36 hours)

$ lscpu | grep P
CPU op-mode(s):      32-bit
CPU(s):              2
On-line CPU(s) list: 0,1
CPU family:          6
Model name:          Intel(R) Atom(TM) CPU N270   @ 1.60GHz
CPU MHz:             800.000
CPU max MHz:         1600.0000
CPU min MHz:         800.0000
BogoMIPS:            3193.13


See also: bug 641786


Last edited by Ant P. on Thu Mar 01, 2018 8:01 am; edited 1 time in total
Back to top
View user's profile Send private message
Juippisi
Developer
Developer


Joined: 30 Sep 2005
Posts: 378
Location: /home

PostPosted: Tue Jan 02, 2018 2:29 pm    Post subject: Reply with quote

Thanks for bringing this out in the open, I wouldnt have found that bug report on my own. This seems to work, and is a huge improvement in compiling Chromium.
Back to top
View user's profile Send private message
Marlo
Veteran
Veteran


Joined: 26 Jul 2003
Posts: 1428

PostPosted: Wed Jan 03, 2018 6:04 am    Post subject: Reply with quote

Yes it works.
At 100 it crashed but 75 gives:
Code:

 # genlop -t chromium
 * www-client/chromium

     Wed Dec 20 08:13:44 2017 >>> www-client/chromium-64.0.3282.24
       merge time: 4 hours, 17 minutes and 54 seconds.

     Mon Dec 25 02:38:39 2017 >>> www-client/chromium-64.0.3282.39
       merge time: 4 hours, 22 minutes and 55 seconds.

     Wed Jan  3 02:22:42 2018 >>> www-client/chromium-64.0.3282.39
       merge time: 1 hour, 59 minutes and 58 seconds.


Can you explain to me in simple words how this works?

Ma

EDIT
Is it possible to do that with other programs?
Or in other words. Can it be used as a standard for everything?
_________________
Thank you for your attention, interest and support.
------------------------------------------------------------------
http://radio.garden/
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 6033

PostPosted: Thu Jan 04, 2018 9:05 am    Post subject: Reply with quote

It concatenates 25/75/100 source files together at a time and compiles the result with a single GCC process. Apart from saving 99% of the overhead of starting up the compiler, it also means header files with multiple-include guards won't have to be processed over and over, and a lot more inter-function and dead code optimisation can be done at compile-time (the stuff usually reserved for -flto). Probably the latter thing is where most of the RAM usage comes from, but it also takes some off the final linking step.

It's hardcoded in chromium's build scripts, and is only possible because they already rewrote what autoconf/make does from scratch and control the entire compilation process. Making it work anywhere else would be pretty hard, definitely not something you can throw in make.conf and forget about.
Back to top
View user's profile Send private message
Marlo
Veteran
Veteran


Joined: 26 Jul 2003
Posts: 1428

PostPosted: Thu Jan 04, 2018 9:21 am    Post subject: Reply with quote

Thanks Ant. P. I wanted to know that.
This tip is a fantastic time saver.

Thanks again
Ma
_________________
Thank you for your attention, interest and support.
------------------------------------------------------------------
http://radio.garden/
Back to top
View user's profile Send private message
wxqc11h
n00b
n00b


Joined: 18 Sep 2007
Posts: 26

PostPosted: Tue Jan 23, 2018 11:32 am    Post subject: Reply with quote

Does it work for webkit-gtk too? Man merging it takes ages.
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 6033

PostPosted: Wed Jan 24, 2018 6:01 am    Post subject: Reply with quote

Nope, chromium-only. I don't recall Webkit-GTK having compile times as dire as this last time I had it installed though... was closer to firefox (minus rust) if anything.
Back to top
View user's profile Send private message
Princess Nell
l33t
l33t


Joined: 15 Apr 2005
Posts: 769

PostPosted: Sun Jan 28, 2018 9:09 pm    Post subject: Reply with quote

Off-topic here, but qlop must have serious bugs. Looking at emerge.log, genlop is quite correct.
Code:

# genlop -t webkit-gtk |tail -n 3
     Tue Jan 16 00:04:25 2018 >>> net-libs/webkit-gtk-2.18.5
       merge time: 2 hours, 40 minutes and 40 seconds.

# qlop -vgH webkit-gtk |tail -n 3
webkit-gtk-2.18.4: Thu Dec 28 22:31:26 2017: 2 hours, 39 minutes, 55 seconds
webkit-gtk-2.18.5: Sun Jan 14 18:22:37 2018: 1 day, 5 hours, 41 minutes, 48 seconds
webkit-gtk: 19 times
#
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 6033

PostPosted: Sun Jan 28, 2018 11:37 pm    Post subject: Reply with quote

Yeah, I noticed the same. It works if you give it a very narrow date range but seems to drop recent entries otherwise.
Back to top
View user's profile Send private message
Princess Nell
l33t
l33t


Joined: 15 Apr 2005
Posts: 769

PostPosted: Wed Jan 31, 2018 7:59 am    Post subject: Reply with quote

Impressive. jumbo_file_merge_limit=60 (100/90/80/70 crashed).
Code:

     Tue Jan 30 06:08:52 2018 >>> www-client/chromium-64.0.3282.123
       merge time: 8 hours, 20 minutes and 21 seconds.

     Wed Jan 31 03:18:11 2018 >>> www-client/chromium-64.0.3282.123
       merge time: 4 hours, 7 minutes and 32 seconds.
Back to top
View user's profile Send private message
walkingcorpse
n00b
n00b


Joined: 24 Apr 2017
Posts: 21
Location: Europe

PostPosted: Fri Feb 09, 2018 7:04 pm    Post subject: Reply with quote

jumbo_file_merge_limit=10 8GB ram and -j4. :?

Code:
     Sat Jan 20 20:30:37 2018 >>> www-client/chromium-65.0.3315.3
       merge time: 4 hours, 10 minutes and 39 seconds.

     Fri Feb  9 05:26:53 2018 >>> www-client/chromium-65.0.3315.3
       merge time: 2 hours, 5 minutes and 23 seconds


About the jumbo_file_merge_limit from the Chromium-dev google forum:

Quote:
Eventually we could have 10 or 20 for people with many cores, 50 for people with few cores, 100 or infinity for single core/full compilation.
Back to top
View user's profile Send private message
Section_8
Guru
Guru


Joined: 22 May 2004
Posts: 578
Location: Arlington, TX, US

PostPosted: Fri Feb 09, 2018 9:02 pm    Post subject: Reply with quote

I noticed the chromium ebuild has a jumbo-build use flag that must implement this:
Code:
john@officepc ~ $ grep jumbo `equery which chromium`
IUSE="component-build cups gnome-keyring +hangouts jumbo-build kerberos neon pic +proprietary-codecs pulseaudio selinux +suid +system-ffmpeg +system-icu +system-libvpx +tcmalloc widevine"
        # https://chromium.googlesource.com/chromium/src/+/lkcr/docs/jumbo.md
        myconf_gn+=" use_jumbo_build=$(usex jumbo-build true false)"

I wonder what file merge limit it uses.
Back to top
View user's profile Send private message
walkingcorpse
n00b
n00b


Joined: 24 Apr 2017
Posts: 21
Location: Europe

PostPosted: Fri Feb 09, 2018 9:12 pm    Post subject: Reply with quote

It should be 200:
Quote:
By default at most 200 files are merged at a time. The more files are merged, the less total CPU time is needed, but parallelism is reduced. This can be changed by setting jumbo_file_merge_limit to something else than 200.

https://chromium.googlesource.com/chromium/src/+/master/docs/jumbo.md#tuning
Back to top
View user's profile Send private message
ceric35
Tux's lil' helper
Tux's lil' helper


Joined: 27 Aug 2006
Posts: 115

PostPosted: Thu Nov 14, 2019 11:02 am    Post subject: Reply with quote

walkingcorpse wrote:
https://chromium.googlesource.com/chromium/src/+/master/docs/jumbo.md#tuning


This links now says "Jumbo / Unity builds are no longer supported in Chromium."

EDIT : use has been removed from chromium-80
Back to top
View user's profile Send private message
Chiitoo
Administrator
Administrator


Joined: 28 Feb 2010
Posts: 1886
Location: Here and Away Again

PostPosted: Thu Nov 14, 2019 1:34 pm    Post subject: Reply with quote

ceric35 wrote:
walkingcorpse wrote:
https://chromium.googlesource.com/chromium/src/+/master/docs/jumbo.md#tuning


This links now says "Jumbo / Unity builds are no longer supported in Chromium."

EDIT : use has been removed from chromium-80

Yep.

- https://chromium.googlesource.com/chromium/src/+/6e1ce9f4

chromium/src/6e1ce9f4 wrote:
Force jumbo builds to fail.

We no longer have any builders supporting the jumbo build config,
and it's time to now update all of the build files to remove
the jumbo templates. Before we do that, though, it's probably good
to figure out how to handle existing uses of the jumbo builds.

This CL asserts that |use_jumbo_build| is set to false. Once
we've run with this for a while (to ensure that people have
adjusted their configs), we can probably update any CI build
configs to remove the use of it and the other jumbo-related GN
args and then remove the actual declare_args() of the three
jumbo args, turning uses of them into warnings.

Also

- https://groups.google.com/a/chromium.org/forum/#!topic/chromium-dev/DP9TQszzQLI/discussion

Dirk wrote:
Hi all,

We have decided to remove support for the jumbo (aka unity) build configuration from the project.

We are very conscious of the fact that Chromium is very expensive and slow to compile, and that that represents a real barrier to contributing.

However, based on the experience we've gained with this configuration over the past many months it's been supported, we've reached the conclusion that the way jumbo is currently implemented, it interferes too much with the way people frequently write C++ code; you have to work around jumbo-specific conflicts in a way that can be awkward, unnatural, or otherwise unnecessary. As a result, we feel that the cost to the project of maintaining jumbo is not worth the benefit it is providing.

We continue to invest in other approaches to reducing build time. We've released the Goma build accelerator code and hope to continue making it easier to use for others. We are also working to enable our Goma clusters for external contributors (anyone who has tryjob access) who don't have their own build acceleration tools. We're starting with Linux support now and will have more to share on that soon. If you are interested in getting access to this system, please fill out this form. We're also working on Clang and LLVM and have made significant improvements this year to cycle time as a result, and we expect to continue to do so for the foreseeable future.

We plan to remove support a week from now, on August 22nd.

Please let me know if you have questions about this.

-- Dirk

_________________
Kind regards,
Chiitoo.

You might remember me from Gentoo projects such as Forums, LXQt, Qt, and Wine.
Back to top
View user's profile Send private message
Goverp
l33t
l33t


Joined: 07 Mar 2007
Posts: 729

PostPosted: Fri Nov 15, 2019 8:58 am    Post subject: Reply with quote

ceric35 wrote:

...
This links now says "Jumbo / Unity builds are no longer supported in Chromium."

EDIT : use has been removed from chromium-80

Arrggghhhh! I don't run chromium for many reasons, but KDE uses qtwebengine in several places. Since qtwebengine is based on chromium, presumably it too will stop supporting jumbo build, and I'll be stuck with a 24hr or longer emerge. :-(

Pretty please, can someone start a qtwebengine-bin package as an alternative?
_________________
Greybeard
Back to top
View user's profile Send private message
Chiitoo
Administrator
Administrator


Joined: 28 Feb 2010
Posts: 1886
Location: Here and Away Again

PostPosted: Fri Nov 15, 2019 9:23 am    Post subject: Reply with quote

Goverp wrote:
Pretty please, can someone start a qtwebengine-bin package as an alternative?

I would, if it were a simple thing to do. Or at least not as much work as it likely will be.

See also: Bug 669082 - dev-qt/qtwebengine-bin package
_________________
Kind regards,
Chiitoo.

You might remember me from Gentoo projects such as Forums, LXQt, Qt, and Wine.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum