Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
ccache 3.0pre0
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
Shining Arcanine
Veteran
Veteran


Joined: 24 Sep 2009
Posts: 1110

PostPosted: Wed Mar 17, 2010 7:23 pm    Post subject: ccache 3.0pre0 Reply with quote

A pre-release version of ccache 3.0 is avaiable:

http://ccache.samba.org/news.html#2010-02-28

The site's unscientific benchmarks show a factor of 10 improvement over ccache 2.4. Has anyone tried installing this on Gentoo to use it with portage?
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6296

PostPosted: Wed Mar 17, 2010 8:17 pm    Post subject: Reply with quote

I haven't tried it, but this "direct mode" feature will render ccache almost useless for things like gentoo. It seems that portage should need to disable it by default if ccache should get a revbump in the tree.
Back to top
View user's profile Send private message
Shining Arcanine
Veteran
Veteran


Joined: 24 Sep 2009
Posts: 1110

PostPosted: Wed Mar 17, 2010 9:49 pm    Post subject: Reply with quote

mv wrote:
I haven't tried it, but this "direct mode" feature will render ccache almost useless for things like gentoo. It seems that portage should need to disable it by default if ccache should get a revbump in the tree.


Well, it does have the CCACHE_NODIRECT variable that can be set to disable this, but I am not sure why this would render ccache almost useless for Gentoo. Would you elaborate on your reasoning?
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6296

PostPosted: Thu Mar 18, 2010 6:50 am    Post subject: Reply with quote

Shining Arcanine wrote:
]Well, it does have the CCACHE_NODIRECT variable that can be set to disable this

That's why I suggest that portage should set a corresponding setting. (Maybe also the behavior can be inverted by some ./configure option? This would probably be even more appropriate for gentoo).
Quote:
but I am not sure why this would render ccache almost useless for Gentoo.

ccache is meant to be used by developers which recompile one project several times. In such a situation, it occurs rarely that system-wide header files have essential changes. And even if they have, the developer will realize this quickly. For such a purpose the direct mode is useful. However, in gentoo most users have a rather large ccache directory, and it happens frequently that packages are updated after quite a while and then often some of the (unchanged) sources files of the previous version are still in cache. In this situation it is rather likely that some of the used libraries (and with it its header files or indirect header files) have been updated. Without running a preprocessor, ccache will not realize this change and happily "compile" the package, producing .o files with obsolete headers. This can lead to all sorts of problems during runtime of the package which may be very subtle and, if they occur, the user will have forgotten (or might perhaps even not know) that ccache was the cause.
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 2870
Location: Bay Area, CA

PostPosted: Thu Mar 18, 2010 9:30 pm    Post subject: Reply with quote

mv wrote:
Shining Arcanine wrote:
]Well, it does have the CCACHE_NODIRECT variable that can be set to disable this

That's why I suggest that portage should set a corresponding setting. (Maybe also the behavior can be inverted by some ./configure option? This would probably be even more appropriate for gentoo).
Quote:
but I am not sure why this would render ccache almost useless for Gentoo.

ccache is meant to be used by developers which recompile one project several times. In such a situation, it occurs rarely that system-wide header files have essential changes. And even if they have, the developer will realize this quickly. For such a purpose the direct mode is useful. However, in gentoo most users have a rather large ccache directory, and it happens frequently that packages are updated after quite a while and then often some of the (unchanged) sources files of the previous version are still in cache. In this situation it is rather likely that some of the used libraries (and with it its header files or indirect header files) have been updated. Without running a preprocessor, ccache will not realize this change and happily "compile" the package, producing .o files with obsolete headers. This can lead to all sorts of problems during runtime of the package which may be very subtle and, if they occur, the user will have forgotten (or might perhaps even not know) that ccache was the cause.
Is that what they have done with ccache 3.0? WOW! Talk about correctness and reliability!

Ccache is interesting with SSDs because good random IO and negligible IO latencies make it attractive for older processors. But if this direct mode (as described above) is default, we are in lot of trouble and will need to switch it off.
Back to top
View user's profile Send private message
Bill Cosby
Guru
Guru


Joined: 22 Jan 2005
Posts: 430
Location: Aachen, Germany

PostPosted: Tue Mar 30, 2010 8:24 am    Post subject: Reply with quote

mv wrote:
It seems that portage should need to disable it by default if ccache should get a revbump in the tree.

Isn't this the case already? I was thinking about enabling it, that's how I found this thread, however, it seems its pros don't outweigh the cons, I prefer running a simple system in which compile times aren't that time consuming anyways.
_________________
The Creature from Jekyll Island.
Back to top
View user's profile Send private message
yoshi314
l33t
l33t


Joined: 30 Dec 2004
Posts: 848
Location: PL

PostPosted: Tue Mar 30, 2010 9:11 am    Post subject: Reply with quote

aside from the aforementioned issue - support for compressed cache. now that is sweet.

i just can't wait to test this.
_________________
~amd64
shrink your /usr/portage with squashfs+aufs
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6296

PostPosted: Tue Mar 30, 2010 4:59 pm    Post subject: Reply with quote

Bill Cosby wrote:
mv wrote:
It seems that portage should need to disable it by default if ccache should get a revbump in the tree.

Isn't this the case already?

I just synced, and the newest version is dev-util/ccache-2.4-r8. Maybe you are using some overlay.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6296

PostPosted: Tue Mar 30, 2010 5:00 pm    Post subject: Reply with quote

yoshi314 wrote:
aside from the aforementioned issue - support for compressed cache.

Yes, I didn't say it is bad. It just needs CCACHE_NODIRECT=1 in make.globals (or at least in your personal /etc/make.conf).
Back to top
View user's profile Send private message
yoshi314
l33t
l33t


Joined: 30 Dec 2004
Posts: 848
Location: PL

PostPosted: Wed Mar 31, 2010 3:33 pm    Post subject: Reply with quote

yeah that's what i'm using.

i've hand compiled current git, and it looks really promising disk-wise :

Code:
cache directory                     /mnt/storage/ccache
cache hit (direct)                     0
cache hit (preprocessed)           11883
cache miss                         42107
called for link                     3488
multiple source files                 15
compile failed                      1388
ccache internal error                  1
preprocessor error                   589
not a C/C++ file                    1681
autoconf compile/link               7074
unsupported compiler option          119
no input file                       2789
files in cache                      2803
cache size                          17.7 Mbytes
max cache size                     976.6 Mbytes


~3K files in 18mb, not bad. it would take much more on 2.4
_________________
~amd64
shrink your /usr/portage with squashfs+aufs
Back to top
View user's profile Send private message
rhill
Retired Dev
Retired Dev


Joined: 22 Oct 2004
Posts: 1629
Location: sk.ca

PostPosted: Sat Apr 03, 2010 8:40 am    Post subject: Reply with quote

mv wrote:
However, in gentoo most users have a rather large ccache directory, and it happens frequently that packages are updated after quite a while and then often some of the (unchanged) sources files of the previous version are still in cache. In this situation it is rather likely that some of the used libraries (and with it its header files or indirect header files) have been updated. Without running a preprocessor, ccache will not realize this change and happily "compile" the package, producing .o files with obsolete headers.


IIUC, in direct mode a manifest is created containing the hashes of all the header files the preprocessor included. These are compared against the hashes and mtimes of the current headers.

I'm really looking forward to the absolute -> relative pathname mangling. This means that revbumps won't invalidate the cache due to the version number in the path changing anymore*. Also nice is that comments are now ignored when hashing input files.


* (you need to set CCACHE_BASEDIR to /var/tmp/portage for this to work)
_________________
by design, by neglect
for a fact or just for effect
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6296

PostPosted: Sat Apr 03, 2010 9:38 am    Post subject: Reply with quote

dirtyepic wrote:
IIUC, in direct mode a manifest is created containing the hashes of all the header files the preprocessor included.

You are right: I relied only on the webpage information and there it appeared different to me. :oops:
So my guess was wrong, ccache-3.* does indeed still hash the include files in direct mode. I am very happy that I can withdraw my critics. :)
Quote:
I'm really looking forward to the absolute -> relative pathname mangling. This means that revbumps won't invalidate the cache due to the version number in the path changing anymore*.

What do you mean by "anymore"? AFAIK, in ccache-2.4 there was not such a problem since the output of full the preprocessor was hashed, and why should that contain some paths? To what I understand this should concern only the "direct mode" which needs to store the .h files individually.
Back to top
View user's profile Send private message
rhill
Retired Dev
Retired Dev


Joined: 22 Oct 2004
Posts: 1629
Location: sk.ca

PostPosted: Sat Apr 03, 2010 11:04 am    Post subject: Reply with quote

mv wrote:
Quote:
I'm really looking forward to the absolute -> relative pathname mangling. This means that revbumps won't invalidate the cache due to the version number in the path changing anymore*.

What do you mean by "anymore"? AFAIK, in ccache-2.4 there was not such a problem since the output of full the preprocessor was hashed, and why should that contain some paths? To what I understand this should concern only the "direct mode" which needs to store the .h files individually.


ccache also hashes the command line passed to the compiler. this is how it keeps from serving up old object files if the optimization flags change, among other things. if that commandline contains absolute paths, the change in the version number creates a different hash, causing a cache miss.

you can see how a bump to -r6 would cause a hash of this commandline to change:

Code:
x86_64-unknown-linux-gnu-gcc -c -o wxregex_regcomp.o -D__WXGTK__  -fPIC -DPIC -D_FILE_OFFSET_BITS=64 -D_LARGE_FILES -I/var/tmp/portage/x11-libs/wxGTK-2.8.10.1-r5/work/wxPython-src-2.8.10.1/wxgtk_build/lib/wx/include/gtk2-unicode-release-2.8 -I/var/tmp/portage/x11-libs/wxGTK-2.8.10.1-r5/work/wxPython-src-2.8.10.1/include -pthread -I/usr/include/gtk-2.0 -I/usr/lib64/gtk-2.0/include -I/usr/include/atk-1.0 -I/usr/include/cairo -I/usr/include/pango-1.0 -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/include/pixman-1 -I/usr/include/freetype2 -I/usr/include/libpng12 -pthread -DORBIT2=1 -I/usr/include/gstreamer-0.10 -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/include/libxml2 -I/usr/include/gconf/2 -I/usr/include/orbit-2.0 -I/usr/include/dbus-1.0 -I/usr/lib64/dbus-1.0/include -pthread -Wall -Wundef -O2 -fno-strict-aliasing -O2 -march=native -g -fomit-frame-pointer -pipe -fno-strict-aliasing /var/tmp/portage/x11-libs/wxGTK-2.8.10.1-r5/work/wxPython-src-2.8.10.1/src/regex/regcomp.c


with relative path rewriting this would become something like:

Code:
x86_64-unknown-linux-gnu-gcc -c -o wxregex_regcomp.o -D__WXGTK__  -fPIC -DPIC -D_FILE_OFFSET_BITS=64 -D_LARGE_FILES -I../wxgtk_build/lib/wx/include/gtk2-unicode-release-2.8 -I../include -pthread -I/usr/include/gtk-2.0 -I/usr/lib64/gtk-2.0/include -I/usr/include/atk-1.0 -I/usr/include/cairo -I/usr/include/pango-1.0 -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/include/pixman-1 -I/usr/include/freetype2 -I/usr/include/libpng12 -pthread -DORBIT2=1 -I/usr/include/gstreamer-0.10 -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include -I/usr/include/libxml2 -I/usr/include/gconf/2 -I/usr/include/orbit-2.0 -I/usr/include/dbus-1.0 -I/usr/lib64/dbus-1.0/include -pthread -Wall -Wundef -O2 -fno-strict-aliasing -O2 -march=native -g -fomit-frame-pointer -pipe -fno-strict-aliasing ../src/regex/regcomp.c

_________________
by design, by neglect
for a fact or just for effect
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6296

PostPosted: Sat Apr 03, 2010 11:32 am    Post subject: Reply with quote

dirtyepic wrote:
ccache also hashes the command line passed to the compiler. this is how it keeps from serving up old object files if the optimization flags change, among other things. if that commandline contains absolute paths, the change in the version number creates a different hash, causing a cache miss.

The documentation is really unclear:
man ccache wrote:
CCACHE_BASEDIR
When hashing the output from the preprocessor, absolute paths are rewritten to relative paths, but
only for paths under the directory specified by CCACHE_BASEDIR. Paths specified by -I and similar
options get the same treatment. This is done to get cache hits even when compiling with -g and when
using absolute include directory paths. The default value of CCACHE_BASEDIR is the current working
directory.

I would interpret that this involves really only paths to header files not to C source files as in your example. Anyway, I am not sure how many build systems use absolute paths; usually there is no need for it with autotools project.

For those interested in testing: There is now an experimental ebuild in the mv overlay (should be available with layman after possibly layman -f).
Back to top
View user's profile Send private message
rhill
Retired Dev
Retired Dev


Joined: 22 Oct 2004
Posts: 1629
Location: sk.ca

PostPosted: Sat Apr 03, 2010 8:51 pm    Post subject: Reply with quote

mv wrote:
dirtyepic wrote:
ccache also hashes the command line passed to the compiler. this is how it keeps from serving up old object files if the optimization flags change, among other things. if that commandline contains absolute paths, the change in the version number creates a different hash, causing a cache miss.

The documentation is really unclear:
man ccache wrote:
CCACHE_BASEDIR
When hashing the output from the preprocessor, absolute paths are rewritten to relative paths, but
only for paths under the directory specified by CCACHE_BASEDIR. Paths specified by -I and similar
options get the same treatment. This is done to get cache hits even when compiling with -g and when
using absolute include directory paths. The default value of CCACHE_BASEDIR is the current working
directory.

I would interpret that this involves really only paths to header files not to C source files as in your example.


Any path under CCACHE_BASEDIR is rewritten, except in preprocessor mode where -I, -D, -include, etc. paths aren't rewritten but they also aren't hashed either under the assumption that the the preprocessor output will change anyways. Yes, the documentation is vague but I've looked at the code and tested it.

Quote:
Anyway, I am not sure how many build systems use absolute paths; usually there is no need for it with autotools project.


You'd be suprised.
_________________
by design, by neglect
for a fact or just for effect
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6296

PostPosted: Sun Apr 04, 2010 9:38 am    Post subject: Reply with quote

dirtyepic wrote:
Quote:
Anyway, I am not sure how many build systems use absolute paths; usually there is no need for it with autotools project.


You'd be suprised.
I am using script to keep exactly one compressed (compile/install) log for each emerged package. So I can quickly check:
Code:
& ( for i in /var/log/portage/*.xz; xzcat $i|egrep -q 'g(cc|\+\+).*/var/tmp/portage/[^ ]*[0-9]' && echo $i ) |wc
    128     128    8374
& echo /var/log/portage/*.xz|wc       
      1    1201   79789
Checking the output in more detail, I found several false positives (where the matching line is actually from some install command); on the other hand, some packages using --silent-rules or cmake are probably false negatives. So a rough estimate shows that perhaps 10% of the packages are involved; the listed packages are mainly small libs.
Back to top
View user's profile Send private message
rhill
Retired Dev
Retired Dev


Joined: 22 Oct 2004
Posts: 1629
Location: sk.ca

PostPosted: Mon Apr 05, 2010 12:58 am    Post subject: Reply with quote

hmm, looks like i'd be suprised then. :lol:

i guess a disproportionate number of things i maintain build outside of the source tree or don't use autotools
_________________
by design, by neglect
for a fact or just for effect
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum