Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[TIP] Using psyco to speed up emerge
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
jpc82
Guru
Guru


Joined: 09 Mar 2003
Posts: 326

PostPosted: Wed May 26, 2004 12:45 pm    Post subject: Reply with quote

emerge -pUD world went from 10sec to 8sec
emerge -p depclean went from 7sec to 5 sec
Back to top
View user's profile Send private message
papabean
Tux's lil' helper
Tux's lil' helper


Joined: 26 Oct 2002
Posts: 130
Location: Sacramento, California

PostPosted: Thu Jun 03, 2004 10:47 am    Post subject: Reply with quote

What effect, if any, does this have on emerge sync?
_________________
-- The world is full of tough guys. It doesn't need me to be one too.
Back to top
View user's profile Send private message
MK
Tux's lil' helper
Tux's lil' helper


Joined: 27 Nov 2002
Posts: 97
Location: Bærum, Norway

PostPosted: Thu Jun 03, 2004 11:56 am    Post subject: Reply with quote

Hm, tried it and it does improve speed a little

emerge -up world from 5.5s to 4.2s
emerge -s psyco from 1.8s to 1.3s
emerge -S psyco from 36s to 28s

Helps a little so I'll keep it :)
Back to top
View user's profile Send private message
alec
Apprentice
Apprentice


Joined: 19 Apr 2002
Posts: 270
Location: Here

PostPosted: Thu Jun 03, 2004 4:26 pm    Post subject: Reply with quote

papabean wrote:
What effect, if any, does this have on emerge sync?


Probably none. This only speeds up the execution of the python code in /usr/bin/emerge - 'emerge sync' farms everything out to rsync, so you will probably see no improvement in that department. I find it speeds up dependency calculations and searches.
Back to top
View user's profile Send private message
papabean
Tux's lil' helper
Tux's lil' helper


Joined: 26 Oct 2002
Posts: 130
Location: Sacramento, California

PostPosted: Thu Jun 03, 2004 4:39 pm    Post subject: Reply with quote

alec wrote:
papabean wrote:
What effect, if any, does this have on emerge sync?


Probably none. This only speeds up the execution of the python code in /usr/bin/emerge - 'emerge sync' farms everything out to rsync, so you will probably see no improvement in that department. I find it speeds up dependency calculations and searches.


Thanks. I have noticed that searches seem much faster.
_________________
-- The world is full of tough guys. It doesn't need me to be one too.
Back to top
View user's profile Send private message
Pink
Veteran
Veteran


Joined: 24 Jul 2003
Posts: 1062

PostPosted: Thu Jun 03, 2004 4:40 pm    Post subject: Reply with quote

I haven't benchmarked it, but it does seem to make a difference here - even if it is only a few seconds, it's got to be worth it.
Back to top
View user's profile Send private message
knarf
Guru
Guru


Joined: 20 Feb 2003
Posts: 338
Location: Geneva, Switzerland

PostPosted: Sat Jun 05, 2004 10:25 pm    Post subject: Reply with quote

With :
Code:
real    0m28.690s
user    0m21.925s
sys     0m2.540s


Without :
Code:
real    0m26.932s
user    0m24.073s
sys     0m2.663s
Back to top
View user's profile Send private message
Species
n00b
n00b


Joined: 16 Jan 2004
Posts: 21
Location: Germany

PostPosted: Mon Jun 07, 2004 3:28 pm    Post subject: Reply with quote

same speedup effect as described on my laptop :)

without psyco:
Code:
# emerge -pvDe world
real 0m17.276s
user 0m13.736s
sys 0m3.488s


with psyco:
Code:
# emerge -pvDe world
real 0m12.917s
user 0m9.789s
sys 0m3.002s
Back to top
View user's profile Send private message
Gil-galad55
Tux's lil' helper
Tux's lil' helper


Joined: 02 Feb 2004
Posts: 101

PostPosted: Wed Jun 16, 2004 1:57 pm    Post subject: Reply with quote

Since this is an I/O bound process, most likely those people with significant time differences have either small world files (less I/O required) or slower processors, where the more efficient code helps out. As the world file increases, I can see the two times converging as I/O dominates.
_________________
a^2 + b^2 = c^2
"To strive, to seek, to find, and not to yield." Tennyson
Back to top
View user's profile Send private message
guard0
Tux's lil' helper
Tux's lil' helper


Joined: 26 Jun 2003
Posts: 96

PostPosted: Wed Jun 16, 2004 2:59 pm    Post subject: Reply with quote

i doubt it would be of much worth as a USE flag, but as a FEATURES flag (where you put distcc, ccache, sandbox, etc) since it would primarily be used for portage utilities...
_________________
Linux on an AMD processor: an OS for people who dont use deoderant with a CPU for people who can't afford deoderant
Back to top
View user's profile Send private message
aridhol
Guru
Guru


Joined: 20 Jan 2003
Posts: 509
Location: Stockholm, Sweden

PostPosted: Thu Jun 17, 2004 4:49 pm    Post subject: Reply with quote

For you speedfreaks it might be interested to know you can use mySQL to do your searches:

https://forums.gentoo.org/viewtopic.php?t=175461
_________________
72 of Pitcairn Islands 49 inhabitants use Seti@Home
"If you buy a DVD you have a copy. If you want a backup copy you buy another one."
"Anyone who is capable of getting themselves made President should on no account be allowed to do the job."
Back to top
View user's profile Send private message
madmango
Guru
Guru


Joined: 15 Jul 2003
Posts: 507
Location: PA, USA

PostPosted: Thu Jun 17, 2004 5:04 pm    Post subject: Reply with quote

time emerge -peD world


without: 5.997 sec
with: 5.977 sec

wOw! Amazing .2 second decrease!
_________________
word.
Back to top
View user's profile Send private message
polle
Veteran
Veteran


Joined: 28 Feb 2003
Posts: 1498
Location: Belgium

PostPosted: Thu Jun 17, 2004 6:48 pm    Post subject: Reply with quote

I get this error:

File "/usr/bin/emerge", line 12
except ImportError:
^
SyntaxError: invalid syntax

I tried to copy and paste from the above but still get that error
(used nano to edit the file /usr/bin/emerge and tried with and without a space before except)
Back to top
View user's profile Send private message
fca
Guru
Guru


Joined: 22 Feb 2003
Posts: 346
Location: Netherlands

PostPosted: Fri Jul 02, 2004 10:50 am    Post subject: Reply with quote

unfortunately this doesn't work on a Pax protected system, The psyco compiler gets killed then.
Code:

Jul  2 12:41:29 [kernel] PAX: execution attempt in: <anonymous mapping>, 17d6a000-17e1e000 17d6a000
Back to top
View user's profile Send private message
c0balt
Guru
Guru


Joined: 04 Jul 2004
Posts: 441
Location: Germany

PostPosted: Thu Aug 05, 2004 3:22 pm    Post subject: Reply with quote

Hi,
couldnt this be added to portage officially?
afaik the code doesnt brake anything if psyco isnt installed.
everytime theres a new portage package one has to add the lines manually, though thats not that often its annoying and i tend to forget ;)
Back to top
View user's profile Send private message
blu3bird
Retired Dev
Retired Dev


Joined: 04 Oct 2003
Posts: 612
Location: Munich, Germany

PostPosted: Sat Aug 07, 2004 9:35 am    Post subject: Reply with quote

emere -up world:
Code:

without psyco:
7 Secunds
with psyco:
5 Seconds


emere -s gentoo:
Code:

without psyco:
3 Secunds
with psyco:
3 Seconds


not as fast as it should be, but okay :)
Back to top
View user's profile Send private message
thinko
n00b
n00b


Joined: 23 Jul 2003
Posts: 6

PostPosted: Tue Sep 14, 2004 7:22 am    Post subject: Emerge runtimes with psyco enabled. Reply with quote

I think you guys haven't correctly configured psyco.. when I use it with emerge, I see a significant benefit every time.

ex: emerge -pDuv world
Without Psyco:
Code:

real    0m34.392s
user    0m23.512s
sys     0m3.056s

With Psyco:
Code:

real    0m21.905s
user    0m16.532s
sys     0m2.709s


or.. emerge -pe world
Without Psyco:
Code:

real    0m34.925s
user    0m23.368s
sys     0m3.142s

With Psyco:
Code:

real    0m24.636s
user    0m17.210s
sys     0m2.844s


Funny, I think it's well worth it.
Back to top
View user's profile Send private message
Pythonhead
Developer
Developer


Joined: 16 Dec 2002
Posts: 1801
Location: Redondo Beach, Republic of Calif.

PostPosted: Tue Sep 14, 2004 10:11 pm    Post subject: Reply with quote

All these tests are pretty meaningless. I notice everyone is timing the psyco version second. All this stuff is getting cached, so whenever you run a second test, it should be faster, psyco or not:

I boot up my system and run emerge -ep world:

real 0m51.199s
user 0m15.540s
sys 0m1.467s

Then run it again immediately:

real 0m25.222s
user 0m15.376s
sys 0m1.192s

(neither test using psyco)

There was already a bug filed asking to put psyco in portage (where requests go, not these forums, by the way) and it was declined:

https://bugs.gentoo.org/show_bug.cgi?id=45793

Also, psyco also uses an enormous amount of memory, so wouldn't be great for everyone.

psyco may help a tiny bit for searching portage, but its almost all disk i/o that is the bottleneck. Thats why people aren't getting much if any speedups with the mysql portage backend (last I checked).
Back to top
View user's profile Send private message
neysx
Retired Dev
Retired Dev


Joined: 27 Jan 2003
Posts: 795

PostPosted: Tue Sep 14, 2004 10:57 pm    Post subject: Reply with quote

Psyco can improve performances significantly depending on CPU/IO/World file.

On a Via C3-800
#1 w/psyco:
real 0m30.547s
user 0m12.717s
sys 0m1.980s

#2 w/ psyco again (cached data):
real 0m15.492s
user 0m12.352s
sys 0m1.694s

#3 wo/psyco (data still in cache):
real 0m21.210s
user 0m17.691s
sys 0m1.541s

Another run wo/psyco yields the same values.

Same test on an old P100 (2x with psyco, 1x without)
real 1m54.831s
user 1m38.690s
sys 0m7.410s

real 1m45.879s
user 1m38.610s
sys 0m6.620s

real 2m25.927s
user 2m18.590s
sys 0m6.510s

On my 2xMp2400+ (huge world file)
with psyco
real 1m25.650s
user 1m4.695s
sys 0m10.188s

without
real 1m31.772s
user 1m12.129s
sys 0m10.334s

Generally speaking, using psyco does not make much sense on fast modern machines but it has its uses on old equipment.
Back to top
View user's profile Send private message
fbd
n00b
n00b


Joined: 12 Sep 2004
Posts: 21
Location: Akureyri, Iceland

PostPosted: Sat Oct 09, 2004 12:19 pm    Post subject: :) Reply with quote

AlterEgo wrote:
Tried:
emerge -pu world: 1 min 20 sec.
With psyco: 1 min 18 seconds.

Not that amazing :roll:


Try using emerge -s somename

You will see the diffrence :P
_________________
Frailty, thy name is woman!
Back to top
View user's profile Send private message
v0m1t
n00b
n00b


Joined: 20 Aug 2004
Posts: 11

PostPosted: Sat Oct 09, 2004 1:22 pm    Post subject: Reply with quote

emerge -s xmms with psyco
real 0m11.637s
user 0m4.909s
sys 0m0.480s

emerge -s xmms without psyco
real 0m8.575s
user 0m5.106s
sys 0m0.454s



Both tests with cached results.
Back to top
View user's profile Send private message
redshift
Tux's lil' helper
Tux's lil' helper


Joined: 26 Nov 2003
Posts: 98

PostPosted: Tue Oct 12, 2004 2:46 pm    Post subject: Reply with quote

With portage 2.0.51_rc9, this seems to give me constant segfaults. Before emerging rc9 today (from rc7) I never had problems, and psyco did give a little boost.
_________________
Tom
Back to top
View user's profile Send private message
Trevoke
Advocate
Advocate


Joined: 04 Sep 2004
Posts: 4099
Location: NY, NY

PostPosted: Tue Oct 12, 2004 2:53 pm    Post subject: Reply with quote

Same issue here. New portage no like psyco.
_________________
Votre moment detente
What is the nature of conflict?
Back to top
View user's profile Send private message
Jtb
Apprentice
Apprentice


Joined: 19 Dec 2003
Posts: 157
Location: Germany/Darmstadt

PostPosted: Thu Oct 14, 2004 10:00 am    Post subject: Reply with quote

Just tried it on a fast machine (AMD64 3000+ with 1GB but 32bit Gentoo):

time emerge search psyco
with:
real 0m9.545s
user 0m0.778s
sys 0m0.191s

without:
real 0m7.229s
user 0m0.946s
sys 0m0.165s


time emerge -S psyco
with:
real 1m0.649s
user 0m16.814s
sys 0m3.649s

without:
real 1m11.777s
user 0m29.205s
sys 0m3.637s


time emerge -ep world
without:
real 0m44.441s
user 0m13.307s
sys 0m1.066s

with:
real 0m16.807s
user 0m8.227s
sys 0m0.968s


Reboots between with and without:
emerge search psyco, emerge -S psyco [reboot] emerge search psyco, emerge -S psyco [reboot emerge -ep world [reboot] emerge -ep world
_________________
Jens

.. God is real - unless declared integer!
Back to top
View user's profile Send private message
Viha
Tux's lil' helper
Tux's lil' helper


Joined: 04 Oct 2003
Posts: 121
Location: Jon's World

PostPosted: Thu Oct 14, 2004 12:06 pm    Post subject: Reply with quote

Segfaulted here on emerge clean (portage 2.0.51_rc9).
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3  Next
Page 2 of 3

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum