Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
HOWTO: Use MySQL as a DB backend to Portage
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
chratnox
n00b
n00b


Joined: 04 Aug 2004
Posts: 11
Location: Netherlands

PostPosted: Fri Sep 03, 2004 12:22 pm    Post subject: Reply with quote

eGore911 wrote:
Two things:
1.) emerge sync is not slow because of downloading, i think. I sync over a 100 mbit lan. The issue is the reading from and writing to the fs. So this MIGHT be an improvement using mysql.
2.) I didn't look at the script, so I might be wrong: Moving portage to a DB is worth nothing. You have to build a database concept around that, optimized for searching. This would just cause that it's not save in the DB of the FS, but in another one, nothing else. But since I'm not a database expert, I'd better shut up.

I'd leave it as it is for now.


Emerge sync indeed takes a while because of the reading/writing yes, it only downloads approx. 7mb, so that's shouldn't be such a bottleneck. However the reading and writing does speed up through database (from 7 minutes for syncing I went to 3.30 minutes for syncing.. So it does speed up :)). Also, searches are alot faster.. Took about 30 seconds, now takes ~10 seconds.

I'm still struggling with that error with equery, I hope It'll get some feedback soon.
Back to top
View user's profile Send private message
MooktaKiNG
Guru
Guru


Joined: 11 Nov 2002
Posts: 326
Location: London, UK

PostPosted: Fri Sep 03, 2004 1:20 pm    Post subject: Reply with quote

chratnox wrote:
eGore911 wrote:
Two things:
1.) emerge sync is not slow because of downloading, i think. I sync over a 100 mbit lan. The issue is the reading from and writing to the fs. So this MIGHT be an improvement using mysql.
2.) I didn't look at the script, so I might be wrong: Moving portage to a DB is worth nothing. You have to build a database concept around that, optimized for searching. This would just cause that it's not save in the DB of the FS, but in another one, nothing else. But since I'm not a database expert, I'd better shut up.

I'd leave it as it is for now.


Emerge sync indeed takes a while because of the reading/writing yes, it only downloads approx. 7mb, so that's shouldn't be such a bottleneck. However the reading and writing does speed up through database (from 7 minutes for syncing I went to 3.30 minutes for syncing.. So it does speed up :)). Also, searches are alot faster.. Took about 30 seconds, now takes ~10 seconds.

I'm still struggling with that error with equery, I hope It'll get some feedback soon.


Is there a speed increase in Updating Cache too?
_________________
http://www.mooktakim.com
Athlon XP 2001, Giga-Byte GA-7VRXP MB, 640Mb DDR RAM 333MHz, MSI Geforce 4800SE 128Mb DDR, 40x12x48 Liteon CDRW drive, Flower Cooler, ADSL Router
Back to top
View user's profile Send private message
Nutterpc
Tux's lil' helper
Tux's lil' helper


Joined: 02 Feb 2004
Posts: 83

PostPosted: Sat Sep 04, 2004 2:13 am    Post subject: Reply with quote

after having got Mysql backend support for portage setup & working correctly, I've rerun regen twice since then......here's roughly how long it went each run

1st run: 5mins or so
2nd run: 1min48sec
3rd run:
real 0m34.952s
user 0m14.053s
sys 0m1.803s

as u can see, nuff said :)
_________________
If it isn't broke, you ain't tweaked it right
Registered Linux User 353232
Back to top
View user's profile Send private message
MooktaKiNG
Guru
Guru


Joined: 11 Nov 2002
Posts: 326
Location: London, UK

PostPosted: Sat Sep 04, 2004 9:40 am    Post subject: Reply with quote

One more thing...

.. Does this mean i dont need /usr/portage anymore?
I hope thats the case :)
_________________
http://www.mooktakim.com
Athlon XP 2001, Giga-Byte GA-7VRXP MB, 640Mb DDR RAM 333MHz, MSI Geforce 4800SE 128Mb DDR, 40x12x48 Liteon CDRW drive, Flower Cooler, ADSL Router
Back to top
View user's profile Send private message
geekmug
n00b
n00b


Joined: 02 Apr 2004
Posts: 39

PostPosted: Sun Sep 05, 2004 7:56 am    Post subject: Reply with quote

I wanted to also note that using the anydbm module got me a doubling of speed for my rather slow (k6-2 500Mhz) machine. I imagine it is largely to do with the anydbm module not being nearly as expensive to manipulate. Didn't try the mysql module.. I don't like it. I'm sure you get similar results, and I'd rather use a module that will be updated along with future portages.. what happens when the portage cache structure changes.. will you remember the cheap hack and who to complain to get an update? I'd rather not worry about it.
Back to top
View user's profile Send private message
MooktaKiNG
Guru
Guru


Joined: 11 Nov 2002
Posts: 326
Location: London, UK

PostPosted: Sun Sep 05, 2004 8:48 am    Post subject: Reply with quote

geekmug wrote:
I wanted to also note that using the anydbm module got me a doubling of speed for my rather slow (k6-2 500Mhz) machine. I imagine it is largely to do with the anydbm module not being nearly as expensive to manipulate. Didn't try the mysql module.. I don't like it. I'm sure you get similar results, and I'd rather use a module that will be updated along with future portages.. what happens when the portage cache structure changes.. will you remember the cheap hack and who to complain to get an update? I'd rather not worry about it.


Thats true.

So how do we setup the anydbm module?
_________________
http://www.mooktakim.com
Athlon XP 2001, Giga-Byte GA-7VRXP MB, 640Mb DDR RAM 333MHz, MSI Geforce 4800SE 128Mb DDR, 40x12x48 Liteon CDRW drive, Flower Cooler, ADSL Router
Back to top
View user's profile Send private message
JimmyW
Tux's lil' helper
Tux's lil' helper


Joined: 28 Sep 2002
Posts: 119
Location: Sweden

PostPosted: Sun Sep 05, 2004 9:10 am    Post subject: Mysql? Reply with quote

Is there not a point rather using SQLite, than a multiuser big database like mysql, that requires an own process laying in background, whith network support, support for authentications, and lots of lots, I think SQLite is the way to go, its fast and scaleable, and i think there are python wrappers.


/ Jimmy.
Back to top
View user's profile Send private message
geekmug
n00b
n00b


Joined: 02 Apr 2004
Posts: 39

PostPosted: Mon Sep 06, 2004 12:15 am    Post subject: Reply with quote

To use the anydbm module you just do the last step there..

Create /etc/portage/modules:

Code:
portdbapi.auxdbmodule="portage_db_anydbm.database"
eclass_cache.dbmodule="portage_db_anydbm.database"


Again, a "emerge regen" to spend the time now or don't and spend it on the next use of the portage system. On my box, I watched it for a long time, and then decided to go to bed, so I can't say how long that takes, but it is definitely at least an hour on my k6-2 500Mhz.
Back to top
View user's profile Send private message
Wilhelm
Tux's lil' helper
Tux's lil' helper


Joined: 27 May 2003
Posts: 149

PostPosted: Thu Sep 09, 2004 2:32 pm    Post subject: Reply with quote

Is it me or is it possible to remote connect to the mysql db using portage by modifying the python file? I'd like to see my pc's all connecting to my central portage db without the excess crap.

Another thing that disturbed me was the following

Code:

REPLACE INTO user (Host,User,Password) VALUES ('localhost','portage','');


This means no password is used allowing anybody with bad intentions to use your portage to emerge a rootkit or what have you. Mysql can be accessed from any username even a guestaccount by just doing a "mysql -u portage -p portage".

Might want to stress people that they should password it by modifying th python script and the SQL statements.
Back to top
View user's profile Send private message
taveren
Tux's lil' helper
Tux's lil' helper


Joined: 24 Jul 2002
Posts: 145
Location: London, Ontario

PostPosted: Thu Sep 09, 2004 2:51 pm    Post subject: Reply with quote

Quote:

This means no password is used allowing anybody with bad intentions to use your portage to emerge a rootkit or what have you. Mysql can be accessed from any username even a guestaccount by just doing a "mysql -u portage -p portage".


I agree. Please, at least set a password in the instructions. Better yet, make a step saying "Replace any instance of $PASSWORD with a randomly generated password." You can specify the password in the python script so it should work just fine.

Has anyone tried using a single MySQL database for several machines? I will try as soon as I can find the time. If this leads to only having to "Emerge sync" once, that would be most helpful.
Back to top
View user's profile Send private message
Wilhelm
Tux's lil' helper
Tux's lil' helper


Joined: 27 May 2003
Posts: 149

PostPosted: Fri Sep 10, 2004 8:03 am    Post subject: Reply with quote

I'm on it too ;-)

I'll report back if it works.

MyTHEORY: MySQL only allows reading of the database by users and no updating so there is no need for locking or mutual exclusions.
Locking or mutual exclusions should only be nessecary for updating but since the system itself updates and no multiple updates should occur it should work.
In the unlikely event that an ebuild would be partially put into the DB (which should't happen due to monotonic transactions) the ebuild would just fail and no harm is done.

And if it works i'd like to see if you can connect to portage on a dfferent host when building your system for a first time.

Having a single mysql driven portage system will allow new users to chose if they want to use a private mirror or an online one hosted by a volunteer or maybe gentoo.org itself. This could bring gentoo down in size and make it more attractive for small or obsoleted systems like 486'es.
Back to top
View user's profile Send private message
Wilhelm
Tux's lil' helper
Tux's lil' helper


Joined: 27 May 2003
Posts: 149

PostPosted: Fri Sep 10, 2004 9:46 am    Post subject: Reply with quote

Portage works :) Thx m8. I'll test the mirror later when i redo a pc.

For people willing to try themselves here's a few pointers that should work

- Allow the mysql portage user to be accessed from a different host.
- Modify the bind-address of mysql to your network IP address.
- Restart mysql
- Setup the python script and /etc/portage/modules file on your other pc's and modify the python script to point it to your portage server.

For initial install using mysql portage i'd have to look into it since i don't think the bootstrap script will be there or other directories like distfiles etc.. Might work for Stage2 or 3.


HOWTO FOR PASSWORDING PORTAGE
------------------------------------------------

For passworded portage do the following

Code:

# Select a password you wish to use.

# Replace $PASSWORD in following snippets with a password of your choice

# Modify the following line in the portage.sql file
REPLACE INTO user (Host,User,Password) VALUES ('localhost','portage','');
# To:
REPLACE INTO user (Host,User,Password) VALUES ('localhost','portage',PASSWORD('$PASSWORD'));

# Then modify the following line in the python script
defaultOptions = {'host':'localhost', 'port': 3306,  'user': 'portage', 'password': "", 'db': 'portage'}
# To:
 defaultOptions = {'host':'localhost', 'port': 3306,  'user': 'portage', 'password': "$PASSWORD", 'db': 'portage'}
Back to top
View user's profile Send private message
Wilhelm
Tux's lil' helper
Tux's lil' helper


Joined: 27 May 2003
Posts: 149

PostPosted: Fri Sep 10, 2004 5:11 pm    Post subject: Reply with quote

MooktaKiNG wrote:
One more thing...

.. Does this mean i dont need /usr/portage anymore?
I hope thats the case :)


Me thought the same but however the following seems to apply to me.

When i "emerge sync" i need to have mysql running which is OK since it indicates it's using the portage database.
However if i remove let's say all "app-*" directories they come back after an emerge sync which shouldn't happen. The database is updated aswell (i assume).

So this portage system still uses /usr/portage to store the ebuilds AND it stores it all in mysql so double loss of space.

So basically you can't run off the portage database without having the ebuilds in your portage directory.

Also portage still searches in the /usr/portage directory. If i remove a directory it won't find the apps.

:(

/me hopes to find a solution
Back to top
View user's profile Send private message
MooktaKiNG
Guru
Guru


Joined: 11 Nov 2002
Posts: 326
Location: London, UK

PostPosted: Fri Sep 10, 2004 5:44 pm    Post subject: Reply with quote

Wilhelm wrote:
MooktaKiNG wrote:
One more thing...

.. Does this mean i dont need /usr/portage anymore?
I hope thats the case :)


Me thought the same but however the following seems to apply to me.

When i "emerge sync" i need to have mysql running which is OK since it indicates it's using the portage database.
However if i remove let's say all "app-*" directories they come back after an emerge sync which shouldn't happen. The database is updated aswell (i assume).

So this portage system still uses /usr/portage to store the ebuilds AND it stores it all in mysql so double loss of space.

So basically you can't run off the portage database without having the ebuilds in your portage directory.

Also portage still searches in the /usr/portage directory. If i remove a directory it won't find the apps.

:(

/me hopes to find a solution


I think this is the cacje replacement. Not portage replacement. Its only good for searches etc etc.

There was PortageSQL, which was a 100% replacement. I don't know what happened to it. It was kinda nice. Its a bit. Still was in development.
_________________
http://www.mooktakim.com
Athlon XP 2001, Giga-Byte GA-7VRXP MB, 640Mb DDR RAM 333MHz, MSI Geforce 4800SE 128Mb DDR, 40x12x48 Liteon CDRW drive, Flower Cooler, ADSL Router
Back to top
View user's profile Send private message
Ateo
Advocate
Advocate


Joined: 02 Jun 2003
Posts: 2021
Location: Republic of California

PostPosted: Fri Sep 10, 2004 6:20 pm    Post subject: Reply with quote

Choice is wonderful. However, I can't help but wonder why one would want to use mysql to maintain the portage tree. Portage does just fine without it.

Still, a neat idea.
Back to top
View user's profile Send private message
Wilhelm
Tux's lil' helper
Tux's lil' helper


Joined: 27 May 2003
Posts: 149

PostPosted: Fri Sep 10, 2004 6:37 pm    Post subject: Reply with quote

MooktaKiNG wrote:


I think this is the cacje replacement. Not portage replacement. Its only good for searches etc etc.

There was PortageSQL, which was a 100% replacement. I don't know what happened to it. It was kinda nice. Its a bit. Still was in development.



Ok well i'm looking for a portage replacement since i don't think a cache is needed unless you work 100% from the DB.
Back to top
View user's profile Send private message
MooktaKiNG
Guru
Guru


Joined: 11 Nov 2002
Posts: 326
Location: London, UK

PostPosted: Fri Sep 10, 2004 6:44 pm    Post subject: Reply with quote

Wilhelm wrote:
MooktaKiNG wrote:


I think this is the cacje replacement. Not portage replacement. Its only good for searches etc etc.

There was PortageSQL, which was a 100% replacement. I don't know what happened to it. It was kinda nice. Its a bit. Still was in development.



Ok well i'm looking for a portage replacement since i don't think a cache is needed unless you work 100% from the DB.


yeh me too.
I wish someone would rise PortageSQL from the grave :D:D:D
_________________
http://www.mooktakim.com
Athlon XP 2001, Giga-Byte GA-7VRXP MB, 640Mb DDR RAM 333MHz, MSI Geforce 4800SE 128Mb DDR, 40x12x48 Liteon CDRW drive, Flower Cooler, ADSL Router
Back to top
View user's profile Send private message
Wilhelm
Tux's lil' helper
Tux's lil' helper


Joined: 27 May 2003
Posts: 149

PostPosted: Fri Sep 10, 2004 6:51 pm    Post subject: Reply with quote

MooktaKiNG wrote:
Wilhelm wrote:
MooktaKiNG wrote:


I think this is the cacje replacement. Not portage replacement. Its only good for searches etc etc.

There was PortageSQL, which was a 100% replacement. I don't know what happened to it. It was kinda nice. Its a bit. Still was in development.



Ok well i'm looking for a portage replacement since i don't think a cache is needed unless you work 100% from the DB.


yeh me too.
I wish someone would rise PortageSQL from the grave :D:D:D


Portage would pwn if it ran directly from a database. You would only need some /tmp directory like /usr/portage/distfiles and you can rock. The world-file etc. can also be put in there. You can easily share and replicate your portage through your network.

emerge sync could be done by replication for example ..... :P
Back to top
View user's profile Send private message
Wilhelm
Tux's lil' helper
Tux's lil' helper


Joined: 27 May 2003
Posts: 149

PostPosted: Fri Sep 10, 2004 6:54 pm    Post subject: Reply with quote

I think the mysql feature is in portage for the gentoo.org peeps to host an online package db.
Back to top
View user's profile Send private message
xbmodder
Guru
Guru


Joined: 25 Feb 2004
Posts: 404

PostPosted: Sun Oct 24, 2004 9:36 pm    Post subject: Reply with quote

ok i benchmarked it w00t w00t:
/proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 6
model : 8
model name : AMD Athlon(tm) MP 2400+
stepping : 1
cpu MHz : 2000.991
cache size : 256 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse syscall mp mmxext 3dnowext 3dnow
bogomips : 3940.35

processor : 1
vendor_id : AuthenticAMD
cpu family : 6
model : 8
model name : AMD Athlon(tm) Processor
stepping : 1
cpu MHz : 2000.991
cache size : 256 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse syscall mp mmxext 3dnowext 3dnow
bogomips : 3997.69

/proc/meminfo
MemTotal: 903976 kB
MemFree: 318364 kB
Buffers: 49388 kB
Cached: 311760 kB
SwapCached: 0 kB
Active: 361608 kB
Inactive: 175952 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 903976 kB
LowFree: 318364 kB
SwapTotal: 5012228 kB
SwapFree: 5012228 kB
Dirty: 120 kB
Writeback: 0 kB
Mapped: 301200 kB
Slab: 31536 kB
Committed_AS: 302708 kB
PageTables: 1872 kB
VmallocTotal: 122804 kB
VmallocUsed: 42920 kB
VmallocChunk: 77044 kB
HugePages_Total: 0
HugePages_Free: 0
Hugepagesize: 4096 kB
uptime:
14:14:56 up 1:29, 1 user, load average: 1.54, 0.66, 0.28
MYSQL TIMES:
Back to top
View user's profile Send private message
codemaker
Guru
Guru


Joined: 03 Jun 2004
Posts: 398
Location: Lisboa, Portugal

PostPosted: Sun Oct 16, 2005 10:03 pm    Post subject: Reply with quote

Hi!

I would like to try this out but I get this error when running emerge:

Quote:

delphis codemaker # emerge regen
Traceback (most recent call last):
File "/usr/bin/emerge", line 10, in ?
import portage
File "/usr/lib/portage/pym/portage.py", line 7140, in ?
portdb=portdbapi(settings["PORTDIR"])
File "/usr/lib/portage/pym/portage.py", line 5125, in __init__
self.eclassdb = eclass_cache(self.porttree_root, self.mysettings)
File "/usr/lib/portage/pym/portage.py", line 4968, in __init__
self.dbmodule = self.settings.load_best_module("eclass_cache.dbmodule")
File "/usr/lib/portage/pym/portage.py", line 1220, in load_best_module
return load_mod(best_mod)
File "/usr/lib/portage/pym/portage.py", line 139, in load_mod
mod = __import__(modname)
File "/usr/lib/portage/pym/portage_db_mysql.py", line 18
portage_db_template.database.__init__(self, path,category,dbkeys,uid,gid,config_path)
^
IndentationError: expected an indented block
delphis codemaker #



I have mysql-python-1.2.0 emerged.

Any ideas?
Back to top
View user's profile Send private message
Daim
n00b
n00b


Joined: 25 Oct 2004
Posts: 4

PostPosted: Wed Oct 19, 2005 7:55 am    Post subject: Reply with quote

I have the same problem. Any solutions?


Daim

SOLVED: I can't copy&paste ;-)
_________________
My Homepage
Daim-City
Back to top
View user's profile Send private message
nevynxxx
Veteran
Veteran


Joined: 12 Nov 2003
Posts: 1123
Location: Manchester - UK

PostPosted: Wed Oct 19, 2005 10:50 am    Post subject: Reply with quote

Ateo wrote:
Choice is wonderful. However, I can't help but wonder why one would want to use mysql to maintain the portage tree. Portage does just fine without it.

Still, a neat idea.


How about in an enterprise, 500 PCs and just one mysql portage database to be updated.

I think that's about the only advantage.
_________________
My Public Key

Wanted: Instructor in the art of Bowyery
Back to top
View user's profile Send private message
b00l
n00b
n00b


Joined: 08 Dec 2004
Posts: 13

PostPosted: Thu Oct 27, 2005 9:51 am    Post subject: Reply with quote

I get this error.... Anyone a clue?

Quote:

Traceback (most recent call last):
File "/usr/bin/emerge", line 10, in ?
import portage
File "/usr/lib/portage/pym/portage.py", line 7140, in ?
portdb=portdbapi(settings["PORTDIR"])
File "/usr/lib/portage/pym/portage.py", line 5125, in __init__
self.eclassdb = eclass_cache(self.porttree_root, self.mysettings)
File "/usr/lib/portage/pym/portage.py", line 4968, in __init__
self.dbmodule = self.settings.load_best_module("eclass_cache.dbmodule")
File "/usr/lib/portage/pym/portage.py", line 1220, in load_best_module
return load_mod(best_mod)
File "/usr/lib/portage/pym/portage.py", line 139, in load_mod
mod = __import__(modname)
File "/usr/lib/portage/pym/portage_db_mysql.py", line 2
import MySQLdb,cPickle
^
SyntaxError: invalid syntax
Back to top
View user's profile Send private message
sylvanus
n00b
n00b


Joined: 08 Jun 2005
Posts: 25

PostPosted: Thu Oct 27, 2005 11:05 am    Post subject: Reply with quote

can this be used with cdb, which is also to speed up portage:

https://forums.gentoo.org/viewtopic-t-261580.html
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3  Next
Page 2 of 3

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum