Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Help yourself and gentoo out - minimizing bandwidth usage
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
vadius
n00b
n00b


Joined: 08 Oct 2002
Posts: 12
Location: Tulsa, OK

PostPosted: Fri Feb 14, 2003 7:29 am    Post subject: Help yourself and gentoo out - minimizing bandwidth usage Reply with quote

Here is a nifty trick that I use. I run several Gentoo servers on my college campus, and many of my friends use Gentoo as well. We found that it almost seems a waste to have to rsync and waste someone else's bandwidth for redundant information, or to flood ibiblio with requests for the same files, so we set up our own local rsync + (pseudo) ibiblio site. However, due to university restrictions, we cannot provide our services outside of our firewall :( .

But, we have limited our bandwidth burden on the rest of the world, as well as saved ourselves a lot of time (as we all have 100Mbps access to our local net).

You too can help yourself and gentoo!

First, set up an rsync server (detailed here). The only changes I make are that I only rsync every two hours (for minimizing bandwidth), and I am not listed globally (university policy). However, you should always sync at :00 and :30 if you are going to be listed!

So then just add the following line to the end of the /etc/make.conf of anyone you want to let rsync to you (assuming your sever name is server.domain.com):
Code:
SYNC="rsync://server.domain.com/gentoo-x86-portage"


Don't you just hate it when you have two boxes, and they are both downloading mozilla or something huge? Solve this by using a common download directory! It can be a bit hairy though. You could, theoretically, just physically mirror the gentoo files and set up your ibiblio mirror, but this is rather big (many many gigs). The way we do it is through NFS, although Samba could be used if you want to keep all of your files on a Windows box for some reason.

Here are the steps to configure NFS to shorten download times. One of the computers will have to be a designated file server.

  1. Make sure all of the kernels supports network file systems, including NFS (and go ahead and throw in samba as well). NOTE: The kernel from the Live-CD does not support network file systems. This means, sadly, you cannot take advantage of this when installing a new system.
  2. Perform an emerge nfs-utils on the server and every client.
  3. On your server, add the following line to the /etc/exports file:
    Code:
    /usr/portage/distfiles machine1(rw,no_root_squash) machine2(rw,no_root_squash) .....

    And replace "machine1" and "machine2" and so on with the names of the machines you want to export the files too. NOTE: this allows any and all of these machines to have full access to your /usr/dportage/distfiles folder.
  4. On the server, fire up the NFS server:
    Code:
    /etc/init.d/nfs start

    And probably go ahead and stick in the startup list:
    Code:
    rc-update add nfs default

  5. On each of the machines you specified above, add the following line to /etc/fstab:
    Code:
    servername:/usr/portage/distfiles  /usr/portage/distfiles  nfs     rsize=8192,wsize=8192,timeo=14,rw,intr

    replacing "servername" with the name of your server.
  6. On each of the client machines, I'd go ahead and move /usr/portage/distfiles somewhere else, in case something goes wrong (just delete it later if everything works):
    Code:
    mv /usr/portage/distfiles /usr/portage/olddistfiles

    Then go ahead and re-create the directory:
    Code:
    mkdir /usr/portage/distfiles

    And finally, mount it:
    Code:
    mount /usr/portage/distfiles



After doing this you should have a nice, smoother, faster network, and you'll give Gentoo a break :) .

Questions, comments, etc should be sent to ch@utulsa.edu, or just reply to this message.
Back to top
View user's profile Send private message
Matje
l33t
l33t


Joined: 29 Oct 2002
Posts: 619
Location: Hasselt, Belgium

PostPosted: Fri Feb 14, 2003 2:13 pm    Post subject: Reply with quote

I was wondering, what happens when two pc's start the same emerge at the same time. Won't the download get b0rked? I share my distfiles too, and I was just wondering this :-)
_________________
Life is like a box of chocolates... Before you know it, it's empty...
Back to top
View user's profile Send private message
vadius
n00b
n00b


Joined: 08 Oct 2002
Posts: 12
Location: Tulsa, OK

PostPosted: Fri Feb 14, 2003 4:01 pm    Post subject: Reply with quote

Matje wrote:
I was wondering, what happens when two pc's start the same emerge at the same time. Won't the download get b0rked? I share my distfiles too, and I was just wondering this :-)


I've never tested this... :)

I'd imagine that the second emerge would compute a bad MD5 checksum(since it is not finished downloading), and die, leaving the other intact. I've never actually encountered a bad checksum before, so this may not be the case.

Any one else actually know the answer to this?
Back to top
View user's profile Send private message
bumpus
n00b
n00b


Joined: 14 Feb 2003
Posts: 64
Location: Cedar Rapids, IA, USA

PostPosted: Fri Feb 14, 2003 5:26 pm    Post subject: Exactly Reply with quote

Quote:
I'd imagine that the second emerge would compute a bad MD5 checksum(since it is not finished downloading), and die, leaving the other intact. I've never actually encountered a bad checksum before, so this may not be the case.


This is exactly what happens. I've never actually tried it with more than one machine, but I'll have it happen on occasion when I'm trying to "pipeline" the emerge process. Sometimes if I know I have a lot of downloading to do I'll run emerge --fetchonly -u world and then follow up a few minutes later with emerge -u world. That way the compiler never has to wait for a download to finish.
_________________
-------------
Just because I can.
Back to top
View user's profile Send private message
puddpunk
l33t
l33t


Joined: 20 Jul 2002
Posts: 681
Location: New Zealand

PostPosted: Fri Feb 14, 2003 9:54 pm    Post subject: Reply with quote

There is work being done for making the Portage build/fetch process paralell. It has been a common request on bugs.gentoo.org!
Back to top
View user's profile Send private message
ebrostig
Bodhisattva
Bodhisattva


Joined: 20 Jul 2002
Posts: 3152
Location: Orlando, Fl

PostPosted: Fri Feb 14, 2003 11:29 pm    Post subject: Reply with quote

Matje wrote:
I was wondering, what happens when two pc's start the same emerge at the same time. Won't the download get b0rked? I share my distfiles too, and I was just wondering this :-)

No, the download process is just "Read-Only".

Since there are 2 different PC's doing the emerge, you are OK.

The scenario described above is just the same as (in principle) what you do now when the emerge process donwloads files from Ibiblio or any other server.

Erik
_________________
'Yes, Firefox is indeed greater than women. Can women block pops up for you? No. Can Firefox show you naked women? Yes.'
Back to top
View user's profile Send private message
delta407
Bodhisattva
Bodhisattva


Joined: 23 Apr 2002
Posts: 2876
Location: Chicago, IL

PostPosted: Sat Feb 15, 2003 2:20 am    Post subject: Reply with quote

If you use prozilla, the risk of partial files being shared is minimized. Prozilla, being a segmented downloader, doesn't create the original filename until all parts have completed downloading -- instead, it downloads filename.prz[1-4] (or whatever) and only when all data is transferred combines them into filename. I would suggest using prozilla on your mirror boxen.
_________________
I don't believe in witty sigs.
Back to top
View user's profile Send private message
kram
n00b
n00b


Joined: 29 Nov 2002
Posts: 35

PostPosted: Sat Feb 15, 2003 2:27 am    Post subject: Reply with quote

Quote:
NOTE: The kernel from the Live-CD does not support network file systems. This means, sadly, you cannot take advantage of this when installing a new system.

You don't have to use the LiveCD to install Gentoo, I use Knoppix. Much better, IMO, because you can do a lot more with your system while installing Gentoo 8)

You can use the installation guide on the web using Mozilla, and Knoppix supports a lot more stuff than the LiveCD (including NFS, Samba and my TV card :))
Back to top
View user's profile Send private message
scout
Veteran
Veteran


Joined: 08 Mar 2003
Posts: 1991
Location: France, Paris en Semaine / Metz le W-E

PostPosted: Sat Mar 08, 2003 10:46 pm    Post subject: Reply with quote

In my campus we just share our distfile by ftp and add ourselves in GENTOO_MIRRORS in the make.conf
Back to top
View user's profile Send private message
revil
n00b
n00b


Joined: 06 Oct 2002
Posts: 68
Location: Japan

PostPosted: Tue Mar 11, 2003 8:13 am    Post subject: Reply with quote

network engineering at my campus has a gentoo rsync mirror.... :)
Back to top
View user's profile Send private message
flickerfly
l33t
l33t


Joined: 08 Nov 2002
Posts: 677
Location: Lanham, MD

PostPosted: Mon Mar 17, 2003 5:51 pm    Post subject: For the forums Reply with quote

This doesn't apply to the sync discussions, but does hit the topic:

Does the new light theme for the forums help gentoo save bandwidth/process time at all, or is it just a user side change? Could they offer us a less intensive theme? I have phpBB running on a p133, 64m and I setup the default theme to something with almost no pictures. I'd be glad to use this in order to ease server abuse until something better gets worked out.
_________________
An Evil Genious' Guide to Sheeple and How To Avoid Becoming One | 0x4C9EF4A
Back to top
View user's profile Send private message
smarm
n00b
n00b


Joined: 10 Jan 2003
Posts: 36

PostPosted: Thu Mar 20, 2003 3:42 am    Post subject: Reply with quote

I had a very similar idea, while playing tribes of all things. I'd like to expand on the idea if I may.

Why not make something similar to a rsync proxy server, give it a list of your world files, have it download at updates :00 and :30, ie --fetchonly, then have all of your gentoo boxen download the tarballs from the server. Seems it would save you from changing your gentoo boxen and many extraneous downloads. That way you could have a large common world list encompassing all of your boxen behind your proxy and save bandwidth.
Back to top
View user's profile Send private message
kamikaz3
Apprentice
Apprentice


Joined: 06 Feb 2003
Posts: 187

PostPosted: Thu Mar 20, 2003 8:03 pm    Post subject: Reply with quote

This rules
I have 3 gentoo systems
this saves both bandwidth and diskspace

I skipped the rsync part because that isn't really useful in my case
I don't like the part where rsync is running every 2 hour or ..., this doesn't save bandwidth in my case, it creates extra


So now I have one 1 machine keeping all the install files on a seperate partition (I changed portage to /home/portage and /home is on a seperate partition)
Back to top
View user's profile Send private message
pYrania
Retired Dev
Retired Dev


Joined: 27 Oct 2002
Posts: 650
Location: Cologne - Germany

PostPosted: Sat Apr 19, 2003 12:13 am    Post subject: Reply with quote

kamikaz3 wrote:
I skipped the rsync part because that isn't really useful in my case
I don't like the part where rsync is running every 2 hour or ..., this doesn't save bandwidth in my case, it creates extra


exactly that's what i think.
i mean, how many gentoo boxes do you run that it reduces bandwith when syncing every 2 hrs instead of syncing from every single box on demand?
_________________
Markus Nigbur
Back to top
View user's profile Send private message
otaci
n00b
n00b


Joined: 23 Apr 2003
Posts: 3

PostPosted: Wed Apr 23, 2003 8:38 am    Post subject: Reply with quote

Thanks for that link to an rsync mirror, been meaning to do that.

Regarding sharing distribution files, I just set up a squid proxy server, set the http_proxy and ftp_proxy environment variables on each host, and then got automated sharing, with ageing etc. Also have to make sure all the internal hosts are using the same Gentoo mirror. Works like a charm.
Back to top
View user's profile Send private message
Ari Rahikkala
Guru
Guru


Joined: 02 Oct 2002
Posts: 370
Location: Finland

PostPosted: Mon Oct 13, 2003 12:18 pm    Post subject: Reply with quote

Perhaps you should alias "emerge sync" to "ssh root@server emerge sync" ;)
_________________
<laurentius> gentoo linux?
<ari> Yesh.
<laurentius> they look horny
Back to top
View user's profile Send private message
dan2003
Apprentice
Apprentice


Joined: 11 May 2003
Posts: 257
Location: somewhere near london, UK

PostPosted: Sun Nov 16, 2003 12:59 am    Post subject: Reply with quote

quick Q, in the howto it says that syncs should be done every 00 and every 30 past the hour, it doesnt say where to set this. I notice the --timeout=1800 in the example conf file and that that is probably 1800 seconds which happens to be half an hour. Is this related to it? or should a cron job run the thing every half hour? I ask this because i have set up a local on site server to reduce the load on the mirrors on the net as i have several boxes which i sync. I want to have one box sync once per day and have the other boxes sync from it. I have set the sync option in the make.conf of the client boxes to sync from the server, and the server sync= option to the result of the netselect and rsync commands but don't know where to set the update interval.

Cheers
Back to top
View user's profile Send private message
Squinky86
Retired Dev
Retired Dev


Joined: 25 Mar 2003
Posts: 309
Location: Alabama, USA

PostPosted: Sun Nov 16, 2003 2:59 am    Post subject: Reply with quote

Use crontab to run the rsync update script at set times (in this case, 0,30 for the minutes).

I have my local rsync server sync only once every day at ~5:00 in the morning and then send me an email about what needs to be updated. That way, I know if I want to update anything or not! It's a big time/bandwidth saver.
_________________
Me
Back to top
View user's profile Send private message
dan2003
Apprentice
Apprentice


Joined: 11 May 2003
Posts: 257
Location: somewhere near london, UK

PostPosted: Sun Nov 16, 2003 5:25 pm    Post subject: Reply with quote

Another Q or 6,

what should i use to sync? should i be using the script that is detailed on the page? i.e. rsync-gentoo-portage.sh?

if i ran emerge sync from the crontab this would just update the systems own portage tree. I assume if i run the rsync-gentoo-portage.sh script it will update the tree i have in /space/gentoo/portage. Should i make the too the same i.e. use /usr/portage instead of /space/gentoo/portage?

otherwise to sync the box portage tree i have to do an emerge sync that will just sync the /usr/portage to match the /space/gentoo/portage.

I suspect i've got confused somewhere here!

I begin to think i dont need /space/gentoo/portage at all, and that i only need to do emerge sync on the server, have rsyncd running and then my other boxes sync from the server /usr/portage via rsyncd?
Is the /space/gentoo/portage bit and the script only required to host a portage server on a none gentoo box?

Please help me clear this up!
Back to top
View user's profile Send private message
amne
Bodhisattva
Bodhisattva


Joined: 17 Nov 2002
Posts: 6378
Location: Graz / EU

PostPosted: Wed Nov 19, 2003 9:14 am    Post subject: Reply with quote

dan2003 wrote:
quick Q, in the howto it says that syncs should be done every 00 and every 30 past the hour, it doesnt say where to set this.

this is only important for the official gentoo-rsync-mirrors (as a user might sync, sync again to another mirror and have his portage tree downgraded if the mirrors aren't synced at the same time). as a private person with a private rsync mirror for your network, there's no need to sync at a specific time. you also shouldn't sync more often than once a day (except when getting lots of important GLSAs every 2 hours, etc).
Back to top
View user's profile Send private message
hadfield
Retired Dev
Retired Dev


Joined: 18 Mar 2003
Posts: 308
Location: Vancouver, BC, Canada

PostPosted: Wed Nov 19, 2003 10:07 pm    Post subject: Reply with quote

Quote:
as a private person with a private rsync mirror for your network, there's no need to sync at a specific time. you also shouldn't sync more often than once a day


Exactly. For all of you who are going to set up your rsync to automatically run twice an hour for your private rsync server, perhaps you should read the rsync etiquette guidelines first :-).
http://www.gentoo.org/news/en/gwn/20030505-newsletter.xml#doc_chap1_sect3
Back to top
View user's profile Send private message
warthog
Tux's lil' helper
Tux's lil' helper


Joined: 11 Aug 2003
Posts: 82
Location: Seattle (Eastside)

PostPosted: Fri Dec 12, 2003 7:03 pm    Post subject: private rsync questions Reply with quote

dan2003 wrote:


if i ran emerge sync from the crontab this would just update the systems own portage tree. I assume if i run the rsync-gentoo-portage.sh script it will update the tree i have in /space/gentoo/portage. Should i make the too the same i.e. use /usr/portage instead of /space/gentoo/portage?


I have the same question, but I haven't found an answer searching the forums.

I recently setup a private rsync mirror on my own home network so that I can sync my three machines without making three connections to the gentoo mirrors. However, on the machine that is actiing as the private mirror, should it sync to itself? Or should /usr/portage and /space/gentoo/portage be combined? What is the proper thing to do?

dan2003 wrote:

otherwise to sync the box portage tree i have to do an emerge sync that will just sync the /usr/portage to match the /space/gentoo/portage.


That's my confusion exactly!
Back to top
View user's profile Send private message
dan2003
Apprentice
Apprentice


Joined: 11 May 2003
Posts: 257
Location: somewhere near london, UK

PostPosted: Sat Dec 13, 2003 11:33 pm    Post subject: Reply with quote

nice to see im not alon on this one! been a while since i first posted the question and im still none te wiser!
Back to top
View user's profile Send private message
warthog
Tux's lil' helper
Tux's lil' helper


Joined: 11 Aug 2003
Posts: 82
Location: Seattle (Eastside)

PostPosted: Sun Dec 14, 2003 12:44 am    Post subject: Reply with quote

dan2003 wrote:
nice to see im not alon on this one! been a while since i first posted the question and im still none te wiser!


Well... I did some more searching and I found this discussion which I think helps to answer our question, if not bring up some additional ones:

https://forums.gentoo.org/viewtopic.php?t=59134

in this discussion they use /usr/portage instead of a seperate partition. I have since changed my configuration to use /usr/portage and it seems to be working well. Somebody correct me if I'm wrong to do this.
Back to top
View user's profile Send private message
fusion
Tux's lil' helper
Tux's lil' helper


Joined: 02 Nov 2002
Posts: 119

PostPosted: Sun Dec 14, 2003 1:54 am    Post subject: Reply with quote

warthog try
http://www.gentoo.org/doc/en/rsync.xml :twisted: :wink:
_________________
Friends don't let friends send HTML email
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum