Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
HOWTO:Download Cache for your LAN-Http-Replicator (ver 3.0)
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3 ... 5, 6, 7 ... 22, 23, 24  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Sun Sep 05, 2004 6:01 am    Post subject: Reply with quote

I have a couple of things for you to try....

robfantini wrote:
I think that I've followed the install instructions exactally... But I get the following error when I try to emerge a program from the server:

14:25:07 ERROR 400: Bad Request.


This error didn't originate from http-replicator - it came directly from the upstream server, as seen in replicators log.

What happens when you try wget by itself?
Code:

wget   http://gentoo.oregonstate.edu/distfiles/modutils-2.4.26.tar.bz2


Do you get the same error?

Is there really a newline in your make.conf between {FILE} and the "?
Quote:

RESUMECOMMAND=" /usr/bin/wget -t 5 --passive-ftp \${URI} -O \${DISTDIR}/\${FILE}
"


robfantini wrote:


The server is behind a firewall.. I made it so port 80 is forwarded to our server. Apache is running and i can access the Apache test web page from off site.



I don't understand why you mention this and what your trying to do here. Nowhere in the howto does it mention apache nor port forwarding. Http-replicator doesn't need apache running but won't bother it either. It doesn't need port 80 forwarding. You might be causing the problems with your forwarding/firewall setup. But probably not if your just using a typical hardware router like linksys etc.

Setting the debug option in replicator will give more verbose output also. Try that and post the output.
Back to top
View user's profile Send private message
robfantini
Tux's lil' helper
Tux's lil' helper


Joined: 10 Jan 2004
Posts: 106
Location: Boston, Massachusetts

PostPosted: Sun Sep 05, 2004 11:52 am    Post subject: Reply with quote

What happens when you try wget by itself?
Code:

wget   http://gentoo.oregonstate.edu/distfiles/modutils-2.4.26.tar.bz2

wget downloaded the file.
Quote:

Is there really a newline in your make.conf between {FILE} and the "?

No there is not a newline. It loooked like there was one due to the font size i use in konqueror.
Quote:

Setting the debug option in replicator will give more verbose output also. Try that and post the output.

Ok I'll do that
Quote:

I don't understand why you mention this and what your trying to do here. Nowhere in the howto does it mention apache nor port forwarding. Http-replicator doesn't need apache running but won't bother it either. It doesn't need port 80 forwarding. You might be causing the problems with your forwarding/firewall setup. But probably not if your just using a typical hardware router like linksys etc.

I can see how that can be confusing... I did not have apache installed and thought it might be needed... I'm not familiar with apache so it seems complex.. I'm glad to know now that H-P does not need apache or a web server running to work.

Thanks for the suggestions.
Back to top
View user's profile Send private message
robfantini
Tux's lil' helper
Tux's lil' helper


Joined: 10 Jan 2004
Posts: 106
Location: Boston, Massachusetts

PostPosted: Sun Sep 05, 2004 12:10 pm    Post subject: Reply with quote

Debug is enabled in /etc/http-replicator.conf .
Here is output from an emerge.
Also I've posted my make.conf below.

It looks like http downloads are not working and ftp dowlloads are:

Calculating dependencies ...done!
[ebuild N ] sys-kernel/gentoo-dev-sources-2.6.8-r3 -build -doc -ultra1 34,936 kB

Total size of downloads: 34,936 kB

Do you want me to merge these packages? [Yes/No] yes
>>> emerge (1 of 1) sys-kernel/gentoo-dev-sources-2.6.8-r3 to /
>>> Downloading http://gentoo.oregonstate.edu/distfiles/genpatches-2.6-8.50-extras.tar.bz2
--08:01:28-- http://gentoo.oregonstate.edu/distfiles/genpatches-2.6-8.50-extras.tar.bz2
=> `/usr/portage/distfiles/genpatches-2.6-8.50-extras.tar.bz2'
Connecting to 192.168.1.3:8080... connected.
Proxy request sent, awaiting response... 400 Bad Request
08:01:28 ERROR 400: Bad Request.

>>> Downloading http://mirror.datapipe.net/gentoo/distfiles/genpatches-2.6-8.50-extras.tar.bz2
--08:01:28-- http://mirror.datapipe.net/gentoo/distfiles/genpatches-2.6-8.50-extras.tar.bz2
=> `/usr/portage/distfiles/genpatches-2.6-8.50-extras.tar.bz2'
Connecting to 192.168.1.3:8080... connected.
Proxy request sent, awaiting response... 400 Bad Request
08:01:28 ERROR 400: Bad Request.

>>> Downloading http://open-systems.ufl.edu/mirrors/gentoo/distfiles/genpatches-2.6-8.50-extras.tar.bz2
--08:01:28-- http://open-systems.ufl.edu/mirrors/gentoo/distfiles/genpatches-2.6-8.50-extras.tar.bz2
=> `/usr/portage/distfiles/genpatches-2.6-8.50-extras.tar.bz2'
Connecting to 192.168.1.3:8080... connected.
Proxy request sent, awaiting response... 400 Bad Request
08:01:28 ERROR 400: Bad Request.

>>> Downloading ftp://ftp.gtlib.cc.gatech.edu/pub/gentoo/distfiles/genpatches-2.6-8.50-extras.tar.bz2
--08:01:28-- ftp://ftp.gtlib.cc.gatech.edu/pub/gentoo/distfiles/genpatches-2.6-8.50-extras.tar.bz2
=> `/usr/portage/distfiles/genpatches-2.6-8.50-extras.tar.bz2'
Resolving ftp.gtlib.cc.gatech.edu... 130.207.108.134
Connecting to ftp.gtlib.cc.gatech.edu[130.207.108.134]:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD /pub/gentoo/distfiles ... done.
==> PASV ... done. ==> RETR genpatches-2.6-8.50-extras.tar.bz2 ... done.

[ <=> ] 107,808 141.14K/s

08:01:47 (140.74 KB/s) - `/usr/portage/distfiles/genpatches-2.6-8.50-extras.tar.bz2' saved [107808]

>>> Downloading http://gentoo.oregonstate.edu/distfiles/linux-2.6.8.tar.bz2
--08:01:47-- http://gentoo.oregonstate.edu/distfiles/linux-2.6.8.tar.bz2
=> `/usr/portage/distfiles/linux-2.6.8.tar.bz2'
Connecting to 192.168.1.3:8080... connected.
Proxy request sent, awaiting response... 400 Bad Request
08:01:47 ERROR 400: Bad Request.

>>> Downloading http://mirror.datapipe.net/gentoo/distfiles/linux-2.6.8.tar.bz2
--08:01:47-- http://mirror.datapipe.net/gentoo/distfiles/linux-2.6.8.tar.bz2
=> `/usr/portage/distfiles/linux-2.6.8.tar.bz2'
Connecting to 192.168.1.3:8080... connected.
Proxy request sent, awaiting response... 400 Bad Request
08:01:47 ERROR 400: Bad Request.

>>> Downloading http://open-systems.ufl.edu/mirrors/gentoo/distfiles/linux-2.6.8.tar.bz2
--08:01:47-- http://open-systems.ufl.edu/mirrors/gentoo/distfiles/linux-2.6.8.tar.bz2
=> `/usr/portage/distfiles/linux-2.6.8.tar.bz2'
Connecting to 192.168.1.3:8080... connected.
Proxy request sent, awaiting response... 400 Bad Request
08:01:47 ERROR 400: Bad Request.

>>> Downloading ftp://ftp.gtlib.cc.gatech.edu/pub/gentoo/distfiles/linux-2.6.8.tar.bz2
--08:01:47-- ftp://ftp.gtlib.cc.gatech.edu/pub/gentoo/distfiles/linux-2.6.8.tar.bz2
=> `/usr/portage/distfiles/linux-2.6.8.tar.bz2'
Resolving ftp.gtlib.cc.gatech.edu... 130.207.108.134
Connecting to ftp.gtlib.cc.gatech.edu[130.207.108.134]:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD /pub/gentoo/distfiles ... done.
==> PASV ... done. ==> RETR linux-2.6.8.tar.bz2 ... done.

[ <=> ] 7,514,544 156.19K/s


make.conf:
# fbc3 gentoo P4 server
# 2004-09-02 see https://forums.gentoo.org/viewtopic.php?t=173226
# http://gentoo-wiki.com/HOWTO_Download_Cache_for_LAN-Http-Replicator
PORTDIR_OVERLAY=/bkup/portage/server/local
http_proxy="http://192.168.1.3:8080"
RESUMECOMMAND=" /usr/bin/wget -t 5 --passive-ftp \${URI} -O \${DISTDIR}/\${FILE}"

# 07-05-04 changed to /bkup
PKGDIR="/bkup/portage/server/packages"
CFLAGS="-march=i686 -mmmx -msse -O2 -pipe -fomit-frame-pointer"
CXXFLAGS="${CFLAGS}"
CHOST="i686-pc-linux-gnu"

# 08-20-04 added maildir. i noticed it excluded from postfix build.
# 2004-09-03 added mysql apache2 pam ssl xml xml2 for apache
# type env-update (return) to make your changes take effect!
USE=" mysql apache2 pam ssl xml xml2 maildir -X -xv -xmms -xosd -trusted -usb -wx
windows -wavelan -pda -pcmcia -oss -opengl -mozilla -kde -gtkhtml -gtk2 -gtk -gpho
to2 -gnome -dga -3dfx -3dnow -arts nocardbus mmx cups "

# # hyperthreading cpu , use j3 xeon4 on fbc3
MAKEOPTS="-j2"

PORTAGE_NICENESS=15
FEATURES="buildpkg"

GENTOO_MIRRORS="http://mirror.datapipe.net/gentoo http://open-systems.ufl.edu/mirrors/gentoo ftp://ftp.gtlib.cc.gatech.edu/pub/gentoo"
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Sun Sep 05, 2004 4:46 pm    Post subject: Reply with quote

I didn't see anything in your make.conf.

The debug option will put more info in http-replicator's log. Thats what I need to see.

The bad request just shouldn't happen. I wonder if you downloaded the ebuild or opened the archive in windows? If so, the script is probably corrupted. You should download and reinstall http-replicator next.
Back to top
View user's profile Send private message
robfantini
Tux's lil' helper
Tux's lil' helper


Joined: 10 Jan 2004
Posts: 106
Location: Boston, Massachusetts

PostPosted: Sun Sep 05, 2004 11:43 pm    Post subject: Reply with quote

solved.

My problem has nothing to do with H-R .

I stopped H-R , tried to emerge a program and had the same issues. For some reason emerge has errors when trying to wget from HTTP.

Sorry about wasting your time! I'll get this fixed and then look forward to using your excellent program!
Back to top
View user's profile Send private message
fzxdude
n00b
n00b


Joined: 13 Jun 2004
Posts: 72

PostPosted: Mon Sep 06, 2004 5:01 pm    Post subject: Reply with quote

why not jsut nfs the distfiles dir on one of the machines ?

ya download the package once for all the machines with that share
Back to top
View user's profile Send private message
robfantini
Tux's lil' helper
Tux's lil' helper


Joined: 10 Jan 2004
Posts: 106
Location: Boston, Massachusetts

PostPosted: Mon Sep 06, 2004 6:51 pm    Post subject: Reply with quote

fzxdude wrote:
why not jsut nfs the distfiles dir on one of the machines ?

ya download the package once for all the machines with that share


I'm doing that now. The problem is we have 10+ linux pc's. Using nfs works ok, but if 2 pc's tried to download the same distfile at same time then there would be a problem.

Using http-replicator would prevent this.

As I could not get H-R working, I'll try using rsync daily to mirror all gentoo distfiles to my nfs directory.
Back to top
View user's profile Send private message
fzxdude
n00b
n00b


Joined: 13 Jun 2004
Posts: 72

PostPosted: Mon Sep 06, 2004 8:41 pm    Post subject: Reply with quote

ya we just have 7 here ... so its not that bad...
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Tue Sep 07, 2004 12:13 pm    Post subject: Reply with quote

First, let me say that if you own all other boxes nfs sharing is an option. There are corruption and other issues you need to worry about but if you can control all the boxes then you can minimize the chance of corruption.

I don't want to turn this thread into a flamewar about the potential corruption and other problems with sharing the distfile dir. I'd ask that someone start another thread to discuss that issue if anyone has any questions.

This thread is about the advantages of Http-Replicator!!

Http-Replicator is an easy and quick install that relieves the users of virtually all headaches associated with multiple boxen. Once installed, there are no corruption or other issues to worry about. Any box can emerge any package at any time and http-replicator will make sure the downloads happen in a safe and efficient manner!!

Think about this advantage in a situation like a college campus: If you have 10 (or 100) avid gentooers that start a mozilla upgrade at nearly the same time what will happen with nfs sharing?

All 10 or (100) boxes will download a separate copy of the 34438800 byte file!! Why? Because each box will look for the package and will see an incomplete download and start a new download to resume the file from the already heavily loaded volunteer mirrors.

If Http-Replicator were is use, it would intercept all 10 (or 100) download requests from the LAN and would simultaneously download the file from the internet and stream the file to all 10 (or 100) users. The result would be only 1 copy of the package downloaded from the internet and all LAN clients simultaneously receiving the package!! Any client that requests the package after the download is complete will receive the file from the cache at LAN speeds!

The next question you have to ask yourself is what were you thinking when your tried to nfs share your distfile dir with 10 (or 100) semi-strangers on your college campus!!
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Tue Sep 07, 2004 7:59 pm    Post subject: Reply with quote

I agree totally. Http-replicator is the right tool for the job. Sharing over NFS is a horrible hack that will bite you eventually (but I did use it for a long time until I discovered HR).
Back to top
View user's profile Send private message
MattSharp
Apprentice
Apprentice


Joined: 15 Aug 2003
Posts: 258

PostPosted: Tue Sep 14, 2004 4:03 am    Post subject: Reply with quote

Im sure someone has had this problem but I didn't see anything about it. When I try to emerge packages on the server, it has problems downloading the package:

Quote:

lancelot root # emerge mutt
Calculating dependencies ...done!
>>> emerge (1 of 1) mail-client/mutt-1.5.6-r3 to /
>>> Resuming download...
>>> Downloading http://gentoo.chem.wisc.edu/gentoo//distfiles/mutt-1.5.6i.tar.gz
--23:02:56-- http://gentoo.chem.wisc.edu/gentoo/distfiles/mutt-1.5.6i.tar.gz
=> `/mutt-1.5.6i.tar.gz'
Resolving gentoo.chem.wisc.edu... 128.104.70.13
Connecting to gentoo.chem.wisc.edu[128.104.70.13]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2,910,730 [application/x-tar]

100%[=================>] 2,910,730 153.54K/s ETA 00:00

23:03:14 (156.60 KB/s) - `/mutt-1.5.6i.tar.gz' saved [2910730/2910730]

>>> Resuming download...
>>> Downloading http://gentoo.chem.wisc.edu/gentoo/distfiles/mutt-1.5.6i.tar.gz
--23:03:15-- http://gentoo.chem.wisc.edu/gentoo/distfiles/mutt-1.5.6i.tar.gz
=> `/mutt-1.5.6i.tar.gz'
Resolving gentoo.chem.wisc.edu... 128.104.70.13
Connecting to gentoo.chem.wisc.edu[128.104.70.13]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2,910,730 [application/x-tar]

100%[=================>] 2,910,730 155.56K/s ETA 00:00

23:03:33 (156.05 KB/s) - `/mutt-1.5.6i.tar.gz' saved [2910730/2910730]

>>> Resuming download...
>>> Downloading http://mirrors.tds.net/gentoo/distfiles/mutt-1.5.6i.tar.gz
--23:03:33-- http://mirrors.tds.net/gentoo/distfiles/mutt-1.5.6i.tar.gz
=> `/mutt-1.5.6i.tar.gz'
Resolving mirrors.tds.net... 216.170.153.145
Connecting to mirrors.tds.net[216.170.153.145]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2,910,730 [application/x-gzip]

100%[=================>] 2,910,730 360.87K/s ETA 00:00

23:03:47 (223.50 KB/s) - `/mutt-1.5.6i.tar.gz' saved [2910730/2910730]

>>> Resuming download...
>>> Downloading ftp://ftp.mutt.org/mutt/devel/mutt-1.5.6i.tar.gz
--23:03:47-- ftp://ftp.mutt.org/mutt/devel/mutt-1.5.6i.tar.gz
=> `/mutt-1.5.6i.tar.gz'
Resolving ftp.mutt.org... 82.165.34.161
Connecting to ftp.mutt.org[82.165.34.161]:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD /mutt/devel ... done.
==> PASV ... done. ==> RETR mutt-1.5.6i.tar.gz ... done.
Length: 2,910,730 (unauthoritative)

100%[=========================================>] 2,910,730 208.38K/s ETA 00:00

23:04:01 (232.11 KB/s) - `/mutt-1.5.6i.tar.gz' saved [2910730]

!!! Couldn't download mutt-1.5.6i.tar.gz. Aborting.


What is causing this?
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Tue Sep 14, 2004 4:52 am    Post subject: Reply with quote

MattSharp wrote:
Im sure someone has had this problem but I didn't see anything about it. When I try to emerge packages on the server, it has problems downloading the package:



Your connections are all on port 80, so your not downloading the package through http-replicator.... Are you trying to or is this a general gentoo problem your having?
Back to top
View user's profile Send private message
MattSharp
Apprentice
Apprentice


Joined: 15 Aug 2003
Posts: 258

PostPosted: Tue Sep 14, 2004 3:37 pm    Post subject: Reply with quote

flybynite wrote:
MattSharp wrote:
Im sure someone has had this problem but I didn't see anything about it. When I try to emerge packages on the server, it has problems downloading the package:



Your connections are all on port 80, so your not downloading the package through http-replicator.... Are you trying to or is this a general gentoo problem your having?


I am trying to use http-replicator. This is the "server" box. The clients seem to work fine...I think.

But when I try to get a package on there I have that problem. I commented out the http_proxy line cause that didn't work either. Here is what I get when I try it with that line:

Quote:

Calculating dependencies ...done!
>>> emerge (1 of 1) mail-client/mutt-1.5.6-r3 to /
>>> Previously fetched file: patch-1.5.6.rr.compressed.gz MD5 ;-)
>>> Previously fetched file: patch-1.5.6.dw.pgp-timeout.1 MD5 ;-)
>>> Previously fetched file: patch-1.5.6.tg.hcache.0 MD5 ;-)
>>> Previously fetched file: patch-1.5.5.1.cd.edit_threads.9.5-gentoo.bz2 MD5 ;-
)
>>> Previously fetched file: patch-1.5.6.dw.mbox-hook.1 MD5 ;-)
>>> Resuming download...
>>> Downloading http://gentoo.chem.wisc.edu/gentoo//distfiles/mutt-1.5.6i.tar.gz
--10:27:28-- http://gentoo.chem.wisc.edu/gentoo/distfiles/mutt-1.5.6i.tar.gz
=> `/mutt-1.5.6i.tar.gz'
Resolving localhost... 127.0.0.1
Connecting to localhost[127.0.0.1]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 2,910,730

100%[====================================>] 2,910,730 2.23M/s

10:27:30 (2.22 MB/s) - `/mutt-1.5.6i.tar.gz' saved [2910730/2910730]

>>> Resuming download...
>>> Downloading http://gentoo.chem.wisc.edu/gentoo/distfiles/mutt-1.5.6i.tar.gz
--10:27:30-- http://gentoo.chem.wisc.edu/gentoo/distfiles/mutt-1.5.6i.tar.gz
=> `/mutt-1.5.6i.tar.gz'
Resolving localhost... 127.0.0.1
Connecting to localhost[127.0.0.1]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 2,910,730

100%[====================================>] 2,910,730 --.--K/s

10:27:30 (38.12 MB/s) - `/mutt-1.5.6i.tar.gz' saved [2910730/2910730]

>>> Resuming download...
>>> Downloading http://mirrors.tds.net/gentoo/distfiles/mutt-1.5.6i.tar.gz
--10:27:30-- http://mirrors.tds.net/gentoo/distfiles/mutt-1.5.6i.tar.gz
=> `/mutt-1.5.6i.tar.gz'
Resolving localhost... 127.0.0.1
Connecting to localhost[127.0.0.1]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 2,910,730

100%[====================================>] 2,910,730 --.--K/s

10:27:30 (32.61 MB/s) - `/mutt-1.5.6i.tar.gz' saved [2910730/2910730]

>>> Resuming download...
>>> Downloading ftp://ftp.mutt.org/mutt/devel/mutt-1.5.6i.tar.gz
--10:27:30-- ftp://ftp.mutt.org/mutt/devel/mutt-1.5.6i.tar.gz
=> `/mutt-1.5.6i.tar.gz'
Resolving ftp.mutt.org... 82.165.34.161
Connecting to ftp.mutt.org[82.165.34.161]:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD /mutt/devel ... done.
==> PASV ... done. ==> RETR mutt-1.5.6i.tar.gz ... done.
Length: 2,910,730 (unauthoritative)

100%[====================================>] 2,910,730 361.11K/s ETA 00:00

10:27:40 (339.14 KB/s) - `/mutt-1.5.6i.tar.gz' saved [2910730]

!!! Couldn't download mutt-1.5.6i.tar.gz. Aborting.

!!! Fetch for /usr/portage/mail-client/mutt/mutt-1.5.6-r3.ebuild failed, continu
ing...



!!! Some fetch errors were encountered. Please see above for details.


So what am I doing wrong? One thing I did notice. If I try to emerge mutt on one of the "clients" it downloads it from the "server" and works fine. But I can't emerge it on the server. What is the problem?

Also, one scenario I also have is that one of my clients is a desktop and it's not always on the network. Is there an easy way to make it not try to use the server when its no on the network? Maybe comment something out or something? Or maybe write into it, that if the server is missing try something else?
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Wed Sep 15, 2004 1:19 pm    Post subject: Reply with quote

MattSharp wrote:
I am trying to use http-replicator.
...
I commented out the http_proxy line
So what am I doing wrong?


If your using the standard cache and distfile locations, try this:
Code:

mv /var/cache/http-replicator/*  /usr/portage/distfiles/
/usr/bin/repcacheman
rm /usr/portage/distfiles/*


MattSharp wrote:

Also, one scenario I also have is that one of my clients is a desktop and it's not always on the network. Is there an easy way to make it not try to use the server when its no on the network?


What I do is use Quickswitch for my laptop (emerge quickswitch). Quickswitch is designed to allow multiple configs for different networks with different services. I have a setting that switches my /etc/make.conf when I move off my lan.

More info here: http://www.newsforge.com/article.pl?sid=01/12/22/2118213&mode=thread

Homepage http://muthanna.com/quickswitch

How to change on boot with gentoo https://forums.gentoo.org/viewtopic.php?t=96281&highlight=quickswitch+boot
Back to top
View user's profile Send private message
Master One
l33t
l33t


Joined: 25 Aug 2003
Posts: 754
Location: Austria

PostPosted: Wed Sep 15, 2004 4:51 pm    Post subject: Reply with quote

That's so great! Also I was fighting with some strange problem, now it seems to be working just fine.

Nevertheless someone please tell me:
Quote:
Don't forget that portage needs mirrors! Edit GENTOO_MIRRORS in /etc/make.conf to add more http mirrors and place any ftp mirrors LAST. The default mirrors in gentoo leave something to be desired Smile Use mirrorselect if you need help in selecting mirrors.

Is GENTOO_MIRRORS to be set on all machines (server+workstations), or only on the server???

Quote:
Also, some packages in portage have a RESTRICT="nomirror" option which will prevent portage from checking replicator for those packages. The following will override this behavior. Create the file "/etc/portage/mirrors" containing: local http://gentoo.oregonstate.edu

Is this mirrors file to be created on all machines (server+workstations), or only on the server???

I find this mirrors thing a little confusing, because why should the workstations bother about that, if only the server is intended to download and cache the distfiles.

My last question: Why is nothing showing up in the http-replicator log-file??? I can see the file gets generated on startup, but until now I could never find any info in there, also I did some successfull downloads (indeed the file has always a size of zero).
_________________
Las torturas mentales de la CIA
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Wed Sep 15, 2004 8:57 pm    Post subject: Reply with quote

The mirrors need to be specified for every user of the proxy, since a request is sent to the proxy for a file on a particular mirror, and the proxy 'does it's thing' - it's meant to be fairly transparent to the client. I use the same set of mirrors for every node in the network, including the server.

The clients need the mirrors because the server allows the client to download the file 'by proxy'.

As for the log file - check the permissions are OK.
Back to top
View user's profile Send private message
Master One
l33t
l33t


Joined: 25 Aug 2003
Posts: 754
Location: Austria

PostPosted: Fri Sep 17, 2004 1:10 pm    Post subject: Reply with quote

meowsqueak wrote:
The mirrors need to be specified for every user of the proxy, since a request is sent to the proxy for a file on a particular mirror, and the proxy 'does it's thing' - it's meant to be fairly transparent to the client. I use the same set of mirrors for every node in the network, including the server. The clients need the mirrors because the server allows the client to download the file 'by proxy'. As for the log file - check the permissions are OK.

Ok, I've set GENTOO_MIRRORS and /etc/portage/mirrors now on all machines (server + workstations).

It's definitely working fine, but I still could not find out, why the logfile keeps beeing empty. I tried it with chown root:root and portage:portage, but this does not change anything (in http-replicator.conf there is explicitely mentioned, that no write access has to be granted anyway). The logfile definitely gets created on startup, if I delete it before, but until now, nothing showed up in there... :?
_________________
Las torturas mentales de la CIA
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Fri Sep 17, 2004 2:40 pm    Post subject: Reply with quote

Master One wrote:

It's definitely working fine, but I still could not find out, why the logfile keeps beeing empty.


The stable version of http-replicator writes the log file in a very lazy way. It won't actually write the log entries to disk till you fill the buffer. It uses less system resources this way on a busy server but it can take a while to flush to disk on a home lan :-) This has been changed in the latest version of http-replicator which flushes every log entry to disk.

So if your just trying to make sure your setup is OK, it probably is. If you want to see the logs now, just use the telnet monitor and you can see the messages as they happen! Check your config to make sure it is enabled and then just telnet to the port to see any messages in real time!
Back to top
View user's profile Send private message
Master One
l33t
l33t


Joined: 25 Aug 2003
Posts: 754
Location: Austria

PostPosted: Fri Sep 17, 2004 4:44 pm    Post subject: Reply with quote

Thanx, flybynite, as everthing is working, I just wanted to check, if the log confirms that. The less resources it takes, the better. Hopefully it will make it into portage soon now, so that I can benefit from the ongoing development.
_________________
Las torturas mentales de la CIA
Back to top
View user's profile Send private message
jdoe
Tux's lil' helper
Tux's lil' helper


Joined: 02 Jun 2003
Posts: 122
Location: Italy - Florence - Empoli

PostPosted: Sat Sep 18, 2004 3:27 pm    Post subject: Reply with quote

hi..
i'm trying to set up http-rep, for my 3 client lan.. I followed the how-to, but i have some problem on client side.
On the server everything works fine, i can see downloading at 15MB/s a cached file.
On client side i have this error:

Code:
Calculating dependencies ...done!
>>> emerge (1 of 1) net-fs/samba-3.0.7-r1 to /
>>> Resuming download...
>>> Downloading http://samba.idealx.org/dist/smbldap-tools-0.8.5.tgz
--17:39:32--  http://samba.idealx.org/dist/smbldap-tools-0.8.5.tgz
           => `/usr/portage/distfiles/distfiles/smbldap-tools-0.8.5.tgz'
Resolving slave... 192.168.1.3
Connecting to slave[192.168.1.3]:12000... connected.
Proxy request sent, awaiting response... 503 Service Unavailable
17:39:52 ERROR 503: Service Unavailable.

!!! Couldn't download smbldap-tools-0.8.5.tgz. Aborting.


here is conf files:
client make.conf:
Code:
GENTOO_MIRRORS="http://www.die.unipd.it/pub/Linux/distributions/gentoo-sources/ http://sunsite.cnlab-switch.ch/ftp/mirror/gentoo/"
http_proxy="http://slave:12000"
RESUMECOMMAND="/usr/bin/wget -t 5 --passive-ftp \${URI} -O \${DISTDIR}/\${FILE}"


server make.conf
Code:

http_proxy="http://127.0.0.1:12000"
RESUMECOMMAND="/usr/bin/wget -t 5 --passive-ftp \${URI} -O \${DISTDIR}/\${FILE}"
GENTOO_MIRRORS="http://www.die.unipd.it/pub/Linux/distributions/gentoo-sources/ http://sunsite.cnlab-switch.ch/ftp/mirror/gentoo/"


/etc/http-replicator.conf
Code:

PORT = 12000
IP = ['127.0.0.1','192.168.1.2','192.168.1.3']


where i wrong?

Thanks, jdoe
_________________
me
Back to top
View user's profile Send private message
lisa
Retired Dev
Retired Dev


Joined: 01 Jun 2003
Posts: 273
Location: York, UK again! Horray!

PostPosted: Sat Sep 18, 2004 7:42 pm    Post subject: Reply with quote

For distfile sharing I've set up Samba. It's way easier. For Rsync I have my server set up as a rsync mirror for all of my internal Gentoo machines.
_________________
Distcc guide
Visit my website
I maintain Distcc, Ccache, Memcached, and some others (i think)
Back to top
View user's profile Send private message
BlinkEye
Veteran
Veteran


Joined: 21 Oct 2003
Posts: 1046
Location: Gentoo Forums

PostPosted: Sat Sep 18, 2004 8:52 pm    Post subject: Reply with quote

jdoe wrote:
server make.conf
Code:

http_proxy="http://127.0.0.1:12000"
RESUMECOMMAND="/usr/bin/wget -t 5 --passive-ftp \${URI} -O \${DISTDIR}/\${FILE}"
GENTOO_MIRRORS="http://www.die.unipd.it/pub/Linux/distributions/gentoo-sources/ http://sunsite.cnlab-switch.ch/ftp/mirror/gentoo/"


/etc/http-replicator.conf
Code:

PORT = 12000
IP = ['127.0.0.1','192.168.1.2','192.168.1.3']


where i wrong?

Thanks, jdoe


change your http_proxy="http://127.0.0.1:12000" to either your external ip or to 192.168.1.3 (which would be your lan ip), i.e. to http_proxy="http://192.168.1.3:12000
_________________
Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Mon Sep 20, 2004 9:31 pm    Post subject: Reply with quote

lisa wrote:
For distfile sharing I've set up Samba.


That doesn't solve the concurrency problem either, does it?
Back to top
View user's profile Send private message
jdoe
Tux's lil' helper
Tux's lil' helper


Joined: 02 Jun 2003
Posts: 122
Location: Italy - Florence - Empoli

PostPosted: Tue Sep 21, 2004 6:38 am    Post subject: Reply with quote

BlinkEye wrote:

change your http_proxy="http://127.0.0.1:12000" to either your external ip or to 192.168.1.3 (which would be your lan ip), i.e. to http_proxy="http://192.168.1.3:12000


thanks, it works :)
It' really a nice tool...
_________________
me
Back to top
View user's profile Send private message
fritz
n00b
n00b


Joined: 15 Aug 2004
Posts: 7

PostPosted: Sun Oct 03, 2004 4:16 pm    Post subject: cpu usage Reply with quote

it's working but cpu usage is ~25% continuously on server while downloading with single client
doesnt' matter if the file is already in the cache or not

think this is rather high for just downloading, am i the only one with this problem or is this lack of effiency in http-replicator?

cpu is:
model name : AMD Duron(tm) Processor
cpu MHz : 1016.568
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3 ... 5, 6, 7 ... 22, 23, 24  Next
Page 6 of 24

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum