Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
HOWTO:Download Cache for your LAN-Http-Replicator (ver 3.0)
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3 ... 8, 9, 10 ... 22, 23, 24  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
BlinkEye
Veteran
Veteran


Joined: 21 Oct 2003
Posts: 1046
Location: Gentoo Forums

PostPosted: Sat Oct 30, 2004 10:14 am    Post subject: Reply with quote

i do have a problem: repcacheman runs every 5 minutes but even though it should have nothing to do (i.e. i didn't download any packages and /usr/portage/distfiles/ is empy) it eats a lot of my precious memory (my server is an oldie):
Code:
  PID  PPID  UID USER     RUSER    TTY         TIME+  %CPU %MEM S COMMAND
4174  4163    0 root     root     ?          0:28.75 10.6 12.7 R repcacheman

how could i prevent that?
_________________
Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Sat Oct 30, 2004 4:24 pm    Post subject: Reply with quote

The release of portage 2.0.51 has caused a number of problems.

These problems haven't affected how well http-replicator itself runs, only with the support script repcacheman.

I have updated ebuilds with changes to support portage 2.0.51. I need help testing:

http://www.updatedlinux.com/replicator/http-replicator-flybynite-1.6a.tar.bz2

This release contains 5 versions:
* net-misc/http-replicator :
[ ] 2.0-r2 (0) OVERLAY
[ ] 2.0-r3 (0) OVERLAY
[ ] 2.1_rc3 (0) OVERLAY
[ ] 2.1_rc3-r1 (0) OVERLAY
[ I] 2.1 (0) OVERLAY


2.0-r2 and 2.1_rc3 DEPEND on portage 2.0.50 and won't work with the current stable portage 2.0.51 I left these in only for the 1 or 2 gentooers who don't like to upgrade :-)

2.0-r3 , 2.1_rc3_r1 and 2.1 will work with any current portage version. 2.1 primarily adds external proxy support and improved logging.

2.1 is the latest and is stable, but the changes aren't fully documented in the howto yet.

Follow the howto at the start of this thread for all versions. Then If you upgraded to 2.1 -look at these changes:

**Note The config file location has changed in 2.1. Edit /etc/conf.d/http-replicator to add your external proxy and check other defaults.

**Note repcacheman has changed in 2.1. If you changed the default cache dir or the default user, you must call repcacheman with those options.
Code:

repcacheman --user USER --dir /path/to/cache


Last edited by flybynite on Sat Oct 30, 2004 9:56 pm; edited 2 times in total
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Sat Oct 30, 2004 4:27 pm    Post subject: Reply with quote

BlinkEye wrote:

how could i prevent that?


Fix is in the above post....
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Sat Oct 30, 2004 4:47 pm    Post subject: Reply with quote

I had a question about repcacheman and what it does and how often it should be run.

Repcacheman only needs to run after emerge is run on the server ( except emerge sync)

Here is some more info on repcacheman:
Quote:


Replicator can't share the same dir with portage because portage doesn't play nice.

This means only on the server, emerged packages exist in both replicator's cache and /distfiles. The files in /distfiles are wasted space and dups. One replicator function is to delete the dups in distfiles.

The other function of repcacheman is to move any new files from /distfiles to replicators cache. The only way to get files in /distfiles that aren't in replicators cache is by ftp, since replicator doesn't support ftp. So if you run emerge on the server and the package isn't on any http mirror portage will fall back to ftp and retrieve the file. Running repcacheman will check the md5 and then move the new file into the cache.

The only time files are ftp'd into /distfiles is after emerge is running on the server.

The only time dups occur is after running emerge on the server.

The only time repcacheman needs to run is after any emerge's on the server.

The best option is to run emerge's on the server like this:
Code:

emerge -uDva world && repcacheman


This will update world including dep's - asking if you won't to continue - and then run repcacheman after the emerge is finished.

Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Mon Nov 01, 2004 4:33 am    Post subject: Reply with quote

I upgraded to 2.1 but I'm getting this problem on one of my LANs. The following spews out about 20 times:

Code:
...
Connecting to 10.16.10.224:8080... connected.
Proxy request sent, awaiting response... 301 Moved Permanently
Location: http://gentoo.osuosl.org/distfiles/fontconfig-2.2.3.tar.gz [following]
--17:30:43--  http://gentoo.osuosl.org/distfiles/fontconfig-2.2.3.tar.gz
           => `/usr/portage/distfiles/fontconfig-2.2.3.tar.gz'
Connecting to 10.16.10.224:8080... connected.
Proxy request sent, awaiting response... 301 Moved Permanently
Location: http://gentoo.osuosl.org/distfiles/fontconfig-2.2.3.tar.gz [following]
...
20 redirections exceeded.
!!! Couldn't download fontconfig-2.2.3.tar.gz. Aborting.


Funnily enough, it works fine on another LAN I look after. I'm using an external proxy too, but that IP address (10.16.10.224) is the address of my http-replicator server, and I'm running 'emerge -Duav world' on the same host, so it's trying to connect to itself and then failing. My conf file includes '--ip 10.*.*.*'
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Mon Nov 01, 2004 8:12 pm    Post subject: Reply with quote

Ok, I did some reading into the 301 response:

http://www.checkupdown.com/status/E301.html

According to the output, wget first tries to download http://gentoo.oregonstate.edu/distfiles/fontconfig-2.2.3.tar.gz. The proxy request is sent and the response is a 301 error (presumably from the original server?). The Location specifies http://gentoo.osuosl.org/distfiles/fontconfig-2.2.3.tar.gz as the alternative URL, and wget follows this. But when it tries to download this, it gets another 301 with the location set to the exact same URL and the process repeats until the limit (20) is hit. So, is this a problem with this server in particular, or is it a problem with the http-replicator proxy? I suspect the server is fine because a manual wget of the osuosl.org file works fine (no followed location). Does http-replicator cache server responses or something?
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Wed Nov 03, 2004 6:56 pm    Post subject: Reply with quote

First, go ahead and update the link to http://gentoo.osuosl.org/ - since http://gentoo.oregonstate.edu/ is outdated anyway. I've updated the howto.

The response isn't cached by http-replicator- might be cached by your ISP or somewhere else.

Then if you still have this problem, upgrade to http-replicator 2.1, set debug in the config and post the logs...
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Wed Nov 03, 2004 8:26 pm    Post subject: Reply with quote

Hmmm, I wonder if my squid proxy is caching something - here's the log:

Code:
04 Nov 2004 09:23:19 DEBUG: HttpClient 170 connected to 10.16.10.224:42290
04 Nov 2004 09:23:19 DEBUG: HttpClient 170 received header:

  GET http://gentoo.oregonstate.edu/distfiles/fontconfig-2.2.3.tar.gz HTTP/1.0
  User-Agent: Wget/1.9
  Host: gentoo.oregonstate.edu
  Accept: */*

04 Nov 2004 09:23:19 INFO: HttpClient 170 received request for http://gentoo.oregonstate.edu/distfiles/fontconfig-2.2.3.tar.gz
04 Nov 2004 09:23:19 DEBUG: HttpClient 170 connecting to gentoo.oregonstate.edu
04 Nov 2004 09:23:19 DEBUG: HttpServer 170 connected to 10.10.130.123:3128
04 Nov 2004 09:23:19 DEBUG: HttpServer 170 received header:

  HTTP/1.0 301 Moved Permanently
  Date: Wed, 03 Nov 2004 20:23:19 GMT
  Server: Apache/2.0.52 (Debian GNU/Linux)
  Location: http://gentoo.osuosl.org/distfiles/fontconfig-2.2.3.tar.gz
  Content-Length: 266
  Content-Type: text/html; charset=iso-8859-1
  X-Cache: MISS from squid-proxy
  Proxy-Connection: close

04 Nov 2004 09:23:19 DEBUG: HttpClient 170 closed



Why is the first GET still trying http://gentoo.oregonstate.edu ? I don't have that in make.conf any more. Emerge says

Code:
>>> Downloading http://gentoo.oregonstate.edu/distfiles/fontconfig-2.2.3.tar.gz


Is http://gentoo.mirrors.pair.com providing me with this perhaps? I tried:

Code:
$ GENTOO_MIRRORS="http://gentoo.osuosl.org" sudo emerge -a --oneshot fontconfig
...
>>> Downloading http://gentoo.oregonstate.edu/distfiles/fontconfig-2.2.3.tar.gz


Not sure I understand what is going on here...
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Thu Nov 04, 2004 6:56 pm    Post subject: Reply with quote

meowsqueak wrote:
Hmmm, I wonder if my squid proxy is caching something


Squid's a likely culprit, but lets fix the link first......

meowsqueak wrote:

Why is the first GET still trying http://gentoo.oregonstate.edu ? I don't have that in make.conf any more.
Not sure I understand what is going on here...


If you followed the howto, you put gentoo.oregonstate.edu into your /etc/portage/mirrors.

That "local" mirror is downloaded before the make.conf mirrors. I'd bet you just need to change the /etc/portage/mirrors....
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Thu Nov 04, 2004 9:30 pm    Post subject: Reply with quote

flybynite wrote:
That "local" mirror is downloaded before the make.conf mirrors. I'd bet you just need to change the /etc/portage/mirrors....


Bingo! I completely forgot about that file. Thanks for the help, I'm sorted now.
Back to top
View user's profile Send private message
Maxwell
Tux's lil' helper
Tux's lil' helper


Joined: 10 Dec 2003
Posts: 96

PostPosted: Mon Nov 08, 2004 9:33 pm    Post subject: Problems using it Reply with quote

Well, i suppose i must be doing something stupid, but i need some help.

I installed http_replicator as i should, but as i have an http proxy port=8080, i switched replicator's port to 5000. But it looks like http_replicator isn't used by my notebook, in my internal lan. the value "http_proxy" in make.conf is the ip and door of the server, correct?

Both my /usr/portage/distfiles and my /var/log/http-replicator are empty. The log file also doesn't report anything when i emerge a file at the server.

Suggestions?
_________________
Freedom works. Use it!
Linux, by Gentoo
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Mon Nov 08, 2004 9:50 pm    Post subject: Re: Problems using it Reply with quote

Maxwell wrote:
Both my /usr/portage/distfiles and my /var/log/http-replicator are empty. The log file also doesn't report anything when i emerge a file at the server.


/var/cache/http-replicator?

Did you start the http-replicator daemon?

Code:
 # /etc/init.d/http-replicator start
Back to top
View user's profile Send private message
Maxwell
Tux's lil' helper
Tux's lil' helper


Joined: 10 Dec 2003
Posts: 96

PostPosted: Mon Nov 08, 2004 11:03 pm    Post subject: Re: Problems using it Reply with quote

yes, i did start it

The output of lsof -i is that it is listening at port 5000.
But it doesn't get connected by any client...
_________________
Freedom works. Use it!
Linux, by Gentoo
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Wed Nov 10, 2004 12:56 am    Post subject: Reply with quote

First, I need to see the client's emerge output. Copy the output showing where the file is downloaded.
Back to top
View user's profile Send private message
Maxwell
Tux's lil' helper
Tux's lil' helper


Joined: 10 Dec 2003
Posts: 96

PostPosted: Wed Nov 10, 2004 10:16 am    Post subject: Re: Problem using it Reply with quote

Ok
In a client, when i emerge something, the download part output is:

Quote:

emerge (1 of 1) mail-client/mozilla-thunderbird-0.9 to /
>>> Downloading http://darkstar.ist.utl.pt/gentoo/distfiles/thunderbird-0.9-source.tar.bz2
--10:06:57-- http://darkstar.ist.utl.pt/gentoo/distfiles/thunderbird-0.9-source.tar.bz2
=> `/usr/portage/distfiles/thunderbird-0.9-source.tar.bz2'
Resolving proxy.d... x.x.x.x
Connecting to proxy.d[x.x.x.x]:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: 33,197,138 [application/x-bzip2]

0% [ ] 326,958 40.00K/s ETA 12:36


The client's make.conf part is:
Quote:

# Default fetch command (5 tries, passive ftp for firewall compatibility)
HTTP_PROXY="http://x.x.x.y:5000"
#FETCHCOMMAND="/usr/bin/wget -t 5 --passive-ftp \${URI} -P \${DISTDIR}"
RESUMECOMMAND="/usr/bin/wget -t 5 --passive-ftp \${URI} -O \${DISTDIR}/\${FILE}
"

GENTOO_MIRRORS="http://darkstar.ist.utl.pt/gentoo ..."



In the server i have the following in make.conf

Quote:

# PORTDIR_OVERLAY is a directory where local ebuilds may be stored without
# concern that they will be deleted by rsync updates. Default is not
# defined.
PORTDIR_OVERLAY="/usr/local/portage"

http_proxy="http://x.x.x.y:5000"

# Default fetch command (5 tries, passive ftp for firewall compatibility)
#FETCHCOMMAND="/usr/bin/wget -t 5 --passive-ftp \${URI} -P \${DISTDIR}"
RESUMECOMMAND="/usr/bin/wget -t 5 --passive-ftp \${URI} -O \${DISTDIR}/\${FILE}"



The proxy setting that appears when i emerge something in the client is defined in /etc/profile.

Can anyone find a problem?

Help would be much apreciated
_________________
Freedom works. Use it!
Linux, by Gentoo
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Wed Nov 10, 2004 9:18 pm    Post subject: Re: Problem using it Reply with quote

Maxwell wrote:

Connecting to proxy.d[x.x.x.x]:3128... connected.


This shows that the client portage is connecting to a proxy other than http-replicator. You must have another proxy set somewhere..


Maxwell wrote:

The client's make.conf part is:

HTTP_PROXY="http://x.x.x.y:5000"


This shows one reason why the other proxy is being used. http_proxy should be in lower case.

After you lower case the client and server http_proxy try the emerge again. If portage still uses the other proxy you need to find out where you have a proxy set to xxxx:3128.

This could be set in a couple of places.

Maxwell wrote:

The proxy setting that appears when i emerge something in the client is defined in /etc/profile.


Ok already seem to know where the other proxy is being set. It shouldn't surprise you that it is being used. I guess you don't know how to unset it.

One of the files in /etc/env.d probably contains the xxx:3128 proxy definition. Remove that definition. Then you need to run

Code:

env-update
source /etc/profile


You may need to completely log out and log back in again to reset all terminals.

Do you need to use the xxx:3128 proxy to get to the net? If you still need it to get to the net than put that proxy in /etc/conf.d/http-replicator.
Back to top
View user's profile Send private message
Maxwell
Tux's lil' helper
Tux's lil' helper


Joined: 10 Dec 2003
Posts: 96

PostPosted: Wed Nov 10, 2004 10:23 pm    Post subject: Re: Problem using it Reply with quote

Ok, big mistake!
http_proxy now is in lower case. Now i can connect to http_replicator!! (yes!)
But i need a proxy to connet to net. How i set http_replicator to use it? I've already set it in http_replicator.conf and it didn't fetch anything.

Thanks in advance
_________________
Freedom works. Use it!
Linux, by Gentoo
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Wed Nov 10, 2004 10:37 pm    Post subject: Reply with quote

In /etc/conf.d/http-replicator (not /etc/http-replicator.conf) you simply add a line that looks like this:

Code:
DAEMON_OPTS="$DAEMON_OPTS --external 192.168.0.1:3128"


I'm not sure how to deal with authentication, if required.
Back to top
View user's profile Send private message
Maxwell
Tux's lil' helper
Tux's lil' helper


Joined: 10 Dec 2003
Posts: 96

PostPosted: Thu Nov 11, 2004 2:45 pm    Post subject: Re: Problem using it Reply with quote

Now i've done it!

I was using an ultra old version. It didn't had /etc/conf.d/http_replicator.conf file. Now it works as it should. If i find any problems i will tell you guys!

Thank you all!!
_________________
Freedom works. Use it!
Linux, by Gentoo
Back to top
View user's profile Send private message
johntramp
Guru
Guru


Joined: 03 Feb 2004
Posts: 457
Location: New Zealand

PostPosted: Mon Nov 15, 2004 10:39 am    Post subject: Reply with quote

Hi. I am trying to set up Http-Replicator with not much success. At the moment I am just trying with a client and a server. I am sure I have something wrong with the make.conf's as I was a little confused with the wording of the guide.
Quote:
Add http_proxy="http://YourProxyHere.com:8080" Line replacing YourProxyHere.com with your proxy hostname or IP address.
What does it mean by "YourProxyHere" is that the address of the http-replicator server or my router? Also do I have to change the address of the gentoo_mirrors on the client to be the server of the replicator?
Here are my 2 make.conf's, first the server then the client.
Quote:
CHOST="i686-pc-linux-gnu"
CFLAGS="-O2 -march=pentium2 -fomit-frame-pointer -pipe"
CXXFLAGS="${CFLAGS}"
MAKEOPTS="-j3"
#FEATURES="distcc"
USE="-gtk -x -alsa -oss -gnome -kde -qt -gtk2 -dvd network"
GENTOO_MIRRORS="http://gentoo.recoil.net.nz/gentoo/ ftp://debian.co.nz/Gentoo/ ftp://203.96.92.91/dist/gentoo/"
SYNC="rsync://gentoo.recoil.net.nz/gentoo-portage"
PORTDIR_OVERLAY=/usr/local/portage

# Default fetch command (5 tries, passive ftp for firewall compatibility)
http_proxy="http://192.168.1.33:8080"
#FETCHCOMMAND="/usr/bin/wget -t 5 --passive-ftp \${URI} -P \${DISTDIR}"
RESUMECOMMAND=" /usr/bin/wget -t 5 --passive-ftp \${URI} -O \${DISTDIR}/\${FILE}"



Quote:
CFLAGS="-O3 -march=athlon-xp -pipe -fomit-frame-pointer"
CHOST="i686-pc-linux-gnu"
CXXFLAGS="${CFLAGS}"
USE="X kde -gnome- 3dnow directfb alsa sse rtc network truetype nas mmx live fbdev divx4linux"

PORTAGE_NICENESS="5"
ACCEPT_KEYWORDS="~x86"
SYNC="rsync://192.168.1.33/gentoo-portage"
GENTOO_MIRRORS="http://192.168.1.33:8080/ ftp://203.96.92.91/dist/gentoo/" #http://gentoo.oregonstate.edu http://www.ibiblio.org/pub/Linux/distributions/gentoo"
LDFLAGS="-Wl,-O1"
MAKEOPTS="-j2"
PORTDIR_OVERLAY=/usr/local/portage

# Default fetch command (5 tries, passive ftp for firewall compatibility)
http_proxy="http://192.168.1.33:8080"
#FETCHCOMMAND="/usr/bin/wget -t 5 --passive-ftp \${URI} -P \${DISTDIR}"
RESUMECOMMAND=" /usr/bin/wget -t 5 --passive-ftp \${URI} -O \${DISTDIR}/\${FILE}"


If anyone can point me in the right direction it would be appreciated :)

Thanks
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Mon Nov 15, 2004 10:15 pm    Post subject: Reply with quote

YourProxyHere.com:Port is the IP address or DNS name of your http-replicator server and the access port. You set this on all client machines that you want to use the replicator. You can set this on the http-replicator server too, so that it uses itself to download source.

Keep the mirrors as is, just make sure they are http mirrors. The client makes a request for a file from one of your mirrors via the http-replicator proxy which then downloads and caches the file (or retrieves it from the cache if it's already downloaded), as well as relaying the incoming file to the client.

I hope this makes more sense?
Back to top
View user's profile Send private message
johntramp
Guru
Guru


Joined: 03 Feb 2004
Posts: 457
Location: New Zealand

PostPosted: Tue Nov 16, 2004 1:07 am    Post subject: Reply with quote

so there is no difference between the server and client's make.conf?
Back to top
View user's profile Send private message
JayBee
n00b
n00b


Joined: 10 Nov 2002
Posts: 26
Location: Christchurch, New Zealand

PostPosted: Tue Nov 16, 2004 2:53 am    Post subject: Reply with quote

Not sure if this has been covered in the thread - too long to wade through, and if so, could it be put near the front?

When I start an emerge from a client machine, I get a 401 not found error (from memory - away from the box at the moment), and it tries from all the mirrors it has listed. It appears that http-replicator is requesting the file on the server and downloading it, but the client isn't waiting to receive from the server.

Any ideas as to how to solve this?

Cheers
_________________
"There is no patch for stupidity"
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Tue Nov 16, 2004 3:31 am    Post subject: Reply with quote

johntramp wrote:
so there is no difference between the server and client's make.conf?


Indeed - no difference in respect to http-replicator. Consider the http-replicator daemon (or service, or server, or provider) to be independent from the host it actually runs on. A client (or a server) is a program rather than a host, it just happens to run on a host. So wget on the http-replicator machine is actually a client to a server on the same host. It's still client-server, just doesn't go through the external network interface.


Last edited by meowsqueak on Tue Nov 16, 2004 8:06 pm; edited 1 time in total
Back to top
View user's profile Send private message
johntramp
Guru
Guru


Joined: 03 Feb 2004
Posts: 457
Location: New Zealand

PostPosted: Tue Nov 16, 2004 10:48 am    Post subject: Reply with quote

ah :) I understand now, thanks meowsqueak.
Q: what will happen if I set this up but the computer with the daemon on is offline? Will my computer just emerge as it would with out the replicator? - also what will happen when it needs to use the ftp?
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3 ... 8, 9, 10 ... 22, 23, 24  Next
Page 9 of 24

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum