Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
HOWTO:Download Cache for your LAN-Http-Replicator (ver 3.0)
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3 ... 17, 18, 19 ... 22, 23, 24  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
Haakon
n00b
n00b


Joined: 16 Jan 2006
Posts: 22

PostPosted: Fri Apr 21, 2006 4:42 am    Post subject: Reply with quote

Okies.... I did the command:
Code:

source /etc/conf.d/http-replicator
/usr/bin/http-replicator

on the server.

On the client, a diferent client than the first, I typed this command as you asked:
Code:

http_proxy='192.168.x.1:8080' wget http://gentoo.chem.wisc.edu/gentoo/distfiles/tzdata2006a.tar.gz

The "x" is still the same variable as it has always been in my censors. :)

Now comes the fun part. This is what the client spat out:
Code:

localclient etc # http_proxy='192.168.x.1:8080' wget http://gentoo.chem.wisc.edu/gentoo/distfiles/tzdata2006a.tar.gz
--20:11:24--  http://gentoo.chem.wisc.edu/gentoo/distfiles/tzdata2006a.tar.gz
           => `tzdata2006a.tar.gz'
Connecting to 192.168.x.1:8080... connected.
Proxy request sent, awaiting response... No data received.
Retrying.

--20:11:25--  http://gentoo.chem.wisc.edu/gentoo/distfiles/tzdata2006a.tar.gz
  (try: 2) => `tzdata2006a.tar.gz'
Connecting to 192.168.x.1:8080... connected.
Proxy request sent, awaiting response... Read error (Connection reset by peer) in headers.
Retrying.

--20:11:27--  http://gentoo.chem.wisc.edu/gentoo/distfiles/tzdata2006a.tar.gz
  (try: 3) => `tzdata2006a.tar.gz'
Connecting to 192.168.x.1:8080... connected.
Proxy request sent, awaiting response... Read error (Connection reset by peer) in headers.
Retrying.



I broke the command after the 8th try, but I didn't think it was necessary to post all 8 retries of the same thing. Three spams of the retries is enough. :P

Now, at the same time, this is what the server was seeing:
Code:

localserver conf.d # source /etc/conf.d/http-replicator
localserver conf.d # /usr/bin/http-replicator
INFO: HttpReplicator started
WARNING: HttpReplicator blocked incoming request from 192.168.x.2:39424
WARNING: HttpReplicator blocked incoming request from 192.168.x.2:39425
WARNING: HttpReplicator blocked incoming request from 192.168.x.2:39426
WARNING: HttpReplicator blocked incoming request from 192.168.x.2:39427
WARNING: HttpReplicator blocked incoming request from 192.168.x.2:39428
WARNING: HttpReplicator blocked incoming request from 192.168.x.2:39429
WARNING: HttpReplicator blocked incoming request from 192.168.x.2:39430
WARNING: HttpReplicator blocked incoming request from 192.168.x.2:39431
INFO: HttpReplicator terminated
localserver conf.d #



"Localserver", "localclient", and "x" are censors again. The x.2 ip is the ip of the client that is trying to make the contact. I can to the same thing with another client and get the same thing, just the last octet changes.

I also tried your other download selection and tried this command:
Code:

http_proxy='192.168.x.1:8080' wget http://gentoo.osuosl.org/distfiles/netcat-110-patches-1.0.tar.bz2


This command produces the same output as the first file we tried to download.

At this point, I took it upon my self to have the server download the file of the first command manually with out http-replicator trying to download it:
Code:

localserver conf.d # wget http://gentoo.chem.wisc.edu/gentoo/distfiles/tzdata2006a.tar.gz
--19:10:42--  http://gentoo.chem.wisc.edu/gentoo/distfiles/tzdata2006a.tar.gz
           => `tzdata2006a.tar.gz'
Resolving gentoo.chem.wisc.edu... 128.104.70.13
Connecting to gentoo.chem.wisc.edu|128.104.70.13|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 149,612 (146K) [application/x-tar]

100%[========================================================================>] 149,612      121.08K/s

19:10:44 (120.85 KB/s) - `tzdata2006a.tar.gz' saved [149612/149612]



I then moved it to the http-replicator cache and gave it the same permissions of all the other files in the cache:
Code:

localserver conf.d # mv tzdata2006a.tar.gz /var/cache/http-replicator/
localserver conf.d # cd /var/cache/http-replicator/
localserver http-replicator # ls -l tzdata2006a.tar.gz
-rw-r--r--  1 root root 149612 Jan 31 23:01 tzdata2006a.tar.gz
localserver http-replicator # ls -l zip23.tar.gz
-rw-r--r--  1 portage portage 723283 Apr 13 21:18 zip23.tar.gz
localserver http-replicator # chown portage:portage tzdata2006a.tar.gz
localserver http-replicator # ls -l tzdata2006a.tar.gz
-rw-r--r--  1 portage portage 149612 Jan 31 23:01 tzdata2006a.tar.gz


I then tried the original command you suggested to download the file "tzdata2006a.tar.gz", but the errors are identical to what we see above for both the client and the server.

The computer that http-replicator is on has two ethernet ports, would that be a problem? This computer acts as my gateway to the Internet, so ipchains and masquerading is in use. I will also note, that turning off ipchains didn't change the outcome. I still got the same errors we see above.

As near as I can see, it doesn't make a difference what server or file combination I use, I will get the same error when I attempted different combinations.


Last edited by Haakon on Tue Apr 25, 2006 2:57 pm; edited 1 time in total
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Sat Apr 22, 2006 6:19 am    Post subject: Reply with quote

oops, so sorry. My bad...


The error is obvious but not the one your having. I meant to say start replicator like this:

Code:

source /etc/conf.d/http-replicator
 /usr/bin/http-replicator $DAEMON_OPTS


This will pass the options correctly and should eliminate the security warning. Please retry the test...
Back to top
View user's profile Send private message
Haakon
n00b
n00b


Joined: 16 Jan 2006
Posts: 22

PostPosted: Sat Apr 22, 2006 6:23 pm    Post subject: Reply with quote

Hehe not a problem. Do I win Simon Says? j/k :lol:

Oh, and I will be using the normal censors in the last post. :)


Ok, I ran the revised commands as you said:
Code:

localserver ~ # source /etc/conf.d/http-replicator
localserver ~ # /usr/bin/http-replicator $DAEMON_OPTS


I then ran these commands to download three different files from three different mirrors on the client:

Code:

localclient ~ # http_proxy='192.168.x.1:8080' wget http://gentoo.chem.wisc.edu/gentoo/distfiles/tzdata2006a.tar.gz
localclient ~ # http_proxy='192.168.x.1:8080' wget http://gentoo.osuosl.org/distfiles/netcat-110-patches-1.0.tar.bz2
localclient ~ # http_proxy='192.168.x.1:8080' wget http://distro.ibiblio.org/pub/linux/distributions/gentoo/distfiles/gcc-3.4.5-patches-1.4.tar.bz2


Now, for the output spam...

Here is what the client had seen:
Code:

localclient ~ # http_proxy='192.168.x.1:8080' wget http://gentoo.chem.wisc.edu/gentoo/distfiles/tzdata2006a.tar.gz
--10:03:05--  http://gentoo.chem.wisc.edu/gentoo/distfiles/tzdata2006a.tar.gz
           => `tzdata2006a.tar.gz.1'
Connecting to 192.168.x.1:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 149,612 (146K) [application/x-tar]

100%[=======================================================================>] 149,612      134.87K/s

10:03:07 (134.60 KB/s) - `tzdata2006a.tar.gz.1' saved [149612/149612]

localclient ~ # http_proxy='192.168.x.1:8080' wget http://gentoo.osuosl.org/distfiles/netcat-110-patches-1.0.tar.bz2
--10:03:21--  http://gentoo.osuosl.org/distfiles/netcat-110-patches-1.0.tar.bz2
           => `netcat-110-patches-1.0.tar.bz2.1'
Connecting to 192.168.x.1:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 25,751 (25K) [application/x-tar]

100%[=======================================================================>] 25,751        54.53K/s

10:03:22 (54.37 KB/s) - `netcat-110-patches-1.0.tar.bz2.1' saved [25751/25751]

localclient ~ # http_proxy='192.168.x.1:8080' wget http://distro.ibiblio.org/pub/linux/distributions/gentoo/distfiles/gcc-3.4.5-patches-1.4.tar.bz2
--10:03:27--  http://distro.ibiblio.org/pub/linux/distributions/gentoo/distfiles/gcc-3.4.5-patches-1.4.tar.bz2
           => `gcc-3.4.5-patches-1.4.tar.bz2.1'
Connecting to 192.168.x.1:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 52,228 (51K) [application/x-bzip2]

100%[=======================================================================>] 52,228        25.49K/s

10:03:30 (25.42 KB/s) - `gcc-3.4.5-patches-1.4.tar.bz2.1' saved [52228/52228]

localclient ~ #



and the server output from the three command executed on the client:
Code:

localserver ~ # source /etc/conf.d/http-replicator
localserver ~ # /usr/bin/http-replicator $DAEMON_OPTS
INFO: HttpReplicator started
STAT: HttpClient 1 bound to 192.168.x.2
INFO: HttpClient 1 proxy request for http://gentoo.chem.wisc.edu/gentoo/distfiles/tzdata2006a.tar.gz
DEBUG: HttpClient 1 cache position: gentoo.chem.wisc.edu/gentoo/distfiles/tzdata2006a.tar.gz
DEBUG: HttpClient 1 connecting to gentoo.chem.wisc.edu
STAT: HttpServer 1 bound to gentoo.chem.wisc.edu
DEBUG: HttpServer 1 received header:

  GET /gentoo/distfiles/tzdata2006a.tar.gz HTTP/1.0
  host: gentoo.chem.wisc.edu
  connection: close
  accept: */*
  user-agent: Wget/1.10.2

INFO: HttpServer 1 serving file from remote host
DEBUG: HttpClient 1 received header:

  HTTP/1.1 200 OK
  content-length: 149612
  content-encoding: x-gzip
  accept-ranges: bytes
  server: Apache/2.0.54 (Gentoo/Linux)
  last-modified: Wed, 01 Feb 2006 08:01:12 GMT
  connection: close
  etag: "837a-2486c-155a4200"
  date: Sat, 22 Apr 2006 18:04:01 GMT
  content-type: application/x-tar

DEBUG: HttpServer 1 closed
STAT: HttpServer 1 sent 149612 bytes
INFO: HttpServer 1 cached gentoo.chem.wisc.edu/gentoo/distfiles/tzdata2006a.tar.gz
DEBUG: HttpClient 1 closed
STAT: HttpClient 1 received 149612 bytes
STAT: HttpClient 2 bound to 192.168.x.2
INFO: HttpClient 2 proxy request for http://gentoo.osuosl.org/distfiles/netcat-110-patches-1.0.tar.bz2
DEBUG: HttpClient 2 cache position: gentoo.osuosl.org/distfiles/netcat-110-patches-1.0.tar.bz2
DEBUG: HttpClient 2 connecting to gentoo.osuosl.org
STAT: HttpServer 2 bound to gentoo.osuosl.org
DEBUG: HttpServer 2 received header:

  GET /distfiles/netcat-110-patches-1.0.tar.bz2 HTTP/1.0
  host: gentoo.osuosl.org
  connection: close
  accept: */*
  user-agent: Wget/1.10.2

INFO: HttpServer 2 serving file from remote host
DEBUG: HttpClient 2 received header:

  HTTP/1.1 200 OK
  content-length: 25751
  accept-ranges: bytes
  server: Apache
  last-modified: Fri, 06 May 2005 23:03:43 GMT
  connection: close
  etag: "1170b561-6497-1a0251c0"
  date: Sat, 22 Apr 2006 18:04:10 GMT
  content-type: application/x-tar

DEBUG: HttpServer 2 closed
STAT: HttpServer 2 sent 25751 bytes
INFO: HttpServer 2 cached gentoo.osuosl.org/distfiles/netcat-110-patches-1.0.tar.bz2
DEBUG: HttpClient 2 closed
STAT: HttpClient 2 received 25751 bytes
STAT: HttpClient 3 bound to 192.168.x.2
INFO: HttpClient 3 proxy request for http://distro.ibiblio.org/pub/linux/distributions/gentoo/distfiles/gcc-3.4.5-patches-1.4.tar.bz2
DEBUG: HttpClient 3 cache position: distro.ibiblio.org/pub/linux/distributions/gentoo/distfiles/gcc-3.4.5-patches-1.4.tar.bz2
DEBUG: HttpClient 3 connecting to distro.ibiblio.org
STAT: HttpServer 3 bound to distro.ibiblio.org
DEBUG: HttpServer 3 received header:

  GET /pub/linux/distributions/gentoo/distfiles/gcc-3.4.5-patches-1.4.tar.bz2 HTTP/1.0
  host: distro.ibiblio.org
  connection: close
  accept: */*
  user-agent: Wget/1.10.2

INFO: HttpServer 3 serving file from remote host
DEBUG: HttpClient 3 received header:

  HTTP/1.1 200 OK
  content-length: 52228
  accept-ranges: bytes
  server: Apache/2.0.46 (Red Hat)
  last-modified: Tue, 07 Mar 2006 00:08:02 GMT
  connection: close
  etag: "1b32993-cc04-6fc7fc80"
  date: Sat, 22 Apr 2006 18:04:16 GMT
  content-type: application/x-bzip2

DEBUG: HttpServer 3 closed
STAT: HttpServer 3 sent 52228 bytes
INFO: HttpServer 3 cached distro.ibiblio.org/pub/linux/distributions/gentoo/distfiles/gcc-3.4.5-patches-1.4.tar.bz2
DEBUG: HttpClient 3 closed
STAT: HttpClient 3 received 52228 bytes



And all this again once those files were in cache:

Client:
Code:

localclient ~ # http_proxy='192.168.x.1:8080' wget http://gentoo.chem.wisc.edu/gentoo/distfiles/tzdata2006a.tar.gz
--10:10:03--  http://gentoo.chem.wisc.edu/gentoo/distfiles/tzdata2006a.tar.gz
           => `tzdata2006a.tar.gz.2'
Connecting to 192.168.x.1:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 149,612 (146K)

100%[=======================================================================>] 149,612       --.--K/s

10:10:03 (10.96 MB/s) - `tzdata2006a.tar.gz.2' saved [149612/149612]

localclient ~ # http_proxy='192.168.x.1:8080' wget http://gentoo.osuosl.org/distfiles/netcat-110-patches-1.0.tar.bz2
--10:10:09--  http://gentoo.osuosl.org/distfiles/netcat-110-patches-1.0.tar.bz2
           => `netcat-110-patches-1.0.tar.bz2.2'
Connecting to 192.168.x.1:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 25,751 (25K)

100%[=======================================================================>] 25,751        --.--K/s

10:10:09 (10.46 MB/s) - `netcat-110-patches-1.0.tar.bz2.2' saved [25751/25751]

localclient ~ # http_proxy='192.168.x.1:8080' wget http://distro.ibiblio.org/pub/linux/distributions/gentoo/distfiles/gcc-3.4.5-patches-1.4.tar.bz2
--10:10:21--  http://distro.ibiblio.org/pub/linux/distributions/gentoo/distfiles/gcc-3.4.5-patches-1.4.tar.bz2
           => `gcc-3.4.5-patches-1.4.tar.bz2.2'
Connecting to 192.168.x.1:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 52,228 (51K)

100%[=======================================================================>] 52,228        --.--K/s

10:10:25 (10.76 MB/s) - `gcc-3.4.5-patches-1.4.tar.bz2.2' saved [52228/52228]

localclient ~ #


Server:
Code:

localserver ~ # source /etc/conf.d/http-replicator
localserver ~ # /usr/bin/http-replicator $DAEMON_OPTS
INFO: HttpReplicator started
STAT: HttpClient 1 bound to 192.168.x.2
INFO: HttpClient 1 proxy request for http://gentoo.chem.wisc.edu/gentoo/distfiles/tzdata2006a.tar.gz
DEBUG: HttpClient 1 cache position: gentoo.chem.wisc.edu/gentoo/distfiles/tzdata2006a.tar.gz
DEBUG: HttpClient 1 checking modification since Sat Apr 22 10:04:01 2006
DEBUG: HttpClient 1 connecting to gentoo.chem.wisc.edu
STAT: HttpServer 1 bound to gentoo.chem.wisc.edu
DEBUG: HttpServer 1 received header:

  GET /gentoo/distfiles/tzdata2006a.tar.gz HTTP/1.0
  host: gentoo.chem.wisc.edu
  if-modified-since: Sat, 22 Apr 2006 18:04:01 GMT
  connection: close
  accept: */*
  user-agent: Wget/1.10.2

INFO: HttpServer 1 serving file from cache
DEBUG: HttpClient 1 received header:

  HTTP/1.1 200 OK
  date: Sat, 22 Apr 2006 18:10:59 GMT
  connection: close
  etag: "837a-2486c-155a4200"
  content-length: 149612
  server: Apache/2.0.54 (Gentoo/Linux)

DEBUG: HttpServer 1 closed
DEBUG: HttpClient 1 closed
STAT: HttpClient 1 received 149612 bytes
STAT: HttpClient 2 bound to 192.168.x.2
INFO: HttpClient 2 proxy request for http://gentoo.osuosl.org/distfiles/netcat-110-patches-1.0.tar.bz2
DEBUG: HttpClient 2 cache position: gentoo.osuosl.org/distfiles/netcat-110-patches-1.0.tar.bz2
DEBUG: HttpClient 2 checking modification since Sat Apr 22 10:04:10 2006
DEBUG: HttpClient 2 connecting to gentoo.osuosl.org
STAT: HttpServer 2 bound to gentoo.osuosl.org
DEBUG: HttpServer 2 received header:

  GET /distfiles/netcat-110-patches-1.0.tar.bz2 HTTP/1.0
  host: gentoo.osuosl.org
  if-modified-since: Sat, 22 Apr 2006 18:04:10 GMT
  connection: close
  accept: */*
  user-agent: Wget/1.10.2

INFO: HttpServer 2 serving file from cache
DEBUG: HttpClient 2 received header:

  HTTP/1.1 200 OK
  date: Sat, 22 Apr 2006 18:10:57 GMT
  connection: close
  etag: "2a8bb01-6497-1a0251c0"
  content-length: 25751
  server: Apache

DEBUG: HttpServer 2 closed
DEBUG: HttpClient 2 closed
STAT: HttpClient 2 received 25751 bytes
STAT: HttpClient 3 bound to 192.168.x.2
INFO: HttpClient 3 proxy request for http://distro.ibiblio.org/pub/linux/distributions/gentoo/distfiles/gcc-3.4.5-patches-1.4.tar.bz2
DEBUG: HttpClient 3 cache position: distro.ibiblio.org/pub/linux/distributions/gentoo/distfiles/gcc-3.4.5-patches-1.4.tar.bz2
DEBUG: HttpClient 3 checking modification since Sat Apr 22 10:04:16 2006
DEBUG: HttpClient 3 connecting to distro.ibiblio.org
STAT: HttpServer 3 bound to distro.ibiblio.org
DEBUG: HttpServer 3 received header:

  GET /pub/linux/distributions/gentoo/distfiles/gcc-3.4.5-patches-1.4.tar.bz2 HTTP/1.0
  host: distro.ibiblio.org
  if-modified-since: Sat, 22 Apr 2006 18:04:16 GMT
  connection: close
  accept: */*
  user-agent: Wget/1.10.2

INFO: HttpServer 3 serving file from cache
DEBUG: HttpClient 3 received header:

  HTTP/1.1 200 OK
  date: Sat, 22 Apr 2006 18:11:13 GMT
  connection: close
  etag: "1b32993-cc04-6fc7fc80"
  content-length: 52228
  server: Apache/2.0.46 (Red Hat)

DEBUG: HttpServer 3 closed
DEBUG: HttpClient 3 closed
STAT: HttpClient 3 received 52228 bytes



If I am putting too much information on the posts, let me know. I flinch everytime I drop debug output on a post. :)

Seems to work like a champ when running this way. I checked the location of the downloaded files and they are in directories according to the mirror from which they were downloaded in /var/cache/http-replicator.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Sat Apr 22, 2006 7:49 pm    Post subject: Reply with quote

Ok, everything worked normal.

That means replicator is working correctly. This test eliminated everything but your client setup, because portage uses wget to download files just like the test you did.

So wget can download everything ok, but portage calling wget creates an error. This really narrows it down. Probably the client config.

Find all the proxies and configs that affect the client first.

Here is a start, but may not be all the possible settings.

Code:

source /etc/make.conf
echo $FETCHCOMMAND
echo $RESUMECOMMAND
echo $PROXY
echo $ftp_proxy
echo $http_proxy



All but the last should be blank.

Are there any other proxies in use or set anywhere now or in the past?
Back to top
View user's profile Send private message
Haakon
n00b
n00b


Joined: 16 Jan 2006
Posts: 22

PostPosted: Sun Apr 23, 2006 1:38 am    Post subject: Reply with quote

Alrighty, I really apologize if this turns out to be something I totally over looked...

Here is the output you requested:
Code:

localserver etc # source /etc/make.conf
localserver etc # echo $FETCHCOMMAND

localserver etc # echo $RESUMECOMMAND

localserver etc # echo $PROXY

localserver etc # echo $ftp_proxy

localserver etc # echo $http_proxy
http://192.168.x.1:8080
localserver etc #


"x" is a censor. The number it repesents is the same number that all my computers have right now.

The computer that the http-replicator is installed on is the only thing close to a proxy that I have, and all the other computers go through it to get to the Internet. However, there is no proxy service installed. The gateway computer has a stateful firewall and that is what all the clients run through to get to the Internet. There hasn't ever been a proxy on my network.

Now, I did, at first, set "127.0.0.1" as the http_proxy variable within the /etc/make.conf file. I have, however, switched it to the 192.168.x.1 address that we have been seeing. That change didn't produce any different results.

When you are saying "client" in your last post, are you referring to the computer which has http-replicator installed? I made the assumption you did.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Sun Apr 23, 2006 5:16 am    Post subject: Reply with quote

The box that runs replicator I call the server. Your other boxes I call the clients. Client/server isn't always clear especially in replicators case, but I think of it that way.

I've assumed the errors you've had were from a box other than the one that runs replicator. I also assumed that the successfull wget test was from a client box that emerge will fail from.

It really doesn't matter much since the server box running replicator has the same setup as the clients. The only important info is do both server and clients boxes pass the wget test, and fail the emerge -f test in the same way.


We need to check your /etc/make.conf for an error. A single space or possibly an unmatched " or ' could cause errors that are hard to spot in these posts. Check the make.conf on both the server and the clients. I assume the clients are the same so we can work with just one make.conf from a clients and get that box working first.

Try renaming your make.conf make.conf.good and create a new make.conf with only the few lines that don't start with a # and see if we can narrow it down that way.

Posting the make.conf can help but it is hard to spot spaces and odd characters in posts.
Back to top
View user's profile Send private message
Haakon
n00b
n00b


Joined: 16 Jan 2006
Posts: 22

PostPosted: Sun Apr 23, 2006 7:34 am    Post subject: Reply with quote

flybynite wrote:

The box that runs replicator I call the server. Your other boxes I call the clients. Client/server isn't always clear especially in replicators case, but I think of it that way.

That is the way I have been seen it as well, except for the last post.

Here is where I got confused... I have activated http-replicator like this on the server:
Code:

source /etc/conf.d/http-replicator
/usr/bin/http-replicator $DAEMON_OPTS

and ran the command "emerge -uDf" on the client and it worked perfectly. The files began to fetch, no errors. Only when I tried bringing up http-replicator as a service did I have the problem, so I figured it was more the server's problem and not the client's because the client would be making an anonymous http request as if it where a web browser, right?

Right after I typed that I tried it out using a regular browser (Konquerer in this case) on the client and was not able to download the file "tzdata2006a.tar.gz" from http-replicator at the server. I did remember to put "192.168.x.1:8080" as the http proxy for Konquerer. When looking at the /var/log/http-replicator log on the server, I see it is the same error with Konquerer as it is with wget. The moment I tried to use wget as the same regular user that used the browser, it would fail with the same familiar error. Here is the wget command I used:
Code:

http_proxy='192.168.x.1:8080' wget http://gentoo.chem.wisc.edu/gentoo/distfiles/tzdata2006a.tar.gz


Before I ran all those experiments I recreated the /etc/make.conf file as you suggested first.

Here it is with the usual censors:
Code:

localclient ~ $ cat /etc/make.conf
USE="X gtk gtk2 3dnow dvd a52 aac avi cdr encode mp3 mpeg jpeg mozilla java javascript ogg oggvorbis oss samba svga usb gnome alsa gif kde opengl quicktime sse sse2 mmx svga acpi"
CHOST="i386-pc-linux-gnu"
CFLAGS="-mcpu=i586 -O3 -pipe"
CXXFLAGS="${CFLAGS}"
GENTOO_MIRRORS="http://gentoo.chem.wisc.edu/gentoo http://gentoo.osuosl.org http://www.ibiblio.org/pub/Linux/distributions/gentoo"
SYNC="rsync://192.168.x.1/gentoo-portage"
MAKEOPTS="-j2"
AUTOCLEAN="yes"
CCACHE_SIZE="1024M"
http_proxy="192.168.x.1:8080"
localclient ~ $


flybynite wrote:

I've assumed the errors you've had were from a box other than the one that runs replicator. I also assumed that the successfull wget test was from a client box that emerge will fail from.

You are right here, too. It has all been the clients that are unable to use the http-replicator, though I haven't tried using "emerge -f" with the server through http-replicator.

Again, right after I said this, I tried the server and got the same error as the clients. But you figured this to be the case already. :)
Back to top
View user's profile Send private message
Kobboi
l33t
l33t


Joined: 29 Jul 2005
Posts: 662
Location: Belgium

PostPosted: Sun Apr 23, 2006 10:10 pm    Post subject: Reply with quote

I have recently installed a server that functions as a local portage tree mirror (through rsyncd) and as a package file cache (through http-replicator). Now it would be cool if the server was aware of the packages installed on the clients in the network (their world files), so that, after the server syncs its portage tree, it could prefetch the package files that will be needed by one or more clients in case they wanted to "emerge -avuD world". Is there an elegant way to do this?
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Sun Apr 23, 2006 10:57 pm    Post subject: Reply with quote

Haakon wrote:

Here is where I got confused... I have activated http-replicator like this on the server:
Code:

source /etc/conf.d/http-replicator
/usr/bin/http-replicator $DAEMON_OPTS

and ran the command "emerge -uDf" on the client and it worked perfectly. The files began to fetch, no errors. Only when I tried bringing up http-replicator as a service did I have the problem


That sounds interesting. So replicator works when started on the command line, it doesn't work when started by /etc/init.d/http-replicator?


I'd like to see /etc/init.d/http-replicator. But again, spaces are hard to spot, so after posting, rename that file and emerge http-replicator again so it will reinstall a new one and see what happens.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Sun Apr 23, 2006 11:12 pm    Post subject: Reply with quote

Kobboi wrote:
Now it would be cool if the server was aware of the packages installed on the clients in the network (their world files)


The individual clients USE settings and the world file, and more affect what packages it needs. There is no way to know what use settings etc are on other boxes so there isn't an easy way to due this. Portage just doesn't work like this.


However, you're probably remotely running "emerge sync" on each client with a script, so you could just add "emerge -uDvf world" to that script and you get the same result.

So instead of trying to get replicator to magically know what every other client needs, just have all clients perform a fetch of what it needs when you sync them. You can control when the downloads happen if that is your goal. Replicator will still ensure that only one copy of each file is downloaded from the net no matter how many clients request that file, even if they all request the file at the same time or in sequence.
Back to top
View user's profile Send private message
Haakon
n00b
n00b


Joined: 16 Jan 2006
Posts: 22

PostPosted: Mon Apr 24, 2006 3:28 am    Post subject: Reply with quote

flybynite wrote:

That sounds interesting. So replicator works when started on the command line, it doesn't work when started by /etc/init.d/http-replicator?

I did a bit more work and found that it is really when http-replicator is ran as anything other than root. I had changed the --user option in /etc/conf.d/http-replicator to "root" from "portage". I executed /etc/init.d/http-replicator and ran a wget from the client and everything worked correctly. I tried changing the --user option to a regular user and I get the happy error we are accustomed to seeing. I modified the /var/cache/http-replicator permissions to allow anyone full control to make sure it wasn't a permission problem there. Once I found that wasn't it, I put the permissions back to normal. I tried running http-replicator from the command line as a regular user instead of root, and again, the same error.

flybynite wrote:

I'd like to see /etc/init.d/http-replicator. But again, spaces are hard to spot, so after posting, rename that file and emerge http-replicator again so it will reinstall a new one and see what happens.


I did the above action before I started to mess around with http-replicator. Nevertheless, here is the /etc/init.d/http-replicator file after the remerge :)
Code:

localserver ~ # cat /etc/init.d/http-replicator
#!/sbin/runscript
# Copyright 1999-2004 Gentoo Technologies, Inc.
# Distributed under the terms of the GNU General Public License v2
# $Header: /var/cvsroot/gentoo-x86/net-proxy/http-replicator/files/http-replicator-3.0.init,v 1.1 2005/06/02 06:33:24 griffon26 Exp $

depend() {
        need net
}

start() {
        ebegin "Starting Http-Replicator"
        start-stop-daemon --start --pidfile /var/run/http-replicator.pid --name http-replicator \
                --startas /usr/bin/http-replicator -- -s -f --pid /var/run/http-replicator.pid --daemon $DAEMON_OPTS
        eend $? "Failed to start Http-Replicator"
}

stop() {
        ebegin "Stopping Http-Replicator"
        start-stop-daemon --stop --pidfile /var/run/http-replicator.pid --name http-replicator \
                --signal 2 --oknodo
        eend $? "Failed to stop Http-Replicator"
}
localserver ~ #
Back to top
View user's profile Send private message
Kobboi
l33t
l33t


Joined: 29 Jul 2005
Posts: 662
Location: Belgium

PostPosted: Mon Apr 24, 2006 12:26 pm    Post subject: Reply with quote

flybynite wrote:
The individual clients USE settings and the world file, and more affect what packages it needs. There is no way to know what use settings etc are on other boxes so there isn't an easy way to due this. Portage just doesn't work like this.

I understand. I guess I was a little too eager in speeding things up. Thanks for your reply.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Mon Apr 24, 2006 10:28 pm    Post subject: Reply with quote

Haakon wrote:

I did a bit more work and found that it is really when http-replicator is ran as anything other than root.


Just to make sure I understand correctly. You started replicator in a terminal logged in as root, with the --user option as both root and portage? It works as --user root but not portage?

What do you get from this?

Code:

groups portage
ls -l /var/cache/
ls -l /var/log/http-replicator.log
ls -l /var/run/http-replicator.pid



Are you running any sort of security options on this server that would limit user abilities or access like pax, selinux, gresecurity, lids etc.?
Back to top
View user's profile Send private message
Haakon
n00b
n00b


Joined: 16 Jan 2006
Posts: 22

PostPosted: Mon Apr 24, 2006 10:44 pm    Post subject: Reply with quote

flybynite wrote:

That sounds interesting. So replicator works when started on the command line, it doesn't work when started by /etc/init.d/http-replicator?


I'd like to see /etc/init.d/http-replicator. But again, spaces are hard to spot, so after posting, rename that file and emerge http-replicator again so it will reinstall a new one and see what happens.


Ok, I am really starting to think there may be something wrong with the python install. I think the problem may have been a while ago when the original image of the tarball I use had a python update, I didn't do anything like execute "python-updater". I only learned about that function after you had me reinstall python. I got to see the warning message that would have otherwise been pushed off the buffer on a full world update.

Granted, it could be any number of dev packages that may not have been updated correctly.

Also, I ran http-replicator on a laptop I have at work that had Gentoo installed normally and http-replicator worked like a champ the first time.

So, I am starting to lean towards grabbing everything in my /etc directory on the server and doing a reload. However, if you want to explore the issue some more, I have no problems working this down until we find the exact problem.
Back to top
View user's profile Send private message
Haakon
n00b
n00b


Joined: 16 Jan 2006
Posts: 22

PostPosted: Mon Apr 24, 2006 11:13 pm    Post subject: Reply with quote

flybynite wrote:

Just to make sure I understand correctly. You started replicator in a terminal logged in as root, with the --user option as both root and portage? It works as --user root but not portage?


Yes, I started the replicator from a terminal as root by executing this command:
Code:

/etc/init.d/http-replicator start


No, I changed the --user option in the /etc/conf.d/http-replicator file from portage to root, so root would be the only thing after --user, besides the required syntax. Here is that line from the /etc/conf.d/http-replicator file:
Code:

GENERAL_OPTS="$GENERAL_OPTS --user root"


flybynite wrote:

What do you get from this?

Code:

groups portage
ls -l /var/cache/
ls -l /var/log/http-replicator.log
ls -l /var/run/http-replicator.pid



Sure thing:
Code:

localserver ~ # groups portage
portage
localserver ~ # ls -l /var/cache
total 48
-rw-------   1 root    root      136 Sep  4  2005 dhcpcd-eth0.cache
-rw-------   1 root    root      136 Jul 16  2005 dhcpcd-eth1.cache
drwxrwxr-x   3 root    portage  4096 Apr 23 17:11 edb
drwxrwxr-x   3 portage portage 20480 Apr 23 18:26 http-replicator
drwxrwxr-x  19 root    man      4096 Mar 10  2005 man
drwxr-xr-x   4 root    root     4096 Apr 24 13:50 samba
drwxr-xr-x   5 root    root     4096 Jul 17  2005 setup-tool-backends
drwxr-xr-x   2 squid   squid    4096 Jan 14 20:32 squid
localserver ~ # ls -l /var/log/http-replicator.log
-rw-r--r--  1 root root 9087 Apr 23 18:27 /var/log/http-replicator.log
localserver ~ # ls -l /var/run/http-replicator.pid
-rw-r--r--  1 root root 5 Apr 23 18:27 /var/run/http-replicator.pid
localserver ~ #


flybynite wrote:

Are you running any sort of security options on this server that would limit user abilities or access like pax, selinux, gresecurity, lids etc.?


I didn't install any of these.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Tue Apr 25, 2006 1:00 am    Post subject: Reply with quote

Haakon wrote:

Ok, I am really starting to think there may be something wrong with the python install.



Me too. But I'm also curious so maybe one or two more things.


First make sure replicator is stopped by running:
Code:

/etc/init.d/http-replicator stop
killall http-replicator


Then in a terminal as root and run replicator from there to try and figure out exactly which option gives problems. Here is how replicator is started by /etc/init.d/http-replicator.

Code:

/usr/bin/http-replicator -s -f --pid /var/run/http-replicator.pid --daemon --dir /var/cache/http-replicator --user portage --alias /usr/portage/packages/All:All --log /var/log/http-replicator.log --debug --ip 192.168.x.* --port 8080



Everything after --daemon is from your $DAEMON_OPTS from an earlier post.
Remove your censored x in the ip and start replicator this way and ensure it doesn't work for your clients. Check with either with emerge -f or the wget method from earlier, your choice.

You can confirm the user replicator is running by this, prior to testing with the clients since it might die after testing.
Code:

ps aux | grep replicator



Starting like this replicator will background itself and you have to kill it by
Code:

killall http-replicator


Then only change --user portage to --user root and try again. This should work for the clients? Then kill replicator with the killall command again.




Next try to remove the --deamon and --user opts like this.
Code:

/usr/bin/http-replicator -s -f --pid /var/run/http-replicator.pid  --dir /var/cache/http-replicator  --alias /usr/portage/packages/All:All --log /var/log/http-replicator.log --debug --ip 192.168.x.* --port 8080


Run without the --daemon option it won't background and will run as root. This should work for the clients also? Control-c will kill it this time.
Back to top
View user's profile Send private message
Haakon
n00b
n00b


Joined: 16 Jan 2006
Posts: 22

PostPosted: Tue Apr 25, 2006 2:48 pm    Post subject: Reply with quote

Ok, ran these commands on the server:
Code:

localserver ~ # /etc/init.d/http-replicator stop
 * ERROR:  "http-replicator" has not yet been started.
localserver ~ # killall http-replicator
http-replicator: no process killed
localserver ~ # /usr/bin/http-replicator -s -f --pid /var/run/http-replicator.pid --daemon --dir /var/cache/http-replicator --user portage --alias /usr/portage/packages/All:All --log /var/log/http-replicator.log --debug --ip 192.168.x.* --port 8080


Ran this on the client:
Code:

localclient ~ # http_proxy='192.168.x.1:8080' wget http://gentoo.chem.wisc.edu/gentoo/distfiles/tzdata2006a.tar.gz


And got the same error as before.

Next, I ran these commands on the server:
Code:

localserver ~ # killall http-replicator
http-replicator: no process killed
localserver ~ # /usr/bin/http-replicator -s -f --pid /var/run/http-replicator.pid --daemon --dir /var/cache/http-replicator --user root --alias /usr/portage/packages/All:All --log /var/log/http-replicator.log --debug --ip 192.168.x.* --port 8080


When I ran the same client command above on the client, http-replicator worked and I downloaded the file to the client.

Lastly, I ran this command on the server:
Code:

localserver ~ # killall http-replicator
localserver ~ # /usr/bin/http-replicator -s -f --pid /var/run/http-replicator.pid  --dir /var/cache/http-replicator  --alias /usr/portage/packages/All:All --log /var/log/http-replicator.log --debug --ip 192.168.x.* --port 8080
INFO: HttpReplicator started



Http-replicator again worked correctly when running the wget command on the client.


Oh, the last command will work for the clients. :P
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Thu Apr 27, 2006 8:03 pm    Post subject: Reply with quote

That narrows it down to a small section of code, but the trouble is not in that section of code. If it was it would fail with any user, not just a particular user.


The trouble is "which user", not the code. That particular system seems to have trouble with users other than root performing socket operations.


The only unknowns left are the networking setup and the kernel. You said that box is a server. Does it have two networks and two network cards? Your running iptables. That creates the possiblity that some type of packet mangling could cause a problem. But I'm pretty sure you tried the test after flushing all the rules? I believe /etc/init.d/iptables stop does that nicely. That would eliminate that possibility.

The other half of the equation is the kernel. I'm not familiar with genkernel but I do know it is software so it will have the occasional bug.
The last resort might be to upgrade the kernel. Make sure to use the same gcc version that compiled python.


You said replicator worked on your laptop. Might try that kernel version if it is different? If it isn't different, try another version anyway.
Back to top
View user's profile Send private message
Haakon
n00b
n00b


Joined: 16 Jan 2006
Posts: 22

PostPosted: Thu Apr 27, 2006 10:20 pm    Post subject: Reply with quote

Ok, I am recompiling the kernel. I didn't do that after the gcc updating that was done. I will work with the network settings and see what happens. Yes, I already tried turning off the iptables service and still had the same problem. I will try turning off the outside interface and changing the default route to the other gateway.

While I am working on this, I was curious about something. Would the package dnsmasq cause a problem? The same server that I want to run http-replicator on also runs dnsmasq, but it looks at another dns server to update its own cache.

Oh, the kernels between the server and laptop are the same version. :) It is the latest which portage supplied me.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Sat Apr 29, 2006 2:56 am    Post subject: Reply with quote

Haakon wrote:
Would the package dnsmasq cause a problem?


I've never used it but I did take a quick look at the manual. I don't think it will give problems. replicator filters based on network addresses, not names. The dhcp portion could cause trouble if your config gives out network addresses not in the allowed network you have setup in replicators config.

You have --ip 192.168.x.* in replicators config, if dnsmasq gives out a 192.168.Y.* address replicator will block the request but should give a security warning, like we got when I didn't have you add $DAEMON_OPTS in the command line run of the server.
Back to top
View user's profile Send private message
jacooper
n00b
n00b


Joined: 20 Apr 2004
Posts: 6

PostPosted: Sat Apr 29, 2006 11:33 pm    Post subject: Reply with quote

I have a suggestion to try:
Code:
chmod 777 /tmp

I was having the exact same issues, http-replicator would work as root but not as anyone else. This was a freshly installed system without iptables installed.
It turns out that the reason that I has having issues is that /tmp had permissions of 0755. I changed that to 0777 and everything worked for me. Since only root had write permission to /tmp, only root could create sockets.
Jeff
_________________
Jeff Cooper
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Sun Apr 30, 2006 3:44 am    Post subject: Reply with quote

jacooper wrote:
Since only root had write permission to /tmp, only root could create sockets.
Jeff



Dude, awsome! Thats why I like some troubleshooting. Gets you to know your stuff!! Its always the simple things...


I don't know if thats it until haakon checks in, but it fits the problem like a glove right down to the exact error!!!


Do you know how those permissions got there? Was it that way in the initial install??
Back to top
View user's profile Send private message
Haakon
n00b
n00b


Joined: 16 Jan 2006
Posts: 22

PostPosted: Sun Apr 30, 2006 4:36 pm    Post subject: Reply with quote

OMG, that was exactly it. It is suposed to have permission 777 with the sticky bit for "others":
Code:

drwxrwxrwt   5 root root  4096 Apr 29 04:20 tmp

This is usually something I change after my tarball install and I obviously forgot about it. The tarball install leaves the /tmp directory as 755 as jacooper suggested. Sorry, you two, for the run on this, espcially you, Flybynite. :( Ironically, I have already made myself a reminder document on what to change after the tarball install but it was created after this server's installation.

Also, I have caught this problem before, too,(which is how it got into my document :) ) but it was on systems that have X installed and X will not load if the users can not create or open a file in the /tmp directory. Naturally, X was not installed on the server we've been working on to fix.

edit: Almost forgot to answer your questions, Flybynite. Since it is a tarball installation, there isn't really an initial install, but the original computer's /tmp dir is correct. the /tmp, like the /sys and /proc dir, is a dir that I do not put into the tarball because of the files are not needed or shouldn't be added. The /tmp dir I create manually before the chroot and that is how it got messed up. After the chroot, I would normally change the permissions for the /tmp dir, but for whatever reason, I forgot to do so.

Thank you, for the tip, jacooper.

Flybynite, thank you for your patience in all of this and I apologize that this was more my fault than anything else.
Back to top
View user's profile Send private message
jacooper
n00b
n00b


Joined: 20 Apr 2004
Posts: 6

PostPosted: Sun Apr 30, 2006 8:26 pm    Post subject: Reply with quote

You're welcome. flybynite, did most of the work in diagnosing the issue. I just happened to have a flash iof intuition after reading all of that...

I'm not sure how it got that way. I'm setting up another system right now and I've just finished the state3 tarball extraction and the portage tarball extraction and tmp is still drwxrwxrwt. I'll keep a watch on it and see if I can catch anything changing it.
_________________
Jeff Cooper
Back to top
View user's profile Send private message
Haakon
n00b
n00b


Joined: 16 Jan 2006
Posts: 22

PostPosted: Mon May 01, 2006 6:11 am    Post subject: Reply with quote

No, I didn't mean any of the stage tarballs. Since I have around 20 computers at home, I made one build as i586 and tarballed that system while it was active. Since it was active, certain directories shouldn't be placed in the tar file. The /tmp, /sys, and /proc are the main directories I omit and manually create them.

Also, using the system wide tar file is how I do a backup on my computers. This server probably got its /tmp directory messed up when I restored the build after installing a new hard drive.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3 ... 17, 18, 19 ... 22, 23, 24  Next
Page 18 of 24

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum