Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
HOWTO:Download Cache for your LAN-Http-Replicator (ver 3.0)
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3 ... 16, 17, 18 ... 22, 23, 24  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
fcw
n00b
n00b


Joined: 07 Jan 2005
Posts: 9

PostPosted: Fri Dec 30, 2005 2:26 pm    Post subject: Re: "Deleting invalid distfile" Reply with quote

lannocc wrote:

Secondly (multi-part), is it http-replicator that is returning the 404 redirect? I know what a 404 is and what a redirect is, but what is a 404 redirect? Is there a different http error status that http-replicator could return in this instance that might make more sense?


I got this same error message from emerge after I made a local web mirror available; it happened whenever the file requested was not available locally.

It turns out that the web server was configured to present a normal-looking friendly page with links to various places in place of a bare 404 error message. This is fine for users, but apparently not for whatever portage uses (wget?), which can't make sense of the response.

By altering just the HTTP status code on the friendly page to 404, while leaving the body of the page the same for end-users, emerge stopped presenting the '404 redirect' message.
Back to top
View user's profile Send private message
carpman
Advocate
Advocate


Joined: 20 Jun 2002
Posts: 2202
Location: London - UK

PostPosted: Thu Jan 05, 2006 11:38 am    Post subject: Reply with quote

Hello, ok have asked this before but got no reply so wil try again as stil having problem.

The issue is that some files will not download when using httpreplicator, if i comment it out in make.conf then files download fine?

I have created /etc/portage/mirrors


Code:

emerge (2 of 97) media-libs/netpbm-10.30-r1 to /
>>> Downloading http://gentoo.blueyonder.co.uk/distfiles/netpbm-10.30-manpages.tar.bz2
--11:34:44--  http://gentoo.blueyonder.co.uk/distfiles/netpbm-10.30-manpages.tar.bz2
           => `/usr/portage/distfiles/netpbm-10.30-manpages.tar.bz2'
Connecting to 192.168.1.3:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 233,254 (228K) [application/x-tar]

100%[===================================================================>] 233,254       --.--K/s

11:34:44 (2.62 MB/s) - `/usr/portage/distfiles/netpbm-10.30-manpages.tar.bz2' saved [233254/233254]

>>> Downloading http://gentoo.blueyonder.co.uk/distfiles/netpbm-10.30.tgz
--11:34:44--  http://gentoo.blueyonder.co.uk/distfiles/netpbm-10.30.tgz
           => `/usr/portage/distfiles/netpbm-10.30.tgz'
Connecting to 192.168.1.3:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: unspecified

    [ <=>                                                                ] 0             --.--K/s

11:34:44 (0.00 B/s) - `/usr/portage/distfiles/netpbm-10.30.tgz' saved [0]

>>> Resuming download...
>>> Downloading http://gentoo.blueyonder.co.uk/distfiles/netpbm-10.30.tgz
--11:34:44--  http://gentoo.blueyonder.co.uk/distfiles/netpbm-10.30.tgz
           => `/usr/portage/distfiles/netpbm-10.30.tgz'
Connecting to 192.168.1.3:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: unspecified

    [ <=>                                                                ] 0             --.--K/s

11:34:44 (0.00 B/s) - `/usr/portage/distfiles/netpbm-10.30.tgz' saved [0]

>>> Resuming download...
>>> Downloading http://mirror.ovh.net/gentoo-distfiles/distfiles/netpbm-10.30.tgz
--11:34:44--  http://mirror.ovh.net/gentoo-distfiles/distfiles/netpbm-10.30.tgz
           => `/usr/portage/distfiles/netpbm-10.30.tgz'
Connecting to 192.168.1.3:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: unspecified

    [ <=>                                                                ] 0             --.--K/s

_________________
Work Station - 64bit
Gigabyte GA X48-DQ6 Core2duo E8400
8GB GSkill DDR2-1066
SATA Areca 1210 Raid
BFG OC2 8800 GTS 640mb
--------------------------------
Notebook
Samsung Q45 7100 4gb
Back to top
View user's profile Send private message
troubleticket
n00b
n00b


Joined: 03 Jan 2006
Posts: 6

PostPosted: Fri Jan 06, 2006 2:44 pm    Post subject: Reply with quote

Hello everyone,

having had a short thread about the following item in "forums.gentoo.org"
https://forums.gentoo.org/viewtopic-p-3001502.html#3001502 and also this posting on http://gentoo-wiki.com/index.php?title=Talk:HOWTO_Download_Cache_for_LAN-Http-Replicator I again try it here:

To me it seems, that all software not installed on the installation server (http-replicator) itself, will not be updatetd to the cache until the respective client does the emerge.

My aim is to anyhow update only the cache e.g. once a night after an

Code:
emerge sync


on the server (feeding the "Local Rsync Mirror") and afterwards have an consistent state for all the different clients, so that all subsequent updates from any client (except intallation of a completely new packcage never insatalled before on any other machine) can run without internet connect.

In other words: I don't think that the world file on the installation server/cache is being affected by the clients' software updates so an

Code:
emerge -uD world


only on the cache-machine would not do what I want.

Or am I missing something ... ???
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Fri Jan 06, 2006 5:13 pm    Post subject: Reply with quote

carpman wrote:
Hello, ok have asked this before but got no reply so wil try again as stil having problem.


Asked and answered. I guess you missed it....


Here is your original post and the second reply down is the answer quoting your problem...

https://forums.gentoo.org/viewtopic-t-173226-postdays-0-postorder-asc-start-390.html
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Fri Jan 06, 2006 6:01 pm    Post subject: Reply with quote

troubleticket wrote:

To me it seems, that all software not installed on the installation server (http-replicator) itself, will not be updatetd to the cache until the respective client does the emerge.

Or am I missing something ... ???



Sorry I don't follow the wiki. This thread is the only support I give.

Having read all your posts, If I understand correctly, you want your nightly connect to the internet to have everything downloaded for all your different computers ready for them to perform the updates without needing to connect to the net again???


If so, what you want can be done easily, just not how you think it can.

Each individual computer can only tell what it needs to update. No other box can know that. That is the way portage/gentoo works.

But here is how to do what you want.

Code:

emerge sync
emerge -f world


Run this from a cron on your local rsync server first, then 5 minutes later, run it on all other computers you have.

This will have all your computers ready to be updated without connecting to the net again. The local rsync server and http-replicator will ensure that only 1 copy of ALL data (rsync and packages) will come from the internet, the rest will come from the cache.
Back to top
View user's profile Send private message
carpman
Advocate
Advocate


Joined: 20 Jun 2002
Posts: 2202
Location: London - UK

PostPosted: Fri Jan 06, 2006 8:36 pm    Post subject: Reply with quote

flybynite wrote:
carpman wrote:
Hello, ok have asked this before but got no reply so wil try again as stil having problem.


Asked and answered. I guess you missed it....


Here is your original post and the second reply down is the answer quoting your problem...

https://forums.gentoo.org/viewtopic-t-173226-postdays-0-postorder-asc-start-390.html


Apologies that i missed it, thing is i seem to get this sort of thing n regular basis!

I did as sugested and moved all files in http-rep dir to /usr/portage/distfiles and and ran repcacheman which resulted in:
Code:

SUMMARY:
Found 0 duplicate file(s).
        Deleted 0 dupe(s).
Found 737 new file(s).
        Added 682 of those file(s) to the cache.
        Rejected 31 corrupt or incomplete file(s).
        24 Unknown file(s) that are not listed in portage
        You may want to delete them yourself....


Now thing is how do i know which ones are corrupt and which ones are not?

Will watch things and see what occurs.
_________________
Work Station - 64bit
Gigabyte GA X48-DQ6 Core2duo E8400
8GB GSkill DDR2-1066
SATA Areca 1210 Raid
BFG OC2 8800 GTS 640mb
--------------------------------
Notebook
Samsung Q45 7100 4gb
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Sun Jan 08, 2006 4:56 am    Post subject: Reply with quote

carpman wrote:

Now thing is how do i know which ones are corrupt and which ones are not?


unless you use replicator to cache other downloads, delete everything left in /usr/portage/distfiles/ after running repcacheman. I don't have repcachman do this by default because some users will have replicator cache files outside of portage, plus it takes root to delete those files and repcacheman doesn't run as root by default.

If you look closer at the output of repcacheman, it told you which files are ok, corrupt, or not in portage!!
Code:

/usr/portage/distfiles/digikam-0.8.0.tar.bz2
MD5 OK

WARNING xine-lib-patches-21.tar.bz2 is not in portage!!!

/usr/portage/distfiles/libkexif-0.2.2.tar.bz2
MD5 OK
Back to top
View user's profile Send private message
troubleticket
n00b
n00b


Joined: 03 Jan 2006
Posts: 6

PostPosted: Tue Jan 24, 2006 9:23 am    Post subject: Reply with quote

flybynite wrote:


Having read all your posts, If I understand correctly, you want your nightly connect to the internet to have everything downloaded for all your different computers ready for them to perform the updates without needing to connect to the net again???

If so, what you want can be done easily, just not how you think it can.

Each individual computer can only tell what it needs to update. No other box can know that. That is the way portage/gentoo works.

But here is how to do what you want.

Code:

emerge sync
emerge -f world


Run this from a cron on your local rsync server first, then 5 minutes later, run it on all other computers you have.

This will have all your computers ready to be updated without connecting to the net again. The local rsync server and http-replicator will ensure that only 1 copy of ALL data (rsync and packages) will come from the internet, the rest will come from the cache.


First:
Thank you very much for your time, your effort, your work ... your answer.

Second:
Your suggestion assumes me to be "boss" of all of the boxes on my lan, or if not, all boxes to be up and running all night long. Neither one meets my reality.

I had hoped there could be a solution like:
"Look along all my cache's content, extract the packages' "basenames" and feed some simple script with the result"
on the server ...

But it seems ... <sigh>
Back to top
View user's profile Send private message
F.Ultra
Apprentice
Apprentice


Joined: 17 Mar 2004
Posts: 169
Location: Sweden

PostPosted: Tue Feb 14, 2006 11:26 am    Post subject: Reply with quote

First of all, many, many thanks to Flybynite for this excellent tool!

As I understand it http-replicator do not handle FTP-requests, and I wonder how much work it would be to add this functionality? If I am not to misinformed, a ftp_proxy means that wget would send a http request to the proxy with a ftp:// type of request embedded in the http request so the communication between the client and server (http-replicator) would this be the same, the "only" thing would be that http-replicator now had to fetch the file using ftp instead of http. Using wget in http-replicator for this task would perhaps work quite easily with the only side-effect that the file would have to be downloaded in full before the data could be streamed to the clients. Or maybe I am dead wrong here :D

Another thing is the mirrors, it would be very nice if one could simply insert a single mirror in make.conf on the clients with say GENTOO_MIRRORS="http://127.0.0.1", and let http-replicator choose a mirror from its own local make.conf when receiving such a request. In this way one would have a single point of mirror configuration (at the http-replicator server) which would be far easier than maintaining the mirror list on some 1000 clients 8O

This was just my two thoughts on an otherwise brilliant piece of software!
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Tue Feb 14, 2006 10:21 pm    Post subject: Reply with quote

F.Ultra wrote:

As I understand it http-replicator do not handle FTP-requests, and I wonder how much work it would be to add this functionality?

Another thing is the mirrors, it would be very nice if one could simply insert a single mirror in make.conf on the clients with say GENTOO_MIRRORS="http://127.0.0.1", and let http-replicator choose a mirror from its own local make.conf




Correct on all counts.

ftp has been in the works for a while now. I've got sample code to do ftp requests.

What you may not know is that http-replicator started out as a general http and debian specific cache. I worked with the primary developer to get gentoo specific features and http-replicator has taken off and become a supported package under gentoo. I don't know if the project was ever accepted by debian.

Gertjan is the primary developer, http://gertjan.freezope.org/replicator/ as stated in the ebuild. He is a debian user, not a gentoo user.

I worked with Gertjan to start these changes quite a while ago. I gave him working ftp code and he had some good ideas on choosing mirrors. He wouldn't add these features as a simple upgrade but only as a part of a complete rewrite. I haven't heard anything lately so I assume the project bogged down in the complete rewrite. His changelog shows version 3.1 in work as of may 2005. http://gertjan.freezope.org/replicator/changelog/

You may want to drop him a line and see if you can rekindle his interest in http-replicator. It's probably hard for him to add gentoo specific features when he is a debian user....
Back to top
View user's profile Send private message
Inhale
n00b
n00b


Joined: 30 Nov 2003
Posts: 19

PostPosted: Tue Feb 14, 2006 10:51 pm    Post subject: http-replicator 3 - cache miss in error Reply with quote

I'm using http-replicator 3.

When emerging, the client machine's first wget to a local mirror always returns a file not found 404 - the subsequent wget to a second mirror is successful. This is regardless of file being requested.

I confirm that the first mirror and file exists - using wget and the URL pasted from the output on the http-replicator proxy. The same occurs for any cache miss.


# emerge gentoo-vdr-scripts
Calculating dependencies ...done!
>>> emerge (1 of 2) app-admin/sudo-1.6.8_p9-r2 to /
>>> Resuming download...
>>> Downloading ftp://ftp.citylink.co.nz/gentoo/distfiles/sudo-1.6.8p9.tar.gz
--11:48:56-- ftp://ftp.citylink.co.nz/gentoo/distfiles/sudo-1.6.8p9.tar.gz
=> `/home/portage/distfiles/sudo-1.6.8p9.tar.gz'
Resolving luke... 172.17.28.33
Connecting to luke|172.17.28.33|:8080... connected.
Proxy request sent, awaiting response... 404 Not Found
11:48:56 ERROR 404: Not Found.

>>> Resuming download...
>>> Downloading http://gentoo.osuosl.org/distfiles/sudo-1.6.8p9.tar.gz
--11:48:56-- http://gentoo.osuosl.org/distfiles/sudo-1.6.8p9.tar.gz
=> `/home/portage/distfiles/sudo-1.6.8p9.tar.gz'
Resolving luke... 172.17.28.33
Connecting to luke|172.17.28.33|:8080... connected.
Proxy request sent, awaiting response... 206 Partial Content
Length: 585,509 (572K), 568,853 (556K) remaining [application/x-gzip]
-
2% [+ ] 16,656 --.--K/s


>>> Update: Doh! My first mirror is an FTP site which current http-replicator doesn't cache - changed URI to HTTP protocol and it works :oops:
Back to top
View user's profile Send private message
F.Ultra
Apprentice
Apprentice


Joined: 17 Mar 2004
Posts: 169
Location: Sweden

PostPosted: Wed Feb 15, 2006 9:07 pm    Post subject: Reply with quote

flybynite: Many thanks for your info. I will have a look into this but as you say getting a Debian developer do specific Gentoo stuff is probably a dead end. It is a realy pity that I do not know python/bash enough to to these changes myself but perhaps I will get my thumbs out and do a Gentoo specific fork in C. Probably not since I will probably focus more into doing a fully centralized managment system instead of having each machine be managed separately.
Back to top
View user's profile Send private message
Inhale
n00b
n00b


Joined: 30 Nov 2003
Posts: 19

PostPosted: Wed Feb 15, 2006 9:30 pm    Post subject: non-issue Reply with quote

Non-issue - if you see the last line (the one starting with D'oh) 'the technical department have located the source of the problem (points to self) and are endevouring correct the issue' :-)

- I remember reading somewhere that the next version (a major re-write) will also cache FTP URIs so this would have gone away on upgrade anyway..
Back to top
View user's profile Send private message
Haakon
n00b
n00b


Joined: 16 Jan 2006
Posts: 22

PostPosted: Sat Apr 15, 2006 1:30 pm    Post subject: Reply with quote

First, let me not be thankless and thank you for your efforts in creating and maintaining http-replicator as well as putting up with people like me and their problems. :D

I did a quick scan of the recent posts, but I haven't seen anything like I am experiencing with http-replicator. I admit, I did a fast scan so I could have missed it.

Here is a --debug log which shows the error I am encountering:

Code:

15 Apr 2006 04:00:21 INFO: HttpReplicator started
15 Apr 2006 04:00:51 STAT: HttpClient 1 bound to 192.168.x.4
15 Apr 2006 04:00:51 ERROR: HttpClient 1 caught an exception, closing socket
Traceback (most recent call last):
  File "/usr/lib/python2.4/asyncore.py", line 69, in read
    obj.handle_read_event()
  File "/usr/lib/python2.4/asyncore.py", line 391, in handle_read_event
    self.handle_read()
  File "/usr/bin/http-replicator", line 156, in handle_read
    self.data.write(chunk) # append received data
  File "/usr/lib/python2.4/asyncore.py", line 366, in __getattr__
    return getattr(self.socket, attr)
AttributeError: '_socketobject' object has no attribute 'data'


And this is what the client sees during the fetch part of the emerge:

Code:

>>> emerge (2 of 9) sys-libs/timezone-data-2006a to /
>>> Downloading http://gentoo.chem.wisc.edu/gentoo/distfiles/tzdata2006a.tar.gz
--05:21:40--  http://gentoo.chem.wisc.edu/gentoo/distfiles/tzdata2006a.tar.gz
           => `/usr/portage/distfiles/tzdata2006a.tar.gz'
Connecting to 192.168.x.1:8080... connected.
Proxy request sent, awaiting response... No data received.
Retrying.

--05:21:41--  http://gentoo.chem.wisc.edu/gentoo/distfiles/tzdata2006a.tar.gz
  (try: 2) => `/usr/portage/distfiles/tzdata2006a.tar.gz'
Connecting to 192.168.x.1:8080... failed: Connection refused.
>>> Downloading http://gentoo.osuosl.org/distfiles/tzdata2006a.tar.gz
--05:21:41--  http://gentoo.osuosl.org/distfiles/tzdata2006a.tar.gz
           => `/usr/portage/distfiles/tzdata2006a.tar.gz'
Connecting to 192.168.x.1:8080... failed: Connection refused.
>>> Downloading http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/tzdata2006a.tar.gz
--05:21:41--  http://www.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/tzdata2006a.tar.gz
           => `/usr/portage/distfiles/tzdata2006a.tar.gz'
Connecting to 192.168.x.1:8080... failed: Connection refused.
>>> Downloading ftp://elsie.nci.nih.gov/pub/tzdata2006a.tar.gz
--05:21:41--  ftp://elsie.nci.nih.gov/pub/tzdata2006a.tar.gz
           => `/usr/portage/distfiles/tzdata2006a.tar.gz'
Resolving elsie.nci.nih.gov... 137.187.215.78
Connecting to elsie.nci.nih.gov|137.187.215.78|:21... connected.
Logging in as anonymous ...

Exiting on signal 2


At the end here on "signal 2" is where I broke the emerge process due to the error and trying to troubleshoot. The last mirror you see there, the ftp site, is the only one that seems to download anything when I use http-replicator. I have tried different port numbers above 10000 and I have also tried dropping my firewall to ensure that was not causing a problem.

Any help you can provide would be greatly appreciated. I have over 20 computers at home, all of which have Gentoo. Currently, I tend to copy the dist diretory from one machine to the next to keep from having to download all the source files each time I update a machine. This little utility you have written seems to be a godsend. :)
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Mon Apr 17, 2006 4:17 am    Post subject: Reply with quote

Haakon wrote:


Code:

15 Apr 2006 04:00:21 INFO: HttpReplicator started
15 Apr 2006 04:00:51 STAT: HttpClient 1 bound to 192.168.x.4
15 Apr 2006 04:00:51 ERROR: HttpClient 1 caught an exception, closing socket

)



Did you censor the log, or do you really have 192.168.X.4 in the config file?
Back to top
View user's profile Send private message
Haakon
n00b
n00b


Joined: 16 Jan 2006
Posts: 22

PostPosted: Tue Apr 18, 2006 1:11 am    Post subject: Reply with quote

flybynite wrote:

Did you censor the log, or do you really have 192.168.X.4 in the config file?


Sorry about that, I should have said as much in the first place. I censored the third octet because I didn't think it was needed for the problem at hand.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Tue Apr 18, 2006 3:18 am    Post subject: Reply with quote

Ok, just gotta check the obvious first. I don't mean anything by it. I also have to ask if your running ~x86?

It seems python is having trouble opening a socket. Try re-emerging python.

Re-emerge python anyway, but have you udated the kernel/kernel headers, glibc or gcc lately? Of course your running gentoo, so the answer is probably yes to more than one :-) A list of versions for the above and python might be helpful also. Last time I saw something similiar, the re-emerge fixed it, but I expect some api change to hit anytime now....

Code:

emerge -va1 python
Back to top
View user's profile Send private message
Haakon
n00b
n00b


Joined: 16 Jan 2006
Posts: 22

PostPosted: Wed Apr 19, 2006 12:36 am    Post subject: Reply with quote

flybynite wrote:
Ok, just gotta check the obvious first. I don't mean anything by it. I also have to ask if your running ~x86?

It seems python is having trouble opening a socket. Try re-emerging python.

Re-emerge python anyway, but have you udated the kernel/kernel headers, glibc or gcc lately? Of course your running gentoo, so the answer is probably yes to more than one :-) A list of versions for the above and python might be helpful also. Last time I saw something similiar, the re-emerge fixed it, but I expect some api change to hit anytime now....

Code:

emerge -va1 python


Yup, I am running a ~x86 and using Gentoo. I didn't read into anything you said. :) On the machine in question it is a Sempron64 2600+, Asus K8N motherboard, and 512MB of RAM. I don't really think the video card has anything to do with this, but it is a Voodoo3. The hard drive is an 80GB Maxtor on PATA.

glibc = version 2.3.5-r3
gcc = version 3.4.5-r1
gentoo-sources = 2.6.15-r1
linux-headers = 2.6.11-r2
Using profile 2006.0

I did a "emerge --sync" about two weeks ago. After the sync, I did a "emerge --update --deep --newuse world"

I am running the 2.6.15-r1 kernel. It was created using "genkernel --menuconfig all". I ensured the proper initial RAM disk was married up with that kernel in my boot loader. Here is that section from lilo.conf:
Code:

image=/boot/kernel-genkernel-x86-2.6.15-gentoo-r1
        label=default
        read-only
        root=/dev/ram0
        append="video=vesafb:1024x768-16@75 init=/linuxrc ramdisk=8192 real_root=/dev/hda1 udev"
        initrd=/boot/initramfs-genkernel-x86-2.6.15-gentoo-r1


I did see a suggestion about remerging the python package, and I did that, but I did it again just for S&G's and because you asked. :D

Here is the first line before it asked to continue with the python emerge"
Code:

[ebuild   R   ] dev-lang/python-2.4.2  -X +berkdb -bootstrap -build -doc +gdbm -ipv6 +ncurses -nocxx +readline +ssl -tcltk -ucs2 0 kB


And here is the last few lines after the emerge finished. You will also see the version there:
Code:

 * Byte compiling python modules for python-2.4 .. ...                                                                                                 [ ok ]

 *
 * If you have just upgraded from an older version of python you will need to run:
 *
 * /usr/sbin/python-updater
 *
 * This will automatically rebuild all the python dependent modules
 * to run with python-2.4.
 *
 * Your original Python is still installed and can be accessed via
 * /usr/bin/python2.x.
 *
>>> Regenerating /etc/ld.so.cache...
>>> dev-lang/python-2.4.2 merged.

>>> clean: No packages selected for removal.

>>> Auto-cleaning packages ...

>>> No outdated packages were found on your system.


 * GNU info directory index is up-to-date.

localhost ~ # python-updater
 * Can't determine any previous Python version(s).
localhost ~ #

I did give it a name other than "localhost". Just another censor.

Oh, and I tried the python-updater.... :)

If you want, I can sync and update all the packages, however, I am making the assumption we don't want to potentially make more problems than we have already. :)

Thanks again for all your help and efforts.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Wed Apr 19, 2006 7:40 am    Post subject: Reply with quote

Haakon wrote:


glibc = version 2.3.5-r3
gcc = version 3.4.5-r1
gentoo-sources = 2.6.15-r1
linux-headers = 2.6.11-r2
Using profile 2006.0

I am running the 2.6.15-r1 kernel.
dev-lang/python-2.4.2

If you want, I can sync and update all the packages, however, I am making the assumption we don't want to potentially make more problems than we have already. :)


Ok, your running stable gentoo (not ~x86) and just so happens I have the exact same versions of everything, right down to the kernel version!! (What are the odds?) So you haven't found any new api changes.

You don't need to rebuild your whole sytem, yet :-) Although it might be a good idea if you have the time since you may have recently upgraded to gcc 3.4.5. Is this a fresh install with gcc 3.4.5 or did you do the upgrade yourself? If so, did you do the 'emerge -e' complete rebuild upgrade or the revdep-rebuild upgrade from this howto?

http://www.gentoo.org/doc/en/gcc-upgrading.xml#upgrade-3.3-to-3.4

I tried the revdep-rebuild upgrade and found a few packages later that revdep-rebuild upgrade missed so I did the complete rebuild later.


The next step is to check your config. Try the tests in the following post:
https://forums.gentoo.org/viewtopic-t-173226-postdays-0-postorder-asc-start-384.html
Back to top
View user's profile Send private message
Haakon
n00b
n00b


Joined: 16 Jan 2006
Posts: 22

PostPosted: Wed Apr 19, 2006 9:24 pm    Post subject: Reply with quote

flybynite wrote:


Ok, your running stable gentoo (not ~x86) and just so happens I have the exact same versions of everything, right down to the kernel version!! (What are the odds?) So you haven't found any new api changes.

You don't need to rebuild your whole sytem, yet :-) Although it might be a good idea if you have the time since you may have recently upgraded to gcc 3.4.5. Is this a fresh install with gcc 3.4.5 or did you do the upgrade yourself? If so, did you do the 'emerge -e' complete rebuild upgrade or the revdep-rebuild upgrade from this howto?

http://www.gentoo.org/doc/en/gcc-upgrading.xml#upgrade-3.3-to-3.4

I tried the revdep-rebuild upgrade and found a few packages later that revdep-rebuild upgrade missed so I did the complete rebuild later.


The next step is to check your config. Try the tests in the following post:
https://forums.gentoo.org/viewtopic-t-173226-postdays-0-postorder-asc-start-384.html


Well, found one problem... it would seem my gcc profile is stuck on i386-pc-linux-gnu-3.3.6. I changed that to i386-pc-linux-gnu-3.4.5 and I am following the steps of that howto. I will be running an "emerge -eD world" next. I will try http-replicator after all that and see if that works.

I had no idea I was supposed to do all that. I just let emerge do its thing and come back hours later when it is done. I apologize for my noobishness. :oops: I also stick to the i386 side so that I can tarball the entire system up and clone it off to another system as needed. I can always change the make.conf later to customize to that particular machine. Yes, I do have systems that old... :) Hey they work, so why not use them for simple tasks like a home file server? :D

Oh, revdep-rebuild didn't like one missing link, but since I don't have X on my system, I didn't see the problem and neither did revdep. :)
Code:

localhost X11 # revdep-rebuild
Configuring search environment for revdep-rebuild

Checking reverse dependencies...

Packages containing binaries and libraries broken by a package update
will be emerged.

Collecting system binaries and libraries... done.
  (/root/.revdep-rebuild.1_files)

Collecting complete LD_LIBRARY_PATH... done.
  (/root/.revdep-rebuild.2_ldpath)

Checking dynamic linking consistency...
  broken /usr/lib/X11/xkb/xkbcomp (requires  libX11.so.6 libxkbfile.so.1)
 done.
  (/root/.revdep-rebuild.3_rebuild)

Assigning files to ebuilds... done.
  (/root/.revdep-rebuild.4_ebuilds)

Evaluating package order... done.
  (/root/.revdep-rebuild.5_order)

Dynamic linking on your system is consistent... All done.
localhost X11 #


I am not sure how to permanently remove that link(/usr/lib/X11/xkb/xkbcomp), as I have removed the package, via "emerge --unmerge", and used equery to make sure it was all gone and removed the packages that persisted. The orignal tarball included X which I didn't need for this machine so I killed it and did the "emerge -eD world" to freshen the system after those changes and the changes to the make.conf to exclude X. However, I am suspecting this is a completely different issue and has nothing to do with why http-replicator isn't working. I am only providing some insight to the system and how it was created.


I will post back when the tasks have finished and after I tried http-replicator again.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Thu Apr 20, 2006 1:25 am    Post subject: Reply with quote

Haakon wrote:

Well, found one problem... it would seem my gcc profile is stuck on i386-pc-linux-gnu-3.3.6.



Well, due to a blooper by a dev the first ebuild of gcc 3.4.x may have caused some packages to build with the new 3.4.x compiler which could have made those packages inconsistent. The ebuild was quickly fixed. If you just blindly upgraded to 3.4.x nothing would have happened with the fixed ebuild. It depends when you upgraded to know if you were bitten by this blooper.

Might as well finish the upgrade to be sure.


Haakon wrote:

Oh, revdep-rebuild didn't like one missing link, but since I don't have X on my system, I didn't see the problem and neither did revdep. :)
Code:

  broken /usr/lib/X11/xkb/xkbcomp (requires  libX11.so.6 libxkbfile.so.1)
 


I am not sure how to permanently remove that link(/usr/lib/X11/xkb/xkbcomp)



Don't over think this. This orphan file can just be deleted which will keep revdep-rebuild from complaining.
Code:

rm /usr/lib/X11/xkb/xkbcomp
Back to top
View user's profile Send private message
Haakon
n00b
n00b


Joined: 16 Jan 2006
Posts: 22

PostPosted: Thu Apr 20, 2006 3:00 am    Post subject: Reply with quote

Well, the rebuild didn't work. :( I have the exact same error as before.

As for the configuration file for http-replicator, it is as the default with the exceptions of the --debug option enabled and the class A private IP range being disabled.

Is it possible the client is actually the one with the problem? I am pretty certain they all have the gcc profile problem.

Also, I don't have a problem rebuilding the server computer. I have a backup server to provide the services this server provides. Oh, I won't use my tarball install, I will do it the long way with stage 3. Want me to do the complete reinstall?
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Thu Apr 20, 2006 6:43 am    Post subject: Reply with quote

Haakon wrote:

As for the configuration file for http-replicator, it is as the default with the exceptions of the --debug option enabled and the class A private IP range being disabled.



First things first. Just one step at a time.

Did you run the tests in the post above? This helps spot typo's by actually printing what replicator would see, not what you see.
Back to top
View user's profile Send private message
Haakon
n00b
n00b


Joined: 16 Jan 2006
Posts: 22

PostPosted: Thu Apr 20, 2006 8:39 pm    Post subject: Reply with quote

Sorry about that. My mind is racing... Ok, so the first step from your second URL is this command:
Code:

source /etc/conf.d/http-replicator
/usr/bin/http-replicator


which provided this output:
Code:

localhost conf.d # source /etc/conf.d/http-replicator
localhost conf.d # /usr/bin/http-replicator
INFO: HttpReplicator started




I had pressed enter a few times here because I didn't get my command prompt back after executing the binary.

I pressed ctrl-c and got this and my command prompt back:
Code:

INFO: HttpReplicator terminated


The next two steps:
Code:

localhost conf.d # echo $GENERAL_OPTS
--dir /var/cache/http-replicator --user portage
localhost conf.d # echo $DAEMON_OPTS
--dir /var/cache/http-replicator --user portage --alias /usr/portage/packages/All:All --log /var/log/http-replicator.log --debug --ip 192.168.x.* --port 8080
localhost conf.d #


I noticed I didn't have the second --alias that you have, the "--alias /usr/src/:src" one.

Oh, and of course, the "--ip 10.*.*.*" is missing because I took it out and the "--debug" option is in because I put that in. :)

the localhost and x in the ip third octet are censored, however the "x" represents the same number for the client and server. I have an actual number where "x" sits, same goes for the config file below.

Finally, the /etc/conf.d/http-replicator file:
Code:

localhost conf.d # cat http-replicator
## Config file for http-replicator
## sourced by init scripts automatically
## GENERAL_OPTS used by repcacheman
## DAEMON_OPTS used by http-replicator


## Set the cache dir
GENERAL_OPTS="--dir /var/cache/http-replicator"

## Change UID/GID to user after opening the log and pid file.
## 'user' must have read/write access to cache dir:
GENERAL_OPTS="$GENERAL_OPTS --user portage"

## Don't change or comment this out:
DAEMON_OPTS="$GENERAL_OPTS"

## Do you need a proxy to reach the internet?
## This will forward requests to an external proxy server:
## Use one of the following, not both:
#DAEMON_OPTS="$DAEMON_OPTS --external somehost:1234"
#DAEMON_OPTS="$DAEMON_OPTS --external username:password@host:port"

## Local dir to serve clients.  Great for serving binary packages
## See PKDIR and PORTAGE_BINHOST settings in 'man make.conf'
## --alias /path/to/serve:location will make /path/to/serve
## browsable at http://http-replicator.com:port/location
DAEMON_OPTS="$DAEMON_OPTS --alias /usr/portage/packages/All:All"

## Dir to hold the log file:
DAEMON_OPTS="$DAEMON_OPTS --log /var/log/http-replicator.log"

## Make the log messages less and less verbose.
## Up to four times to make it extremely quiet.
#DAEMON_OPTS="$DAEMON_OPTS --quiet"
#DAEMON_OPTS="$DAEMON_OPTS --quiet"

## Make the log messages extra verbose for debugging.
DAEMON_OPTS="$DAEMON_OPTS --debug"

## The ip addresses from which access is allowed. Can be used as many times
## as necessary. Access from localhost is allowed by default.
DAEMON_OPTS="$DAEMON_OPTS --ip 192.168.x.*"
#DAEMON_OPTS="$DAEMON_OPTS --ip 10.*.*.*"

## The proxy port on which the server listens for http requests:
DAEMON_OPTS="$DAEMON_OPTS --port 8080"
localhost conf.d #


I am pretty sure that is everything. Let me know if I missed something.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Thu Apr 20, 2006 10:03 pm    Post subject: Reply with quote

Hmm.. Everything looks good so far from the server side.
Lets start up the server like this.

Code:

source /etc/conf.d/http-replicator
/usr/bin/http-replicator



and then try to see where the problem kicks in. Is the problem only on one file? Or only on one http server?

You can test like this from another terminal without using portage.

Code:

http_proxy='yourproxy:8080' wget http://gentoo.chem.wisc.edu/gentoo/distfiles/tzdata2006a.tar.gz


which is the same file and server of your original problem.


Try it with a few different files and different servers.

Code:

http_proxy='yourproxy:8080' wget http://gentoo.osuosl.org/distfiles/netcat-110-patches-1.0.tar.bz2


If it worked so far, the files are in the cache. Try the above again which will retrieve the files from the cache.

Does it fail from a single box or from another box as well. Try it from the server box also.

I'd like to see at least one failure from the server side and the client side. The other important things are can you narrow it down to a single client, file, or http server?
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3 ... 16, 17, 18 ... 22, 23, 24  Next
Page 17 of 24

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum