Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
HOWTO:Download Cache for your LAN-Http-Replicator (ver 3.0)
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3 ... 9, 10, 11 ... 22, 23, 24  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
RaraRasputin
Tux's lil' helper
Tux's lil' helper


Joined: 04 Jun 2004
Posts: 142
Location: Germany

PostPosted: Wed Nov 17, 2004 9:15 pm    Post subject: Reply with quote

Hi,
why is it suggested to add osuosl.org to the mirrors file? if add it, all mirrors in my make.conf are ignored and all files are downloaded from osuosl.org, which is really slow.
Back to top
View user's profile Send private message
johntramp
Guru
Guru


Joined: 03 Feb 2004
Posts: 457
Location: New Zealand

PostPosted: Thu Nov 18, 2004 12:08 am    Post subject: Reply with quote

I think any http:// mirror will work
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Thu Nov 18, 2004 10:26 pm    Post subject: Reply with quote

GuruSwami wrote:

When I start an emerge from a client machine, I get a 401 not found error (from memory - away from the box at the moment)


You must post the errors. Start with some of the emerge output...

You most likely have a typo in your config...
Back to top
View user's profile Send private message
VinnieNZ
Tux's lil' helper
Tux's lil' helper


Joined: 11 Mar 2004
Posts: 126
Location: New Zealand

PostPosted: Fri Nov 19, 2004 4:09 am    Post subject: Reply with quote

I'm attempting to setup a cascading distfiles solution for work.

Basically we have our main network with smaller networks in different regions connected into our main network. The link to the internet from the main network is fast, the internal link to the regional network is slowish. Because the link to the regional network is slow, but with many clients at the end, there are large advantages to having a system like this working.

I have setup http-replicator on a server in one of the regions, and one on a server locally.

What I want to happen is when a client computer from the regions requests a package it looks at the regional server first. If it doesn't have it, then the regional server should attempt to fetch the file from the main network server, and if the main network server doesn't have it, it should go out to the internet and fetch it. But I want to have a copy of that package left in the cache at every point.

In an example:
A regional client computer requests a new package and none of our servers have the file, so the file should be sourced from the internet then left on our package cache on the main network, as well as on the regional server, and finally in the distfiles dir of the regional client originally requesting the file.

I like the look of using a system like this because it means that we don't have to serve off of a ftp or http style system, and it doesn't interfere with our normal http proxy.

Is what I'm trying to do possible via http-replicator, or should I be looking at something else?

Cheers.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Fri Nov 19, 2004 8:08 am    Post subject: Reply with quote

VinnieNZ wrote:
I'm attempting to setup a cascading distfiles solution for work.


Done....

Http-Replicator has an option for an external proxy. Set the regional http-replicator to use the proxy of the main http-replicator proxy.

Code:

## Do you need a proxy to reach the internet?
## This will forward requests to an external proxy server:
## Use one of the following, not both:
#DAEMON_OPTS="$DAEMON_OPTS --external somehost:1234"
#DAEMON_OPTS="$DAEMON_OPTS --external username:password@host:port"


Then it will work exactly as you requested!!

Although this hasn't been done yet, I don't see any reason why this won't work exactly as you requested. The testing for the external proxy support was done using another instance of http-replicator....
Back to top
View user's profile Send private message
VinnieNZ
Tux's lil' helper
Tux's lil' helper


Joined: 11 Mar 2004
Posts: 126
Location: New Zealand

PostPosted: Mon Nov 22, 2004 1:49 am    Post subject: Reply with quote

Agreed, the above does exactly what I want it to. FANTASTIC piece of work! :D

For people wondering where to find the above, its in /etc/conf.d/http-replicator.

Cheers flybynite!
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Tue Nov 23, 2004 3:47 am    Post subject: Reply with quote

Thanks, once you get this setup, let me know how it works...
Back to top
View user's profile Send private message
VinnieNZ
Tux's lil' helper
Tux's lil' helper


Joined: 11 Mar 2004
Posts: 126
Location: New Zealand

PostPosted: Mon Nov 29, 2004 2:47 am    Post subject: Reply with quote

flybynite wrote:
Thanks, once you get this setup, let me know how it works...


Have got this working on a few servers now and it seems to work great for what we do :)
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Mon Nov 29, 2004 3:28 am    Post subject: Reply with quote

VinnieNZ wrote:

Have got this working on a few servers now and it seems to work great for what we do :)


Thanks for the report!

A new version of http-replicator is due out in the next few days. It requires no changes to the RESUMECOMMAND and supports serving as a BINHOST for binary packages!
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Tue Nov 30, 2004 2:41 am    Post subject: Reply with quote

Http-Replicator 3.0 is here!!

New features:

Resuming downloads is supported client side. This means no changes to the RESUMECOMMAND in /etc/make.conf are required or desired!

Dir serving. Http-replicator will serve a dir of your choosing. One example is the /usr/portage/packages/All dir which holds portage's binary packages!! Http-Replicator is your best option to fully support multiple gentoo boxes on a LAN of any size!! Emerge or transfer binary packages to http-replicator and serve them to your LAN!!

Cache serving. Http-replicator will serve the cache dir. Want to know what is in the cache or download a particular file from the cache? Just point your browser at http-replicator!

Repcacheman updates. Repcacheman will automatically install http-replicator and md5 check your existing packages! Uses same config as http-replicator for ease of use!

Smaller runtime memory footprint. Lean and mean!


I need a few experienced users to upgrade and report before I update the HOWTO! If you need the full HOWTO to upgrade then it will be out in a couple of days, just hang on ....

Download Latest.tar.bz2
and upgrade replicator to ver 3.0.

Comment out the old changes to RESUMECOMMAND in /etc/make.conf by adding the # before RESUMECOMMAND
leave the line: http_proxy=yourbox.com:8080

If you wish, check the other defaults in /etc/conf.d/http-replicator

Restart http-replicator:
/etc/init.d/http-replicator restart



Point your browser to http://localhost:8080/All to see your binary packages. Set client /etc/make.conf PORTAGE_BINHOST=http://serverbox.com:8080/All
to activate fetching binaries and then emerge -gK xxx to test binary package hosting. See man make.conf for more info.

Point your browser to http://localhost:8080 to see your cache dir.

Try to interrupt an emerge fetch with control-c and then restart. Portage should resume the download where it left off.



And then report your findings!!


Last edited by flybynite on Sun Dec 19, 2004 8:47 pm; edited 2 times in total
Back to top
View user's profile Send private message
jbpros
Tux's lil' helper
Tux's lil' helper


Joined: 05 May 2004
Posts: 133
Location: Brussels, Belgium

PostPosted: Thu Dec 02, 2004 3:59 am    Post subject: Reply with quote

Before anything, I have to thank you flybynite as http-replicator is running for months on two LANs I manage and it saved lot of expensive bandwidth! :)

Here is a "little" enhancement request: would it be possible to configure http-replicator in a way that it uses an external command to fetch packages instead of its internal http client?

What I would like to do is to have a deltup system between http-replicator and the remote http servers. This could save yet more bandwidth.

The first issue I can think of is that the resume system would stop working. Maybe you could add two configuration directives: one containing the command to call for normal downloads and another for resumed downloads. It could go even further by adding a third directive to disable resuming (deltup cannot resume downloads AFAIK).

I don't know if it's feasable, realistic.. anyway i think that reducing trafic usage is always a good thing :)

Again thank you!
Back to top
View user's profile Send private message
dripton
n00b
n00b


Joined: 16 Aug 2002
Posts: 65
Location: Virginia USA

PostPosted: Thu Dec 02, 2004 3:29 pm    Post subject: Reply with quote

flybynite:

The name of your archive is a bit misleading. Maybe bump the "1.7" to "3.0"

After untarring your archive under my PORTAGE_OVERLAY directory (/usr/local/portage), "emerge http-replicator" died with
"!!! Security Violation: A file exists that is not in the manifest.
!!! File: files/repcacheman" But this turned out to be cruft from the previous http-replicator tarball, not present in the latest one, and easy to clean up by just removing that file. It's probably a good idea to delete an existing $PORTAGE_OVERLAY/net-misc/http-replicator directory before expanding the new tarball.

Resuming broken downloads worked great for me. I tested breaking and resuming both a file that existed in the cache, and a remote download, with no RESUMECOMMAND.

The cache directory is viewable with a web browser as advertised.

I haven't tested serving binary packages yet.

Thanks again for a very useful program.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Sat Dec 04, 2004 9:48 pm    Post subject: Reply with quote

jbpros wrote:
Before anything, I have to thank you flybynite as http-replicator is running for months on two LANs I manage and it saved lot of expensive bandwidth! :)


Thanks!


jbpros wrote:

Here is a "little" enhancement request: would it be possible to configure http-replicator in a way that it uses an external command to fetch packages instead of its internal http client?



I've tried this before. It seems no external client can pass the necessary headers, errors, and data in a way that is robust and useful. I've even got to the point of trying to hack wget to work, but then if you have to hack it to get it to work why even use it? What do you think this will gain?


jbpros wrote:

What I would like to do is to have a deltup system between http-replicator and the remote http servers. This could save yet more bandwidth.



I'm not sure what exactly your trying to accomplish here. But that seems backwards. Can't deltup just retrieve the updates through replicator so you only have to download each update once? Is that what your trying to achieve?
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Sat Dec 04, 2004 9:57 pm    Post subject: Reply with quote

dripton wrote:
flybynite:

The name of your archive is a bit misleading. Maybe bump the "1.7" to "3.0"


I agree but won't change this yet.

Right now the archive contains http-replicator ebuilds versions 1,2, and the testing version 3. Should I name it http-replicator-1and2andtestingversion3-ebuilds.tar.gz? Thats why I just left it for now.

dripton wrote:

After untarring your archive under my PORTAGE_OVERLAY directory (/usr/local/portage), "emerge http-replicator" died with
"!!! Security Violation: A file exists that is not in the manifest.
!!! File: files/repcacheman"


Yes, I renamed a file for the first time since 1.0

dripton wrote:

It's probably a good idea to delete an existing $PORTAGE_OVERLAY/net-misc/http-replicator directory before expanding the new tarball.


Agreed. HOWTO updated....

Thanks for the report!!
Back to top
View user's profile Send private message
jbpros
Tux's lil' helper
Tux's lil' helper


Joined: 05 May 2004
Posts: 133
Location: Brussels, Belgium

PostPosted: Sun Dec 05, 2004 4:14 pm    Post subject: Reply with quote

Quote:
I'm not sure what exactly your trying to accomplish here. But that seems backwards. Can't deltup just retrieve the updates through replicator so you only have to download each update once? Is that what your trying to achieve?

My idea was to reduce internet trafic by making http replicator use the deltup system trasnparently. Yes I could make portage use the deltup script and specify the replicator proxy in this script but that would not be transparent and depend on the requesting host instead of the replicator "server".

Practically when an host requests some package to http-replicator, replicator checks if the package is in cache, if yes, then it perfoms as usually. If not it would delegate the download process to deltup which could base the download on existing packages in http replicator cache.

Theorically this is possible. Maybe this could be achieved by implementing the deltup system within http-replicator.

The advantage of this solution is the transparency. It's quite a good point in a network containing several gentoo nodes.

That's my idea. Do you think it is completly crazy, hard to do or sort of realistic?
Back to top
View user's profile Send private message
kmarasco
n00b
n00b


Joined: 25 Jul 2003
Posts: 65
Location: Fernandina Beach, FL

PostPosted: Tue Dec 07, 2004 5:41 am    Post subject: Minor issue Reply with quote

One issue that I came across was with programs with fetch restrictions, such as sun-jdk. Even though the source packages are in the http-replicator cache, they are not seen by portage, and must be moved or copied back to distfiles in order for the emerge to move forward.

I noticed this when doing an "emerge -eD world".
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Tue Dec 07, 2004 9:14 am    Post subject: Re: Minor issue Reply with quote

kmarasco wrote:
One issue that I came across was with programs with fetch restrictions, such as sun-jdk. Even though the source packages are in the http-replicator cache, they are not seen by portage, and must be moved or copied back to distfiles in order for the emerge to move forward.


You are correct.

This is entirely a portage weakness, not at all an http-replicator issue. I have a work around for some of portages goofiness.

Make sure you have the /etc/portage/mirrors file portage work-around. It was added to the howto, so it might have been added after you first installed if you're a long time user.

This won't cure all portage's ill's, just some. It makes portage check replicator for RESTRICT=nomirror but not for the RESTRICT=nofetch for packages like sun's java (sun-jdk).


Feel free to file a bug. Portage should check a local mirror even for "no-fetch" packages in my opinion!! Don't even mention http-replicator. Just mention the fact that not checking the "local mirror" for all packages defeats the purpose of a local mirror!!


Here is the nomirror work around from the howto in case you or anyone else needs it:

Quote:

Also, some packages in portage have a RESTRICT="nomirror" option which will prevent portage from checking replicator for those packages. The following will override this behavior. Create the file "/etc/portage/mirrors" containing:
Code:

# Http-Replicator Override for FTP and RESTRICT="nomirror packages
local http://gentoo.osuosl.org

You can replace gento.osuosl.org with your favorite HTTP:// mirror. If you already have a local setting, don't worry, as long as it is an http mirror this will still be effective.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Tue Dec 07, 2004 10:31 am    Post subject: Reply with quote

jbpros wrote:

Theorically this is possible. Maybe this could be achieved by implementing the deltup system within http-replicator.

That's my idea. Do you think it is completly crazy, hard to do or sort of realistic?


Http-Replicator has more development in store before I would consider it mature. Maybe after I consider it mature I would look into deltup again. Until then it would seem your best option is to use them together but as separate programs...
Back to top
View user's profile Send private message
johntramp
Guru
Guru


Joined: 03 Feb 2004
Posts: 457
Location: New Zealand

PostPosted: Tue Dec 07, 2004 12:30 pm    Post subject: Reply with quote

I am sorry if this has been asked before, but is it possible to use this from the beginning of the gentoo install process?
[edit]yes, It seems that installing it with --nodeps before bootstrap works, I will remerge it again later after bootstrap / emerge system. :) [/edit]
Back to top
View user's profile Send private message
gringo
Advocate
Advocate


Joined: 27 Apr 2003
Posts: 3793

PostPosted: Sat Dec 11, 2004 9:30 pm    Post subject: Reply with quote

many thanks for this great tool, was really helpful here and is running fine for two weeks now.

One question: how would you manage packages for several different archs ?
I have a powerbook here and im emerging with ACCEPT_KEYWORDS="ppc" + repcacheman for ppc specific stuff to have them in cache, but maybe theres a better solution ...

TIA
_________________
Error: Failing not supported by current locale
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Sat Dec 11, 2004 11:53 pm    Post subject: Reply with quote

I'm not really sure what you need. Replicator should work fine with different arch's. Do you mean repcacheman - it works on different arch's...
Back to top
View user's profile Send private message
gringo
Advocate
Advocate


Joined: 27 Apr 2003
Posts: 3793

PostPosted: Wed Dec 15, 2004 10:19 am    Post subject: Reply with quote

sorry for my stupid question, i forgot that this app _is_ a proxy so any ppc specific packages will also be cached by the server ( after repcacheman of course) even if you download on other machine of your network.

many thanx again, great app, thats what what i was looking for ;)

cheers
_________________
Error: Failing not supported by current locale
Back to top
View user's profile Send private message
woody77
n00b
n00b


Joined: 06 Feb 2004
Posts: 69
Location: SC Mtns, CA, USA

PostPosted: Sat Dec 18, 2004 11:30 am    Post subject: Reply with quote

Can anyone post a mirror to the downloads for this? The original poster's account appears to be suspended (bandwidth utilization reasons?)?

Thanks,
Woody
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Sun Dec 19, 2004 4:23 pm    Post subject: Reply with quote

woody77 wrote:
Can anyone post a mirror to the downloads for this? The original poster's account appears to be suspended (bandwidth utilization reasons?)?

Thanks,
Woody



Sorry, I decided to switch hosting providers. The links are back up at a temporary location.

md5sum http-replicator-flybynite-1.7.tar.bz2:
f2ef1b7ef73aa6657122748b238947ff

md5sum http-replicator-flybynite-1.6a.tar.bz2:
e02ce85d45f3774b6a98c49507cf8279
Back to top
View user's profile Send private message
hothead
Apprentice
Apprentice


Joined: 02 Jul 2004
Posts: 277
Location: /dev/core

PostPosted: Thu Dec 23, 2004 9:47 pm    Post subject: Reply with quote

Hi,

I've a problem when running repcacheman

Code:
Found 17950 ebuilds.

Extracting the checksums....
Missing digest: app-office/abiword-2.1.90
Missing digest: kde-base/kde-meta-3.3.2
Missing digest: kde-base/kdeaddons-meta-3.3.2
Missing digest: kde-base/kdeadmin-meta-3.3.2
Missing digest: kde-base/kdebase-3.1-r1
Missing digest: kde-base/kdebase-meta-3.3.2
Missing digest: kde-base/kdeedu-meta-3.3.2
Missing digest: kde-base/kdegames-meta-3.3.2
Missing digest: kde-base/kdegraphics-meta-3.3.2
Missing digest: kde-base/kdemultimedia-meta-3.3.2
Missing digest: kde-base/kdenetwork-meta-3.3.2
Missing digest: kde-base/kdepim-meta-3.3.2
Missing digest: kde-base/kdesdk-meta-3.3.2
Missing digest: kde-base/kdetoys-meta-3.3.2
Missing digest: kde-base/kdeutils-meta-3.3.2
Missing digest: kde-base/kdewebdev-meta-3.3.2
Missing digest: media-plugins/xmms-plugins-1.0.1
Missing digest: media-radio/drm-1.0.6
Missing digest: media-video/vdrplugin-mldonkey-0.0.4a
Missing digest: net-mail/mulberry-3.0.0_alpha5
Missing digest: net-mail/mulberry-3.0.0_beta9
Missing digest: net-news/klibido-0.2.0
Traceback (most recent call last):
  File "/usr/bin/repcacheman.py", line 162, in ?
    digestpath = os.path.dirname(digestpath)+"/files/digest-"+pv
  File "/usr/lib/python2.3/posixpath.py", line 119, in dirname
    return split(p)[0]
  File "/usr/lib/python2.3/posixpath.py", line 77, in split
    i = p.rfind('/') + 1
AttributeError: 'NoneType' object has no attribute 'rfind'


I've several overlays (gentoo-de, kde-metaebuilds, bmg-main, usr) beside the main gentoo tree - Maybe thats the reason.
I can deal with the missing digests but I don't know what the last Traceback is about. Does anyone knows how to fix this?

Thanks.

hothead
_________________
Got a question? - http://justfuckinggoogleit.com/
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3 ... 9, 10, 11 ... 22, 23, 24  Next
Page 10 of 24

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum