Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
rsyncing behind a firewall
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
TheHaas
n00b
n00b


Joined: 13 Jun 2002
Posts: 61

PostPosted: Tue Jun 08, 2004 2:01 pm    Post subject: rsyncing behind a firewall Reply with quote

I have a Gentoo machine behind a firewall that doesn't let rsync through. But I can ssh to a machine outside the firewall that can. Sounds like a solution for an SSH tunnel!

1. ) On the Gentoo machine, edit make.conf and change SYNC to:
Code:
SYNC="rsync://localhost:4040/gentoo-portage"

2.) Make an ssh tunnel from your Gentoo box to the remote box to go to rsync.gentoo.org, i.e.
Code:
ssh -L 4040:rsync.gentoo.org:873 remote.host.com

3.) Now rsync as usual from another prompt:
Code:
emerge rsync
Back to top
View user's profile Send private message
ydleiF
Apprentice
Apprentice


Joined: 15 Nov 2002
Posts: 170
Location: Southeast Michigan, USA

PostPosted: Tue Jun 08, 2004 3:46 pm    Post subject: Nice Reply with quote

Slick.

The way we do it here (I'm the admin) is that we have a proxy which allows connections to TCP port 873 which is what rsync uses.
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Wed Dec 01, 2004 3:53 am    Post subject: Reply with quote

Well, I just tried this after coming to the same set of commands myself.

First I created the tunnel to my external rsync-permitted machine:

Code:
$ ssh -L 4040:rsync.gentoo.org:873 user@remote.host.com


Then I tried rsync directly:

Code:
$ rsync rsync://localhost:4040/gentoo-portage
Welcome to hawk.gentoo.org
 
Server Address : 134.68.220.97
...etc


Then I tried emerge sync:

Code:
# SYNC="rsync://localhost:4040/gentoo-portage" emerge sync
>>> starting rsync with rsync://65.19.163.230/gentoo-portage...
>>> checking server timestamp ...
rsync: failed to connect to 65.19.163.230: Connection timed out
rsync error: error in socket IO (code 10) at clientserver.c(88)
>>> retry ...


Any idea why it's not retrieving the timestamp? I'd really like to get this working - emerge-webrsync really isn't ideal for me.

EDIT: Solved! You can't override SYNC from the command line. I'm not sure why I thought you could actually...
Back to top
View user's profile Send private message
statmobile
Apprentice
Apprentice


Joined: 16 Jul 2004
Posts: 286
Location: Chapel Hill, NC

PostPosted: Thu Dec 02, 2004 6:44 pm    Post subject: Reply with quote

Correct me if I'm wrong, but can't you just use the flag?
Code:
rsync --rsh=/usr/bin/ssh

_________________
Comp Specs:
Asus A7V(rev1.02)|AMDXP2100+@1.7GHz|Creative SB128PCI|32MB NVIDIA/TNT2AGP|512MBPC-133
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Thu Dec 02, 2004 8:27 pm    Post subject: Reply with quote

Only if you have ssh access to the final host you are trying to rsync to/from. In this case, we don't. This is about rsync tunnelled over ssh, not rsync using ssh. It's the inside-out problem...
Back to top
View user's profile Send private message
monotux
l33t
l33t


Joined: 09 Sep 2003
Posts: 751
Location: Stockholm, Sweden

PostPosted: Thu Dec 02, 2004 8:55 pm    Post subject: Reply with quote

a webrsync is the fastest (and easiest) way to do it, still :)
_________________
Computer science is no more about computers than astronomy is about telescopes.
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Thu Dec 02, 2004 9:44 pm    Post subject: Reply with quote

I disagree - I've been using emerge-webrsync for the last few weeks and it's hideously slow. It downloads the entire Portage tree every time. The first time I ran it, it took 2 hours! Using rsync properly is a far better solution - that's why Portage uses it.
Back to top
View user's profile Send private message
skander
n00b
n00b


Joined: 30 Dec 2004
Posts: 12

PostPosted: Tue Jan 25, 2005 5:15 pm    Post subject: maybe I'm confused too Reply with quote

The port forwarding method on the OP solves my requisite for not running an rsync server on my home machine (and thus opening up another port on the firewall), but shouldn't there be a way to integrate that into the SYNC="..." setting in make.conf, making it do the equivalent of:

rsync -e ssh myuser@myownmirror:/src /local/dest

I tried setting the RSYNC_RSH environment variable, but that seems to get ignored (presumably because of the "rsync://" prefix in $SYNC. Maybe I could play with setting $SYNC to something like:

SYNC="myuser@myhomecomputer.com:/usr/portage"
Back to top
View user's profile Send private message
skander
n00b
n00b


Joined: 30 Dec 2004
Posts: 12

PostPosted: Tue Jan 25, 2005 7:39 pm    Post subject: nope Reply with quote

Looks like this format for $SYNC is invalid:

SYNC="myuser@myhomecomputer.com:/usr/portage"

I looked through the emerge code and it only seems to accept "rsync://*" and "cvs://*" formats (and the metadata function). Sigh. I guess I'll be running an rsync server at home and just foward the port as outlined in the original posting. That at least lets me get away with not opening up the rsync port.
Back to top
View user's profile Send private message
monu
n00b
n00b


Joined: 26 Jan 2005
Posts: 1

PostPosted: Wed Jan 26, 2005 4:26 am    Post subject: Reply with quote

i get this error

>>> Starting retry 2 of 3 with rsync://rsync.gentoo.org/gentoo-portage
>>> checking server timestamp ...
rsync: getaddrinfo: rsync.gentoo.org 873: Temporary failure in name resolution
rsync error: error in socket IO (code 10) at clientserver.c(88)
>>> retry ...
Notice: (-3, 'Temporary failure in name resolution')


what should i do?
Back to top
View user's profile Send private message
skander
n00b
n00b


Joined: 30 Dec 2004
Posts: 12

PostPosted: Wed Jan 26, 2005 11:46 am    Post subject: Reply with quote

Sounds like you don't have network access.
Back to top
View user's profile Send private message
sekopaa
n00b
n00b


Joined: 03 Jun 2004
Posts: 10
Location: Suomi

PostPosted: Tue Jul 05, 2005 6:39 pm    Post subject: Re: rsyncing behind a firewall Reply with quote

TheHaas wrote:
I have a Gentoo machine behind a firewall that doesn't let rsync through. But I can ssh to a machine outside the firewall that can. Sounds like a solution for an SSH tunnel!

1. ) On the Gentoo machine, edit make.conf and change SYNC to:
Code:
SYNC="rsync://localhost:4040/gentoo-portage"

2.) Make an ssh tunnel from your Gentoo box to the remote box to go to rsync.gentoo.org, i.e.
Code:
ssh -L 4040:rsync.gentoo.org:873 remote.host.com

3.) Now rsync as usual from another prompt:
Code:
emerge rsync


I'm in the same situation and would love to do this, but I have (probably an ssh config problem). My appologies, but I'm an ssh tunneling n00b.

Anyway steps 1 & 2 work fine, but when I "emerge sync" on my local machine I get:
Code:
>>> starting rsync with rsync://127.0.0.1:4040/gentoo-portage...
>>> checking server timestamp ...
rsync: connection unexpectedly closed (0 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(434)
>>> retry ...


on my remote machine I get this error at the same time:
Code:
channel 2: open failed: connect failed: Connection refused


It seems that my ssh tunnel allows me to get to my remote machine, but won't forward me on to the rsync site. I can "emerge sync" from the remote machine, so I know the rsync port is open. My only thought is that it is a ssh config problem. I've looked through the sshd_config file and added "AllowTcpForwarding yes" which sounded like it might do the trick, but no... Any ideas?

-Thanks-
_________________
You do ill if you praise, but worse if you censure, what you do not understand.
- Leonardo da Vinci
Back to top
View user's profile Send private message
TheHaas
n00b
n00b


Joined: 13 Jun 2002
Posts: 61

PostPosted: Wed Jul 06, 2005 1:56 pm    Post subject: Re: rsyncing behind a firewall Reply with quote

sekopaa wrote:

I'm in the same situation and would love to do this, but I have (probably an ssh config problem). My appologies, but I'm an ssh tunneling n00b.

Anyway steps 1 & 2 work fine, but when I "emerge sync" on my local machine I get:
Code:
>>> starting rsync with rsync://127.0.0.1:4040/gentoo-portage...
>>> checking server timestamp ...
rsync: connection unexpectedly closed (0 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(434)
>>> retry ...


on my remote machine I get this error at the same time:
Code:
channel 2: open failed: connect failed: Connection refused


It seems that my ssh tunnel allows me to get to my remote machine, but won't forward me on to the rsync site. I can "emerge sync" from the remote machine, so I know the rsync port is open. My only thought is that it is a ssh config problem. I've looked through the sshd_config file and added "AllowTcpForwarding yes" which sounded like it might do the trick, but no... Any ideas?

-Thanks-


I think that a port is still being blocked someone, which is strange since you can connect via ssh.

Put this in your $HOME/.ssh/config file:
Code:
ForwardAgent yes
Back to top
View user's profile Send private message
sekopaa
n00b
n00b


Joined: 03 Jun 2004
Posts: 10
Location: Suomi

PostPosted: Wed Jul 06, 2005 6:02 pm    Post subject: Re: rsyncing behind a firewall Reply with quote

TheHaas wrote:
I think that a port is still being blocked someone, which is strange since you can connect via ssh.

Put this in your $HOME/.ssh/config file:
Code:
ForwardAgent yes


That did the trick! :D

On a side note, you can set this flag system wide in /etc/ssh/ssh_config. I was only looking at /etc/ssh/sshd_config :oops:

Thanks!
_________________
You do ill if you praise, but worse if you censure, what you do not understand.
- Leonardo da Vinci
Back to top
View user's profile Send private message
TheHaas
n00b
n00b


Joined: 13 Jun 2002
Posts: 61

PostPosted: Wed Jul 06, 2005 6:57 pm    Post subject: Re: rsyncing behind a firewall Reply with quote

sekopaa wrote:

That did the trick! :D

On a side note, you can set this flag system wide in /etc/ssh/ssh_config. I was only looking at /etc/ssh/sshd_config :oops:

Thanks!


I'm glad that it worked.

yeah -- ssh_config is for when you connect via client and sshd_config are options for your ssh server. What's cool is that you can set your own settings in ~/.ssh/config and let your other users suffer with the defaults. :)
Back to top
View user's profile Send private message
dingfelder
Apprentice
Apprentice


Joined: 27 Jun 2007
Posts: 162
Location: New Zealand

PostPosted: Thu Jun 28, 2007 12:22 am    Post subject: Reply with quote

I followed the instructions here and opened a tunnel, and now emerge works :)

but emerge --sync gives:

emerge --sync
>>> Starting rsync with rsync://127.0.0.1:4040/gentoo-portage...
>>> Checking server timestamp ...
rsync: failed to connect to 127.0.0.1: Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(104) [receiver=2.6.9]
>>> Retrying...


>>> Starting retry 1 of 3 with rsync://127.0.0.1:4040/gentoo-portage
>>> Checking server timestamp ...
rsync: failed to connect to 127.0.0.1: Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(104) [receiver=2.6.9]
>>> Retrying...

emerge --sync does work on the machine I am sshing to

thoughts?
Back to top
View user's profile Send private message
dingfelder
Apprentice
Apprentice


Joined: 27 Jun 2007
Posts: 162
Location: New Zealand

PostPosted: Thu Jul 05, 2007 4:20 am    Post subject: Reply with quote

bump... nobody knows...
Back to top
View user's profile Send private message
Fukai
Tux's lil' helper
Tux's lil' helper


Joined: 05 Jul 2006
Posts: 149
Location: São Paulo, Brazil

PostPosted: Wed Jul 11, 2007 12:26 pm    Post subject: Reply with quote

Hi, this is not a solution but another solution ^_^ what I do is

Code:

ssh example.org -ND 1080
tsocks emerge  --sync


You don't need to change the SYNC var, all is done by tsocks it 'route' all traffic by the sock created by ssh
(you can use it with any app in a unifiqued fashion like tsocks wget, tsocks bla bla bla) you need to config the port
in /etc/socks/tsocks.conf
Back to top
View user's profile Send private message
stanleysantos
n00b
n00b


Joined: 16 Aug 2007
Posts: 1

PostPosted: Thu Aug 16, 2007 5:29 pm    Post subject: Reply with quote

I had to put the IP adress instead of rsync.gentoo.org ==>

Code:

ssh -l root -L 4040:134.68.220.73:873 ...


It works now!
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum