Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
ANNOUNCE: gentooget
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
Donald Munro
n00b
n00b


Joined: 08 Oct 2010
Posts: 2

PostPosted: Fri Oct 08, 2010 10:53 am    Post subject: ANNOUNCE: gentooget Reply with quote

gentooget 0.2

Available from http://github.com/donaldmunro/gentooget (git clone "http://github.com/donaldmunro/gentooget.git")

gentooget is a replacement download script for Gentoo Linux portage which acts as a facade for the
aria2 downloader (http://aria2.sourceforge.net/) to enable simultaneous multi-part downloads from
different Gentoo mirrors thus improving download speed.
It also provides a facility to switch between 2 different (probably ADSL) Internet connections. This
is desirable in dumbassed countries where you pay per Gb for ADSL and the costs for local only
are a order of magnitude cheaper than international access. This allows use of a local connection
where possible but if a download fails where the file is not available on a local mirror then
gentooget switches to the international connection, downloads the file and then switches back to the
local connection.

gentooget [-h --help] [-v --version] [-V --verbose] [-D --debug] [-T --true] [-c --continue] [-u --url=]
[-d directory=] [-f --file=] [-a --aria=] [-l --local=] [-i --international=]
-T --true : Always return success even if a download fails.
-h --help : Display help.
-c --continue : Continue download (if not specified then existing file will be overwritten)
Specify this in the RESUMECOMMAND variable in make.conf
-u --url= : The url of the file to be downloaded. This should be passed as ${URI}
in the FETCHCOMMAND and RESUMECOMMAND variables in the make.conf file.
-d directory= : The destination directory to download the file to.
-f --file= : The name of the destination file (full path is specified by -d and -f).
-a --aria= : Path to the aria2 binary if it is not in the PATH.
-l --local= : Path to a script or executable that switches to a local Internet
connection.
-i --international= : Path to a script or executable that switches to a international
Internet connection.
-I --interface= : The interface being switched by --local/--international (defaults to ppp0)
If it is not ppp0 then it must be specified otherwise gentooget cannot
check to see if the interface is up before continuing download.

Enviroment Variables
====================
GENTOO_MIRRORS : Contains a list of urls of mirrors for Gentoo portage dist locations
separated by a space. This is a Portage standard variable and is used if
local and international is not specified in the command line.
Example: GENTOO_MIRRORS="http://ftp.heanet.ie/pub/gentoo/ ftp://ftp.is.co.za//mirror/ftp.gentoo.org"

LOCAL_MIRRORS : Contains a list of urls of local mirrors for Gentoo portage dist locations
Example: LOCAL_MIRRORS="ftp://ftp.is.co.za//mirror/ftp.gentoo.org ftp://ftp.up.ac.za/mirrors/gentoo.org/gentoo"

INTERNATIONAL_MIRRORS : Contains a list of urls of international mirrors for Gentoo portage dist locations
Example: INTERNATIONAL_MIRRORS="http://ftp.heanet.ie/pub/gentoo/ ftp://mirrors.rit.edu/gentoo/"

INTERNAL_MIRRORS : Contains a list of urls of internal network urls to distfiles directories
Example: INTERNAL_MIRRORS="ftp://192.168.0.1/portage"
Internal mirrors are currently assumed to be on a different interface eg eth1 as opposed to ppp0
therefore no connection switching is applied before internal fetches.

Files
=====
~/GENTOO_MIRRORS
May contain the GENTOO_MIRRORS, LOCAL_MIRRORS and INTERNATIONAL_MIRRORS variables (see Enviroment Variables)
Example
cat ~/GENTOO_MIRRORS
GENTOO_MIRRORS="http://ftp.heanet.ie/pub/gentoo/ ftp://ftp.is.co.za//mirror/ftp.gentoo.org"
LOCAL_MIRRORS="ftp://ftp.is.co.za//mirror/ftp.gentoo.org ftp://ftp.up.ac.za/mirrors/gentoo.org/gentoo"
INTERNATIONAL_MIRRORS="http://ftp.heanet.ie/pub/gentoo/ ftp://mirrors.rit.edu/gentoo/"

/etc/make.conf
This is the Portage make.conf file. It should include the GENTOO_MIRRORS variable and optionally
the LOCAL_MIRRORS and INTERNATIONAL_MIRRORS variables if local/international access is used and these
are not specified as environment variables or in ~/GENTOO_MIRRORS.
Additionally the FETCHCOMMAND and RESUMECOMMAND variables must be changed to use gentooget.

Example make.conf (only showing entries pertinent to gentooget)
FETCHCOMMAND="/usr/bin/gentooget.py -V -D -d '${DISTDIR}' -f '${FILE}' -u '${URI}'"
RESUMECOMMAND="/usr/bin/gentooget.py -V -D -c -d '${DISTDIR}' -f '${FILE}' -u '${URI}'"
INTERNAL_MIRRORS="ftp://192.168.0.1/portage"
GENTOO_MIRRORS="ftp://ftp.is.co.za//mirror/ftp.gentoo.org ftp://ftp.up.ac.za/mirrors/gentoo.org/gentoo http://ftp.heanet.ie/pub/gentoo/"

Files
=====
~/GENTOO_MIRRORS
May contain the GENTOO_MIRRORS, LOCAL_MIRRORS and INTERNATIONAL_MIRRORS variables (see Enviroment Variables)
Example
cat ~/GENTOO_MIRRORS
GENTOO_MIRRORS="http://ftp.heanet.ie/pub/gentoo/ ftp://ftp.is.co.za//mirror/ftp.gentoo.org"
LOCAL_MIRRORS="ftp://ftp.is.co.za//mirror/ftp.gentoo.org ftp://ftp.up.ac.za/mirrors/gentoo.org/gentoo"
INTERNATIONAL_MIRRORS="http://ftp.heanet.ie/pub/gentoo/ ftp://mirrors.rit.edu/gentoo/"

/etc/make.conf
This is the Portage make.conf file. It should include the GENTOO_MIRRORS variable and optionally
the LOCAL_MIRRORS and INTERNATIONAL_MIRRORS variables if local/international access is used and these
are not specified as environment variables or in ~/GENTOO_MIRRORS.
Additionally the FETCHCOMMAND and RESUMECOMMAND variables must be changed to use gentooget.

Example make.conf (only showing entries pertinent to gentooget)
FETCHCOMMAND="/usr/bin/gentooget.py -V -D -d '${DISTDIR}' -f '${FILE}' -u '${URI}'"
RESUMECOMMAND="/usr/bin/gentooget.py -V -D -c -d '${DISTDIR}' -f '${FILE}' -u '${URI}'"
INTERNAL_MIRRORS="ftp://192.168.0.1/portage"
GENTOO_MIRRORS="ftp://ftp.is.co.za//mirror/ftp.gentoo.org ftp://ftp.up.ac.za/mirrors/gentoo.org/gentoo http://ftp.heanet.ie/pub/gentoo/"


Example make.conf using local/international switching
FETCHCOMMAND="/usr/bin/gentooget.py -V -D -d '${DISTDIR}' -f '${FILE}' -u '${URI}' -l /etc/ppp/local -i /etc/ppp/shaped"
RESUMECOMMAND="/usr/bin/gentooget.py -V -D -c -d '${DISTDIR}' -f '${FILE}' -u '${URI}' -l /etc/ppp/local -i /etc/ppp/shaped"
INTERNAL_MIRRORS="ftp://192.168.0.1/portage"
GENTOO_MIRRORS="ftp://ftp.is.co.za//mirror/ftp.gentoo.org ftp://ftp.up.ac.za/mirrors/gentoo.org/gentoo http://ftp.heanet.ie/pub/gentoo/"
LOCAL_MIRRORS="ftp://ftp.is.co.za//mirror/ftp.gentoo.org ftp://ftp.up.ac.za/mirrors/gentoo.org/gentoo"
INTERNATIONAL_MIRRORS="http://ftp.heanet.ie/pub/gentoo/ ftp://mirrors.rit.edu/gentoo/"
(assumes the scripts /etc/ppp/local.sh and /etc/ppp/international.sh switch to local and international
Internet connections)

Notes
Emerge will call gentooget as user portage so if using local/international switching with ADSL the script
that does the switching will probably have to use sudo at least when stopping and starting PPPOE eg
sudo pppoe-start /etc/ppp/pppoe.conf.local. In this case user portage will need to be a real user ie
not have /bin/false for its shell in /etc/passwd and user portage will need to be added to group
wheel (/etc/group) if you use group based sudo in /etc/sudoers. For example
/etc/passwd: portage:x:250:250:portage:/var/tmp/portage:/bin/bash
/etc/group: wheel:x:10:root,portage
/etc/sudoers: %wheel ALL=(ALL) NOPASSWD: ALL
Back to top
View user's profile Send private message
Etal
Veteran
Veteran


Joined: 15 Jul 2005
Posts: 1696

PostPosted: Fri Oct 08, 2010 3:11 pm    Post subject: Reply with quote

Awesome, thanks! :D

I was looking for something like this because I have a mirror near me which is usually really fast (several MB/s), but at times it slows down to a crawl (a few KB/s), and it's really annoying because if I want to install a moderately-sized package, I have to edit make.conf to remove it from the list so that portage uses the next one instead.


I have one small request though - could you please package it up in a tarball, so that it could be made into a non-git ebuild, for convenience?

Edit: here's the ebuild:
gentooget-0.3.1.ebuild:
EAPI=3

DESCRIPTION="Script for downloading using aria2 from multiple Gentoo mirrors simultaneously"
HOMEPAGE="http://github.com/donaldmunro/gentooget"
SRC_URI="http://github.com/downloads/donaldmunro/${PN}/${P}.tar.bz2"

LICENSE="GPL-3"
SLOT="0"
KEYWORDS="~x86 ~amd64"
IUSE=""

DEPEND=""
RDEPEND="${DEPEND}
   <dev-lang/python-3.0
   net-misc/aria2"

S="${WORKDIR}/${PN}-0.3/"

src_install() {
   dobin ${PN} || die "Failed to install ${PN}"
}


Last edited by Etal on Mon Oct 11, 2010 1:53 pm; edited 4 times in total
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 18073

PostPosted: Fri Oct 08, 2010 9:43 pm    Post subject: Reply with quote

Moved from Gentoo Chat to Unsupported Software.
_________________
Those who know what's best for us must rise and save us from ourselves.
Back to top
View user's profile Send private message
Etal
Veteran
Veteran


Joined: 15 Jul 2005
Posts: 1696

PostPosted: Fri Oct 08, 2010 10:48 pm    Post subject: Reply with quote

Also, could you make this thing optional?

Code:
298       if status != 0:
299          if VERBOSE:
300             print("Downloading %s/%s failed. Attempting Portage downman" % (address, name))
301          options = options[0:urlStart]
302 #         options.append("-s")
303 #         options.append('1').
304          options.append('http://www.voidspace.org.uk/cgi-bin/voidspace/downman.py?file=' + name)
305          if DEBUG:
306             print('downman: ' + str(options))
307          status = subprocess.call(options)
               ...
Back to top
View user's profile Send private message
tomk
Bodhisattva
Bodhisattva


Joined: 23 Sep 2003
Posts: 7221
Location: Sat in front of my computer

PostPosted: Sat Oct 09, 2010 9:55 am    Post subject: Re: ANNOUNCE: gentooget Reply with quote

Donald Munro wrote:
Notes
Emerge will call gentooget as user portage so if using local/international switching with ADSL the script
that does the switching will probably have to use sudo at least when stopping and starting PPPOE eg
sudo pppoe-start /etc/ppp/pppoe.conf.local. In this case user portage will need to be a real user ie
not have /bin/false for its shell in /etc/passwd and user portage will need to be added to group
wheel (/etc/group) if you use group based sudo in /etc/sudoers. For example
/etc/passwd: portage:x:250:250:portage:/var/tmp/portage:/bin/bash
/etc/group: wheel:x:10:root,portage
/etc/sudoers: %wheel ALL=(ALL) NOPASSWD: ALL


Adding the portage user to wheel and then allowing wheel to use sudo with no password is a really bad idea, one malicious ebuild in an overlay could completely mess up your system. If you need to go down this route it is possible to allow sudo to be used for one command for a particular user without adding them to the wheel group.
_________________
Search | Read | Answer | Report | Strip
Back to top
View user's profile Send private message
Donald Munro
n00b
n00b


Joined: 08 Oct 2010
Posts: 2

PostPosted: Sun Oct 10, 2010 9:02 pm    Post subject: Re: ANNOUNCE: gentooget Reply with quote

Thanks for the suggestions. I have added a binary tar.bz2 download and changed the documentation for the sudo of pppoe-start to recommend per command permissions for user portage. Thanks to GitHub user AlecM for adding an option to make downman.py optional and adding color output.
I have also made a fix to delete the .aria file which aria2 creates and which causes it to attempt to restart a failed download from the same url it was downloading from the last time it aborted/failed and also added a fix to check if the file size is not zero after a download.
Back to top
View user's profile Send private message
Etal
Veteran
Veteran


Joined: 15 Jul 2005
Posts: 1696

PostPosted: Mon Oct 11, 2010 1:38 pm    Post subject: Reply with quote

Great! I added the ebuild above :)
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum