Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
The Gentoo+INND HowTo
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
apokalyptik
n00b
n00b


Joined: 18 Jun 2002
Posts: 62

PostPosted: Fri Dec 20, 2002 8:00 pm    Post subject: The Gentoo+INND HowTo Reply with quote

IMPORTANT NOTE: in this document commands are prefixed with a "# ", and lines in a file are prefaced with a "$ "

Purpose: to establish a working news server storing a limited number of newsgroups. Why? well... my ISP's newsgroup server has a fairly short message retention time frame, which causes me to miss some messages that I would rather not have. This is not going to tell you how to setup a fully functioning news server, the idea is better retrieval, not full newsgroup discussion participation. This may (or may not) start you on your way, but don’t come to me when you can't post something to your news server for everyone to see... you'll wind up with an "I told you so" and a swift kick in the butt.. The first thing we need to do is install our INND daemon. This isn’t as lengthy as compiling a windowing system, but still takes a healthy amount of time.
Lets get up to date:
Code:
   # emerge rsync

The whole reason we're here:
Code:
   # emerge inn

The init script for innd is WRONG, and wont allow innd to start as is... so we change /usr/lib/news/bin/innd to /usr/lib/news/bin/inndstart in the /etc/init.d/innd file. Now... innd is REALLY picky with ownership on the file system so we need to setup the appropriate directory and file owners
Code:
   # chmod u+s /usr/lib/news/bin/inndstart
   # chown -R news.news /var/log/news/
   # chown -R news.news /var/spool/news/

I know this is weird, but innd WILL NOT RUN unless /var/run is owned by the news user AND the news group
Code:
   # chown news.news /var/run

Now a lot of the work that you do on the actual file and directory structure of your news server HAS TO BE DONE AS THE USER NEWS if you have the wrong ownership on the right files you WILL END UP WITH A BROKEN NEWS SERVER (again, ITYS). Unfortunately for us to be able to su to the user news we need to give it a real shell
Code:
   # chsh -s /bin/bash news
   # su news
   # cd /var/spool/news

this file is not created by default, but is required to initialize innd's databases
Code:
   # touch /var/spool/news/history

now we initialize said databases
Code:
   # /usr/lib/news/bin/makedbz -i
   # mv history.n.dir history.dir
   # mv history.n.pag history.pag

First we give innd a hostname, and set the retention (in a number of days) for messages on the server
Code:
   # nano -w /etc/news/inn.conf
   $ domain: domain.com
   $ artcutoff: 10

We then need to define how innd will store our messages This way seems easiest, there are many others, and they can be specified down to a per newsgroup basis, but both those topics are outside the scope of this document.
Code:
   # nano -w /etc/news/storage.conf
   $ method timehash {
   $        newsgroups: *             
   $        class: 1
   $        size: 2,3200000
   $ }

Next add any ip addresses you want to allow full access to the news server to the "localhost" section of readers.conf... I'm on a 10.* local network, thus I want those IP addresses to be able to read. Access control, users, passwords, etc, are beyond the scope of this document.
Code:
   # nano -w /etc/news/readers.conf
   $ auth "localhost" {           
   $     hosts: "localhost, 127.0.0.1, stdin, 10.*"
   $     default: "<localhost>"     
   $ }

Now we exit the "news" user in favor of root and start innd
Code:
   # exit
   # /etc/init.d/innd start
   # ps waux

The process listing should show /usr/lib/news/bin/innd -p4 running as the user news. If not, then something went wrong. Soon we're going to need some perl modules later on, so... lets get that out of the way now.
Code:
   # perl -MCPAN -e shell

If you know what you are doing, and want to configure the CPAN module right now answer "yes" when asked "Are you ready for manual configuration? [yes]" or you can choose the easy way out and answer "no" which will simply go with some sensible defaults and let you get on with using the module. You only have to answer the question once.
Code:
   # install Net::NNTP
   # exit

Now we add each of our newsgroups-to-be into innd like so. (Where a.b.c is the name of the newsgroup we want to eventually be mirrored.)
Code:
   # ~news/bin/ctlinnd newgroup a.b.c y root

Now we're going to make one /etc/news/pullnews-a.b.c file for each of the newsgroups we want mirrored, like so:
Code:
   $ x.y.z
   $        a.b.c

Where x.y.z is the hostname, or the IP address of your ISP's newsgroup server. And a.b.c is the name of the newsgroup we are mirroring. Almost done, only two steps to go... We make a script to pull all the new news from a group on a regular basis. This script will be called at startup, and will run in the background.
Code:
   # nano -w /etc/news/pullnews-script
   $ #! /bin/bash
   $ DELAY=60        # The number, in seconds,
   $        # you want to wait before
   $        # re-syncing the groups again
   $ while [ 1 ]
   $  do
   $   for i in `ls /etc/news/pullnews* | grep -v pullnews-script`
   $    do
   $     ~news/bin/pullnews -q -c $i 2>&1 1>/dev/null
   $   done
   $  sleep $DELAY
   $ done
   # chmod 751 /etc/news/pullnews-script
   

Now, finally, our init script
Code:
   # nano -w /etc/init.d/pullnews
   $ #!/bin/bash
   $ /etc/news/pullnews-script &
   # chmod 711 /etc/init.d/pullnews

Lets add both innd and our new pullnews script to the default runlevel (so they start every bootup) and run our pullnews init script
Code:
   # rc-update add innd default
   # rc-update add pullnews default
   # /etc/init.d/pullnews
   # ps waux

our process list should show something similar to the following:
Quote:
news 14852 1.4 4.7 9472 5964 ? S 04:48 0:25 /usr/lib/news/bin/innd -p4
root 12372 0.6 0.8 2288 1084 pts/0 S 05:18 0:00 /bin/bash /etc/news/pullnews-script
root 2060 40.7 3.1 5012 3904 pts/0 S 05:18 0:00 pullnews

Congratulations. You’re done. Configure your newsgroup client to log into your local news server, and wait for the newsgroups to propagate. THIS WILL TAKE A LONG TIME TO HAPPEN the first time... since this pullnews a single threaded news puller it cannot get messages very fast. you also cannot mirror more then one newsgroup at a time due to the fact that pullnews will not allow multiple instances of itself. Alternatives are welcomed and greatly appreciated (email: apokalyptik@apokalyptik.com)
_________________
http://www.apokalyptik.com/
Back to top
View user's profile Send private message
zojas
Veteran
Veteran


Joined: 22 Apr 2002
Posts: 1138
Location: Phoenix, AZ

PostPosted: Sat Dec 21, 2002 9:08 pm    Post subject: Reply with quote

have you tried "suck"? it's a package which fetches messages from multiple newsgroups and posts them to the local news server. I used to use it a couple years ago to mirror newsgroups to my local server. it handles multiple newsgroups no problem.
Back to top
View user's profile Send private message
apokalyptik
n00b
n00b


Joined: 18 Jun 2002
Posts: 62

PostPosted: Sun Dec 22, 2002 6:12 am    Post subject: Reply with quote

I have, actually, but it still falls short on account that it only uses one data stream to the news server and is not a graceful program. I'm going to soon post a followup about how to use suck in place of pullnews (on account that it is faster.) it also has a major drawback. it stores the data on the filesystem before it is transferred to the news server.
_________________
http://www.apokalyptik.com/
Back to top
View user's profile Send private message
fisch09
n00b
n00b


Joined: 20 Feb 2003
Posts: 9

PostPosted: Thu Feb 20, 2003 1:41 pm    Post subject: innd - seg fault Reply with quote

i exactly followed the howto, but i alwayss get a seg fault when starting innd :(

i use inn 2.3.3

whats wrong?

tschau fisch
Back to top
View user's profile Send private message
apokalyptik
n00b
n00b


Joined: 18 Jun 2002
Posts: 62

PostPosted: Thu Feb 20, 2003 5:44 pm    Post subject: Reply with quote

i dont know. try recompiling innd with less strict c flags?
_________________
http://www.apokalyptik.com/
Back to top
View user's profile Send private message
scale
n00b
n00b


Joined: 09 Dec 2002
Posts: 1

PostPosted: Mon Mar 17, 2003 1:46 pm    Post subject: seg fault Reply with quote

i have the same problem.. with segmentation fault.. does anyone know how to solve this problem?

thx...
greez
Back to top
View user's profile Send private message
Woody
Guru
Guru


Joined: 30 Nov 2002
Posts: 592
Location: Milwaukee

PostPosted: Wed Aug 06, 2003 9:57 pm    Post subject: Reply with quote

Some comments:

* paths to the history files have changed.
* no need to install nntp perl module here.
* should try use pullnews as a cronjob and avoid the init.d
script, but this won't work because its too slow?

Thanks. This was very nice - i'd never be able to do it
without it.
Back to top
View user's profile Send private message
sixshot
n00b
n00b


Joined: 30 Apr 2004
Posts: 52

PostPosted: Tue Jun 15, 2004 6:40 am    Post subject: Reply with quote

Pulling the articles from the nntp server as I type this. So far, it's looking great. I did ran into a problem one time but that's only because I was careless when issuing some commands.

Code:
# ~news/bin/ctlinnd newgroup a.b.c y root

Make sure to correctly issue above command!

I completely missed it and thought it'd be no big deal. WRONG! It was a big deal it seems. It seems that the 'root' part of the text is vital as to who is able to issue the 'pullnews' command. I do not know the technical details to it but I couldn't issue the 'ihave' command at all regardless of what the control.ctl file says.

I'd like to also mention that those on ReiserFS may find that innwatch interferes with the normal operations of INN. It queries on inodes and, unless I am mistaken, ReiserFS doesn't use inodes at all. So...

Code:
# cd /etc/news
# vi innwatch.ctl

(look for the following line and comment it out)
!!! ${INNDF} -i . ! lt ! ${INNWATCHSPOOLNODES} ! throttle ! No space (spool inodes)

Anyway, things are looking great right now. But one question however. With the binary downloader program I use, it can connect multiple times to make downloading faster. Is it possible to run multiple instances of pullnews in order to pull articles faster from a single server (nntp server I use has 4 connection limit)? The server's throughput seems to be around the 50-60K/s area. To be able to make 3 simultaneous connections would make pulling the news article much faster given broadband download speed is in excess of 1.5Mbit/s. Any tips? Or suggestions?
_________________
One shall stand. One shall fall.
Back to top
View user's profile Send private message
apokalyptik
n00b
n00b


Joined: 18 Jun 2002
Posts: 62

PostPosted: Tue Jun 15, 2004 2:06 pm    Post subject: Reply with quote

way back when i wrote this i was looking for the same thing... i never found it...
_________________
http://www.apokalyptik.com/
Back to top
View user's profile Send private message
sixshot
n00b
n00b


Joined: 30 Apr 2004
Posts: 52

PostPosted: Thu Jun 17, 2004 7:47 am    Post subject: Reply with quote

Hmm... after quickly looking through the perl script, looks like a simple hack job can make multi-connections possible. I simply made a copy, edited the pid file it writes to, and off it goes. I guess the only problem it might have is with articles that're cross-posted. But there shouldn't be any issues regarding that anyway if you download from similar groups, right?
_________________
One shall stand. One shall fall.
Back to top
View user's profile Send private message
apokalyptik
n00b
n00b


Joined: 18 Jun 2002
Posts: 62

PostPosted: Thu Jun 17, 2004 2:08 pm    Post subject: Reply with quote

dunno... why dont you test it out and post the hacked script? im sure people are interested
_________________
http://www.apokalyptik.com/
Back to top
View user's profile Send private message
sixshot
n00b
n00b


Joined: 30 Apr 2004
Posts: 52

PostPosted: Mon Jun 21, 2004 4:08 am    Post subject: Reply with quote

Well, the only thing to look for is for the following code below:

Code:
my $lockfile = $ENV{HOME} . "/.pullnews.pid";

The perl script simply writes the PID to the file for temporary and runs the rest of the pullnews code as normal. After that is done, it deletes the file prior to exiting.

Code:
# cd /usr/lib/news/bin
# cp pullnews pullnews2
# vi pullnews2
(edit aforementioned line)
# pullnews -c /etc/news/alternate-pullnews-config

The above procedure should work normally. It makes utilizing your download stream more effectively if the server isn't fast enough. The only drawback is that this is a simple work-around to the problem. Plus I do not know if it's possible to make multiple connections in a single pullnews script.

If anyone else has any other ideas or concepts, I'm all ears for it.
_________________
One shall stand. One shall fall.
Back to top
View user's profile Send private message
sixshot
n00b
n00b


Joined: 30 Apr 2004
Posts: 52

PostPosted: Tue Jun 29, 2004 5:14 am    Post subject: Managing retention Reply with quote

One thing that other guides fail to mention, including a howto that I found through Google, is how retention works and how articles get deleted/expired/pushed.

Code:
# vi /etc/crontab

5  3  * * *     news    /usr/lib/news/bin/news.daily

The above crontab code means that the command news.daily is executed every day 5 minutes after the 3rd hour (3:05AM). It also states that the command will be executed under the username news. The username and command is necessary so that various statistics and management functions are carried out on a daily basis. Logs are rotated, articles are scanned for possible removal and deletion. Without this, articles will continue to pile up and will eat away at your HD space.

I personally do not know what configuration file (either in expire.ctl, storage.conf or in inn.conf) affects the nature of expiration. That subject is tricky and is very scattered. But I do know that after 3 days of housing mostly binary articles, that command deleted appropriate articles and I recovered free space needed to house newer articles.

Hope this tidbit helps those who were wondering about their news server keeping articles longer than a specified date.
_________________
One shall stand. One shall fall.
Back to top
View user's profile Send private message
Leafo
n00b
n00b


Joined: 29 Jun 2004
Posts: 68
Location: Denmark

PostPosted: Tue Jun 29, 2004 3:58 pm    Post subject: Reply with quote

As a recent converted linux-newbie after many years abuse of Windows (but I'm much better now, thankyouverymuch) I'd like to know how i would pull certain groups from several newsservers using the above methods for the sake of completeness.

Would editing pullnews-a.b.c. to something like

Code:

x.y.z. (<-server 1)
     a.b.c.
w.u.v. (<-server 2)
     a.b.c.


do it?
Back to top
View user's profile Send private message
sixshot
n00b
n00b


Joined: 30 Apr 2004
Posts: 52

PostPosted: Tue Jun 29, 2004 11:45 pm    Post subject: Reply with quote

Peter_G wrote:
As a recent converted linux-newbie after many years abuse of Windows (but I'm much better now, thankyouverymuch) I'd like to know how i would pull certain groups from several newsservers using the above methods for the sake of completeness.

Would editing pullnews-a.b.c. to something like
Code:

x.y.z. (<-server 1)
     a.b.c.
w.u.v. (<-server 2)
     a.b.c.

do it?

It should. Type in "man pullnews" for details regarding the pullnews command.
_________________
One shall stand. One shall fall.
Back to top
View user's profile Send private message
Leafo
n00b
n00b


Joined: 29 Jun 2004
Posts: 68
Location: Denmark

PostPosted: Wed Jun 30, 2004 3:00 pm    Post subject: Reply with quote

Well it does work. I read up on pullnews afterwards.

But as suck is in portage (masked) and I'm a fan of doing it by ebuilds (hey I'm a newbie), I think I'm gonna go try that one out first. ;-)
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum