innfeed(8)


NAME

   innfeed, imapfeed - Multi-host, multi-connection, streaming NNTP feeder

SYNOPSIS

   innfeed [-ChmMvxyz] [-a spool-dir] [-b directory] [-c config-file] [-d
   log-level] [-e bytes] [-l logfile] [-o bytes] [-p pid-file] [-s
   command] [-S status-file] [file]

DESCRIPTION

   innfeed implements the NNTP protocol for transferring news between
   computers.  It handles the standard IHAVE protocol as well as the
   CHECK/TAKETHIS streaming extension.  innfeed can feed any number of
   remote hosts at once and will open multiple connections to each host if
   configured to do so.  The only limitations are the process limits for
   open file descriptors and memory.

   As an alternative to using NNTP, INN may also be fed to an IMAP server.
   This is done by using an executable called imapfeed, which is identical
   to innfeed except for the delivery process.  The new version has two
   types of connections:  an LMTP connection to deliver regular messages
   and an IMAP connection to handle control messages.

MODES

   innfeed has three modes of operation:  channel, funnel-file and batch.

   Channel mode is used when no filename is given on the command line, the
   input-file keyword is not given in the config file, and the -x option
   is not given.  In channel mode, innfeed runs with stdin connected via a
   pipe to innd.  Whenever innd closes this pipe (and it has several
   reasons during normal processing to do so), innfeed will exit.  It
   first will try to finish sending all articles it was in the middle of
   transmitting, before issuing a QUIT command.  This means innfeed may
   take a while to exit depending on how slow your peers are.  It never
   (well, almost never) just drops the connection.  The recommended way to
   restart innfeed when run in channel mode is therefore to tell innd to
   close the pipe and spawn a new innfeed process.  This can be done with
   "ctlinnd flush feed" where feed is the name of the innfeed channel feed
   in the newsfeeds file.

   Funnel-file mode is used when a filename is given as an argument or the
   input-file keyword is given in the config file.  In funnel-file mode,
   it reads the specified file for the same formatted information as innd
   would give in channel mode.  It is expected that innd is continually
   writing to this file, so when innfeed reaches the end of the file, it
   will check periodically for new information.  To prevent the funnel
   file from growing without bounds, you will need to periodically move
   the file to the side (or simply remove it) and have innd flush the
   file.  Then, after the file is flushed by innd, you can send innfeed a
   SIGALRM, and it too will close the file and open the new file created
   by innd.  Something like:

       innfeed -p <pathrun in inn.conf>/innfeed.pid my-funnel-file &
       while true; do
           sleep 43200
           rm -f my-funnel-file
           ctlinnd flush funnel-file-site
           kill -ALRM `cat <pathrun>/innfeed.pid`
       done

   Batch mode is used when the -x flag is used.  In batch mode, innfeed
   will ignore stdin, and will simply process any backlog created by a
   previously running innfeed.  This mode is not normally needed as
   innfeed will take care of backlog processing.

CONFIGURATION

   innfeed expects a couple of things to be able to run correctly:  a
   directory where it can store backlog files and a configuration file to
   describe which peers it should handle.

   The configuration file is described in innfeed.conf(5).  The -c option
   can be used to specify a different file.  For each peer (say, "foo"),
   innfeed manages up to 4 files in the backlog directory:

   · A foo.lock file, which prevents other instances of innfeed from
     interfering with this one.

   · A foo.input file which has old article information innfeed is reading
     for re-processing.

   · A foo.output file where innfeed is writing information on articles
     that could not be processed (normally due to a slow or blocked peer).

   · A foo file that is never created by innfeed, but if innfeed notices
     it, it will rename it to foo.input at the next opportunity and will
     start reading from it.  This lets you create a batch file and put it
     in a place where innfeed will find it.

   You should never alter the foo.input or foo.output files of a running
   innfeed.  The format of these last three files is one of the following:

       /path/to/article <message-id>
       @token@ <message-id>

   This is the same as the first two fields of the lines innd feeds to
   innfeed, and the same as the first two fields of the lines of the batch
   file innd will write if innfeed is unavailable for some reason.  When
   innfeed processes its own batch files, it ignores everything after the
   first two whitespace separated fields, so moving the innd-created batch
   file to the appropriate spot will work, even though the lines have
   extra fields.

   The first field can also be a storage API token.  The two types of
   lines can be intermingled; innfeed will use the storage manager if
   appropriate, and otherwise treat the first field as a filename to read
   directly.

   innfeed writes its current status to the file innfeed.status (or the
   file given by the -S option).  This file contains details on the
   process as a whole, and on each peer this instance of innfeed is
   managing.

   If innfeed is told to send an article to a host it is not managing,
   then the article information will be put into a file matching the
   pattern innfeed-dropped.*, with part of the file name matching the pid
   of the innfeed process that is writing to it.  innfeed will not process
   this file except to write to it.  If nothing is written to the file,
   then it will be removed if innfeed exits normally.  Otherwise, the file
   remains, and procbatch can be invoked to process it afterwards.

SIGNALS

   Upon receipt of a SIGALRM, innfeed will close the funnel file specified
   on the command line, and will reopen it (see funnel file description
   above).

   innfeed with catch SIGINT and will write a large debugging snapshot of
   the state of the running system.

   innfeed will catch SIGHUP and will reload both the config and the log
   files.  See innfeed.conf(5) for more details.

   innfeed will catch SIGCHLD and will close and reopen all backlog files.

   innfeed will catch SIGTERM and will do an orderly shutdown.

   Upon receipt of a SIGUSR1, innfeed will increment the debugging level
   by one; receipt of a SIGUSR2 will decrement it by one.  The debugging
   level starts at zero (unless the -d option it used), in which case no
   debugging information is emitted.  A larger value for the level means
   more debugging information. Numbers up to 5 are currently useful.

SYSLOG ENTRIES

   There are 3 different categories of syslog entries for statistics:
   host, connection and global.

   The host statistics are generated for a given peer at regular intervals
   after the first connection is made (or, if the remote is unreachable,
   after spooling starts).  The host statistics give totals over all
   connections that have been active during the given time frame.  For
   example (broken here to fit the page, with "vixie" being the peer):

       May 23 12:49:08 news innfeed[16015]: vixie checkpoint
           seconds 1381 offered 2744 accepted 1286 refused 1021 rejected 437
           missing 0 accsize 8506220 rejsize 142129 spooled 990
           on_close 0 unspooled 240 deferred 10/15.3 requeued 25
           queue 42.1/100:14,35,13,4,24,10

   The meanings of these fields are:

   seconds
     The time since innfeed connected to the host or since the statistics
     were reset by a "final" log entry.

   offered
     The number of IHAVE commands sent to the host if it is not in
     streaming mode.  The sum of the number of TAKETHIS commands sent when
     no-CHECK mode is in effect plus the number of CHECK commands sent in
     streaming mode (when no-CHECK mode is not in effect).

   accepted
     The number of articles which were sent to the remote host and
     accepted by it.

   refused
     The number of articles offered to the host that it indicated it did
     not want because it had already seen the message-ID.  The remote host
     indicates this by sending a 435 response to an IHAVE command or a 438
     response to a CHECK command.

   rejected
     The number of articles transferred to the host that it did not accept
     because it determined either that it already had the article or it
     did not want it because of the article's Newsgroups: or Distribution:
     header fields, etc.  The remote host indicates that it is rejecting
     the article by sending a 437 or 439 response after innfeed sent the
     entire article.

   missing
     The number of articles which innfeed was told to offer to the host
     but which were not present in the article spool.  These articles were
     probably cancelled or expired before innfeed was able to offer them
     to the host.

   accsize
     The number of bytes of all accepted articles transferred to the host.

   rejsize
     The number of bytes of all rejected articles transferred to the host.

   spooled
     The number of article entries that were written to the .output
     backlog file because the articles either could not be sent to the
     host or were refused by it.  Articles are generally spooled either
     because new articles are arriving more quickly than they can be
     offered to the host, or because innfeed closed all the connections to
     the host and pushed all the articles currently in progress to the
     .output backlog file.

   on_close
     The number of articles that were spooled when innfeed closed all the
     connections to the host.

   unspooled
     The number of article entries that were read from the .input backlog
     file.

   deferred
     The first number is the number of articles that the host told innfeed
     to retry later by sending a 431 or 436 response.  innfeed immediately
     puts these articles back on the tail of the queue.

     The second number is the average (mean) size of deferred articles
     during the previous logging interval

   requeued
     The number of articles that were in progress on connections when
     innfeed dropped those connections and put the articles back on the
     queue.  These connections may have been broken by a network problem
     or became unresponsive causing innfeed to time them out.

   queue
     The first number is the average (mean) queue size during the previous
     logging interval.  The second number is the maximum allowable queue
     size.  The third number is the percentage of the time that the queue
     was empty.  The fourth through seventh numbers are the percentages of
     the time that the queue was >0% to 25% full, 25% to 50% full, 50% to
     75% full, and 75% to <100% full.  The last number is the percentage
     of the time that the queue was totally full.

   If the -z option is used (see below), then when the peer stats are
   generated, each connection will log its stats too.  For example, for
   connection number zero (from a set of five):

       May 23 12:49:08 news innfeed[16015]: vixie:0 checkpoint
           seconds 1381 offered 596 accepted 274 refused 225
           rejected 97 accsize 773623 rejsize 86591

   If you only open a maximum of one connection to a remote, then there
   will be a close correlation between connection numbers and host
   numbers, but in general you cannot tie the two sets of number together
   in any easy or very meaningful way.  When a connection closes, it will
   always log its stats.

   If all connections for a host get closed together, then the host logs
   its stats as "final" and resets its counters.  If the feed is so busy
   that there is always at least one connection open and running, then
   after some amount of time (set via the config file), the host stats are
   logged as final and reset.  This is to make generating higher level
   stats from log files, by other programs, easier.

   There is one log entry that is emitted for a host just after its last
   connection closes and innfeed is preparing to exit.  This entry
   contains counts over the entire life of the process.  The "seconds"
   field is from the first time a connection was successfully built, or
   the first time spooling started.  If a host has been completely idle,
   it will have no such log entry.

       May 23 12:49:08 news innfeed[16015]: decwrl global
           seconds 1381 offered 34 accepted 22 refused 3 rejected 7
           missing 0 accsize 81277 rejsize 12738 spooled 0 unspooled 0

   The final log entry is emitted immediately before exiting.  It contains
   a summary of the statistics over the entire life of the process.

       Feb 13 14:43:41 news innfeed[22344]: ME global
           seconds 15742 offered 273441 accepted 45750 refused 222008
           rejected 3334 missing 217 accsize 93647166 rejsize 7421839
           spooled 10 unspooled 0

OPTIONS

   innfeed takes the following options.

   -a spool-dir
       The -a flag is used to specify the top of the article spool tree.
       innfeed does a chdir(2) to this directory, so it should probably be
       an absolute path.  The default is patharticles as set in inn.conf.

   -b directory
       The -b flag may be used to specify a different directory for
       backlog file storage and retrieval, as well as for lock files.  If
       the path is relative, then it is relative to pathspool as set in
       inn.conf.  The default is "innfeed".

   -c config-file
       The -c flag may be used to specify a different config file from the
       default value.  If the path is relative, then it is relative to
       pathetc as set in inn.conf.  The default is innfeed.conf.

   -C  The -C flag is used to have innfeed simply check the config file,
       report on any errors and then exit.

   -d log-level
       The -d flag may be used to specify the initial logging level.  All
       debugging messages go to stderr (which may not be what you want,
       see the -l flag below).

   -e bytes
       The -e flag may be used to specify the size limit (in bytes) for
       the .output backlog files innfeed creates.  If the output file gets
       bigger than 10% more than the given number, innfeed will replace
       the output file with the tail of the original version.  The default
       value is 0, which means there is no limit.

   -h  Use the -h flag to print the usage message.

   -l logfile
       The -l flag may be used to specify a different log file from
       stderr.  As innd starts innfeed with stderr attached to /dev/null,
       using this option can be useful in catching any abnormal error
       messages, or any debugging messages (all "normal" errors messages
       go to syslog).

   -m  The -m flag is used to turn on logging of all missing articles.
       Normally, if an article is missing, innfeed keeps a count, but logs
       no further information.  When this flag is used, details about
       message-IDs and expected path names are logged.

   -M  If innfeed has been built with mmap support, then the -M flag turns
       OFF the use of mmap(); otherwise, it has no effect.

   -o bytes
       The -o flag sets a value of the maximum number of bytes of article
       data innfeed is supposed to keep in memory.  This does not work
       properly yet.

   -p pid-file
       The -p flag is used to specify the file name to write the pid of
       the process into.  A relative path is relative to pathrun as set in
       inn.conf.  The default is innfeed.pid.

   -s command
       The -s flag specifies the name of a command to run in a subprocess
       and read article information from.  This is similar to channel mode
       operation, only that command takes the place usually occupied by
       innd.

   -S status-file
       The -S flag specifies the name of the file to write the periodic
       status to.  If the path is relative, it is considered relative to
       pathlog as set in inn.conf.  The default is innfeed.status.

   -v  When the -v flag is given, version information is printed to stderr
       and then innfeed exits.

   -x  The -x flag is used to tell innfeed not to expect any article
       information from innd but just to process any backlog files that
       exist and then exit.

   -y  The -y flag is used to allow dynamic peer binding.  If this flag is
       used and article information is received from innd that specifies
       an unknown peer, then the peer name is taken to be the IP name too,
       and an association with it is created.  Using this, it is possible
       to only have the global defaults in the innfeed.conf file, provided
       the peer name as used by innd is the same as the IP name.

       Note that innfeed with -y and no peer in innfeed.conf would cause a
       problem that innfeed drops the first article.

   -z  The -z flag is used to cause each connection, in a parallel feed
       configuration, to report statistics when the controller for the
       connections prints its statistics.

BUGS

   When using the -x option, the config file entry's initial-connections
   field will be the total number of connections created and used, no
   matter how many big the batch file, and no matter how big the max-
   connections field specifies.  Thus a value of 0 for initial-connections
   means nothing will happen in -x mode.

   innfeed does not automatically grab the file out of pathoutgoing.  This
   needs to be prepared for it by external means.

   Probably too many other bugs to count.

ALTERNATIVE

   An alternative to innfeed can be innduct, maintained by Ian Jackson and
   available at
   <http://www.chiark.greenend.org.uk/ucgi/~ian/git-manpage/innduct.git/innduct.8>.
   It is intended to solve a design issue in the way innfeed works.  As a
   matter of fact, the program feed protocol spoken between innd and
   innfeed is lossy:  if innfeed dies unexpectedly, articles which innd
   has written to the pipe to innfeed will be skipped.  innd has no way of
   telling which articles those are, no useful records, and no attempts to
   resend these articles.

FILES

   pathbin/innfeed
       The binary program itself.

   pathetc/innfeed.conf
       The configuration file.

   pathspool/innfeed
       The directory for backlog files.

HISTORY

   Written by James Brister <brister@vix.com> for InterNetNews.  Converted
   to POD by Julien Elie.

   Earlier versions of innfeed (up to 0.10.1) were shipped separately;
   innfeed is now part of INN and shares the same version number.

   $Id: innfeed.pod 9588 2013-12-19 17:46:41Z iulius $

SEE ALSO

   ctlinnd(8), inn.conf(5), innfeed.conf(5), innd(8), procbatch(8).





Opportunity


Personal Opportunity - Free software gives you access to billions of dollars of software at no cost. Use this software for your business, personal use or to develop a profitable skill. Access to source code provides access to a level of capabilities/information that companies protect though copyrights. Open source is a core component of the Internet and it is available to you. Leverage the billions of dollars in resources and capabilities to build a career, establish a business or change the world. The potential is endless for those who understand the opportunity.

Business Opportunity - Goldman Sachs, IBM and countless large corporations are leveraging open source to reduce costs, develop products and increase their bottom lines. Learn what these companies know about open source and how open source can give you the advantage.





Free Software


Free Software provides computer programs and capabilities at no cost but more importantly, it provides the freedom to run, edit, contribute to, and share the software. The importance of free software is a matter of access, not price. Software at no cost is a benefit but ownership rights to the software and source code is far more significant.


Free Office Software - The Libre Office suite provides top desktop productivity tools for free. This includes, a word processor, spreadsheet, presentation engine, drawing and flowcharting, database and math applications. Libre Office is available for Linux or Windows.





Free Books


The Free Books Library is a collection of thousands of the most popular public domain books in an online readable format. The collection includes great classical literature and more recent works where the U.S. copyright has expired. These books are yours to read and use without restrictions.


Source Code - Want to change a program or know how it works? Open Source provides the source code for its programs so that anyone can use, modify or learn how to write those programs themselves. Visit the GNU source code repositories to download the source.





Education


Study at Harvard, Stanford or MIT - Open edX provides free online courses from Harvard, MIT, Columbia, UC Berkeley and other top Universities. Hundreds of courses for almost all major subjects and course levels. Open edx also offers some paid courses and selected certifications.


Linux Manual Pages - A man or manual page is a form of software documentation found on Linux/Unix operating systems. Topics covered include computer programs (including library and system calls), formal standards and conventions, and even abstract concepts.