wwwoffle.conf(5)

NAME
wwwoffle.conf - The configuration file for the proxy server for the World Wide Web Offline Explorer.

DESCRIPTION
The wwwoffle.conf file contains the configuration for the wwwoffled proxy HTTP server part of the World Wide Web Offline Explorer program.

The file is split into sections, each of which can be empty or contain one or more lines of configuration information. The sections are named and the order that they appear in the file is not important.

The general format of each of the sections is the same. The name of the section is on a line by itself to mark the start. The contents of the section are enclosed between a pair of lines containing the '{' and '}' characters or '[' and ']' characters. When the '{' and '}' characters are used the lines between contain configuration information. When the '[' and ']' characters are used the there must only be a single non-empty line between them that contains the name of a file (in the same directory) containing the configuration information.

Comments are marked by a '#' character at the start of the line and blank lines are also allowed, both are ignored.

The StartUp section can contain the following:

http-port = <port>
    The port number to use on the local host as the HTTP proxy (default=8080).

wwwoffle-port = <port>
    The port number to use on the local host as the WWWOFFLE control port (default=8081).

spool-dir = <dirname>
    The name of the spool directory to use for the cache. A subdirectory is created in this for each new web server that is contacted (default=/var/spool/wwwoffle).

run-uid = <username> | <uid> | none |
    The username of numeric uid to run the WWWOFFLE server with. To use this option, the program must be started by root.

run-gid = <groupname> | <gid> | none |
    The groupname of numeric gid to run the WWWOFFLE server with. To use this option, the program must be started by root.

use-syslog = yes | no
    The syslog facility can be used to log the important error messages. (default=yes).

password = <word> | none |
    The authorisation password that is required to use the wwwoffle program to configure the server or use the interactive control page (default=none). If this is not present, or set to an empty string or 'none' then there is no password required. If there is a password set, then the -c option to wwwoffle must be used, and the file wwwoffle.conf should be made readable only by authorised users.

max-servers = <integer>
    max-fetch-servers = <integer> The maximum number of servers processes that are started (default=8). The maximum number of server processes that are forked to fetch pages that were requested in offline mode (default = 4). The max-fetch-servers value must be less than max-servers or you will not be able to use WWWOFFLE interactively online while fetching.

dir-perm = <octal integer>
    The permissions to use when creating spool directories (default=0755), this overrides the umask value and must be in octal starting with a '0' character.

file-perm = <octal integer>
    The permissions to use when creating spool files (default=0644), this overrides the umask value and must be in octal starting with a '0' character.

run-online = <filename>
    The name of a program to run when switched to online mode (default=none).

run-offline = <filename>
    The name of a program to run when switched to offline mode (default=none).

run-autodial = <filename>
    The name of a program to run when switched to autodial mode (default=none). The programs run using the run-online, run-offline and run-autodial options are started with a single parameter set to the current mode.

The Options section contains other options that configure the server.

log-level = debug | info | important | warning | fatal
    The error messages that have a priority the same as that specified or greater are recorded on the output, either syslog or stderr (see wwwoffled(1)).

index-latest-days = <age>
    The maximum age in days of pages to show in the index of the latest pages (default=7).

request-changed = <time>
    While online pages will only be fetched if the cached version is older than this specified time in seconds (default=600). A negative value will force the cache to always be used in preference.

request-changed-once = yes | no
    While online pages will only be fetched if the cached version has not already been fetched once this session (default=yes). This option takes precedence over the request-changed option.

request-expired = yes | no
    While online pages that have expired will always be requested again (default=no). This option takes precedence over the request-changed and request-changed-once options.

request-no-cache = yes | no
    While online pages that ask not to be cached will always be requested again (default=no). This option takes precedence over the request-changed and request-changed-once options.

pragma-no-cache = yes | no
    Whether to request a new copy of a page if the request from the browser has "Pragma: no-cache" (default=yes). This option should be set to 'no' if when browsing offline all pages are re-requested by a 'broken' browser.

confirm-requests = yes | no
    Whether to return a page requiring user confirmation instead of automatically recording requests made while offline (default=no).

connect-timeout = <time>
    The time in seconds that WWWOFFLE will wait for the socket connection (default=30).

socket-timeout = <time>
    The time in seconds that WWWOFFLE will wait for data to arrive on a socket connection before timing out and giving an error (default=120 seconds).

connect-retry = yes | no
    If a connection cannot be made to a remote server then try again after a short delay (default=no).

ssl-allow-port = <integer>
    A port number that can be used for Secure Socket Layer (SSL) connections, e.g. https (default=none, for https use 443). There can be more than one of these entries to allow other ports.

no-lasttime-index = yes | no
    Disables creation of the lasttime/prevtime indexes (default=no).

intr-download-keep = yes | no
    If the browser closes the connection while online the currently downloaded part of the page should be kept (default=no).

intr-download-size = <integer>
    If the browser closes the connection while online the page should continue to download if smaller than this size in kB (default=1).

intr-download-percent = <integer>
    If the browser closes the connection while online the page should continue to download if more than this amount complete (default=80).

timeout-download-keep = yes | no
    If the server connection timeouts while reading then the currently downloaded partial page should be kept (default=no).

The FetchOptions section contains options that configure the automated downloading of pages. When pages are requested offline and downloaded later, there is a choice of whether to fetch stylesheets, images, frames, scripts or other objects referenced in the HTML.

stylesheets = yes | no
    Fetch the style sheets from these pages as well (default=no).

images = yes | no
    Fetch the images from these pages as well (default=no).

frames = yes | no
    Fetch the frames from these pages as well (default=no).

scripts = yes | no
    Fetch the scripts from these pages as well (default=no).

objects = yes | no
    Fetch the objects (e.g. Java class files) from these pages as well (default=no).

The ModifyHTML section contains options that control how the HTML that is provided from the cache is modified. They all rely on the HTML being syntactically correct HTML, if it is not then the result is undefined.

enable-modify-html = yes | no
    Enable the HTML modifications in this section (has a speed penalty) (default=no).

add-cache-info = yes | no
    At the bottom of all of the spooled pages the date that the page was cached and some buttons are to be added (default=no).

anchor-cached-begin =<HTML code>
    Anchors (links) that are cached are to have the specified HTML inserted at the beginning (default="").

anchor-cached-end = <HTML code>
    Anchors (links) that are cached are to have the specified HTML inserted at the end (default="").

anchor-requested-begin =<HTML code>
    Anchors (links) that have been requested are to have the specified HTML inserted at the beginning (default="").

anchor-requested-end = <HTML code>
    Anchors (links) that have been requested are to have the specified HTML inserted at the end (default="").

anchor-not-cached-begin = <HTML code>
    Anchors (links) that are not cached or requested are to have the specified HTML inserted at the beginning (default="").

anchor-not-cached-end = <HTML code>
    Anchors (links) that are not cached or requested are to have the specified HTML inserted at the end (default="").

disable-script = yes | no
    Removes all scripts and scripted events (default=no).

disable-blink = yes | no
    Removes the <blink> tag (default=no).

disable-animated-gif = yes | no
    Disables the animation of GIF files (default=no).

The LocalHost section contains a list of possible names or IP addresses that the host running wwwoffled may be known as.

hostname
    The server may be known as hostname so does not need to contact itself to get pages. The entries must match exactly. All of the entries here are also used as if they were in the LocalNet and AllowConnect sections. None of the entries here are fetched via a proxy.

The LocalNet section contains a list of host names or IP addresses that are not to be cached because they are on the local network.

hostname
    A server that matches hostname is on the local network and not to be cached. The matching uses wildcards as described in the WILDCARD section. A host can be excluded by appending a '!' to the start of the name, all possible aliases and IP addresses for the host are also required. All entries here are assumed to be reachable even when offline. All of the entries in the LocalHost section are also not cached as if they were here also. None of the entries here are fetched via a proxy.

The AllowedConnectHosts section contains a list of host names or IP addresses that are allowed to connect to the server.

hostname
    A server that matches hostname is allowed to connect to the server. The matching uses wildcards as described in the WILDCARD section. A host can be excluded by appending a '!' to the start of the name, all possible aliases and IP addresses for the host are also required. All of the entries in the LocalHost section are also allowed to connect.

The AllowedConnectUsers contains a list of the users that are allowed to connect to the server.

<username>:<password>
    The username and password of the users that are allowed to connect to the server. The username and password are both stored in plaintext format. This requires the use of browsers that handle the HTTP/1.1 standard.

The DontCache section contains a way of recognising URLs not to be cached. They will still be cached however if it is fetched non-interactively.

URL-SPECIFICATION
    Don't cache files that match URL-SPECIFICATION. See the URL-SPECIFICATION section for details of the URL-SPECIFICATION option. The URL-SPECIFICATION can be negated, see the URL-SPECIFICATION section.

The DontGet section contains a way of recognising URLs not to be got. This can be used to reject junk adverts for example.

URL-SPECIFICATION [ = <URL> ]
    Don't get files that match URL-SPECIFICATION. Replace them with the optional replacement URL. See the URL-SPECIFICATION section for details of the URL-SPECIFICATION option. The URL-SPECIFICATION can be negated, see the URL-SPECIFICATION section.

The replacement = <URL> option allows a default replacement URL to be specified that will be used to replace all URLs that match any of the URL-SPECIFICATIONS in this section that do not have a replacement specified.

The DontGetRecursive section contains a way of recognising URLs not to be got when getting recursively.

See the DontCache section for a description of the options in this section.

The DontRequestOffline section contains a way of recognising URLs not to requested by users when offline

See the DontCache section for a description of the options in this section.

The CensorHeader section contains a list of the header lines that are to be modified in the request sent from the browser to the server or the reply from the server to the browser.

header = <string> | none |
    The lines in the request that start with header , followed by a ':' are removed before being passed on if there is no string on the right hand side, else that string replaces the one in the header. This option does not allow you to add headers that were not present in the original header.

referer-self = yes | no
    Sets the Referer header to the same as the URL (default = no).

referer-self-dir = yes | no
    Sets the Referer header to the URL directory name (default = no). This option takes precedence over referer-self if both are set.

The FTPOptions section contains the information that is required to be able to do anonymous ftp.

anon-username = <string>
    Specifies the username to use to fetch files using ftp (default is "anonymous", "ftp" is another option).

anon-password = <string>
    Specifies the password to use to fetch files using ftp (default is determined from the user running wwwoffled and the hostname, this may not work reliably especially if you are behind a firewall).

auth-hostname = <host[:port]>
    auth-username = <string> auth-password = <string> Specifies a triplet of hostname, username and password that allow non-anonymous access to a specific server. (These options must come in groups of three.) The auth-hostname must match exactly, no wildcards are used.

The MIMETypes section is a list of the mime type to associate with files that are not fetched using HTTP. This is required by browsers, most browsers come with a list that can be used here.

default = <mime-type>/<subtype>
    The default MIME type to use for files that do not match any of the other rules.

.<file-ext> = <mime-type>/<subtype>
    The MIME type to use for files that match the file extension.

The Proxy section contains a list of the hosts that are to be served via specified proxy servers. If no proxy is required then use 'none' or leave the proxy name blank.

default = <hostname:[port]> | none |
    Specifies the default proxy that all requests are to use.

URL-SPECIFICATION = <hostname:[port]> | none |
    For URLs that match URL-SPECIFICATION use the specified proxy. See the URL-SPECIFICATION section for details of the URL-SPECIFICATION option.

auth-hostname = <host[:port]>
    auth-username = <string> auth-password = <string> Specifies a proxy server host that requires proxy authentication by username and password to use it. (These options must come in groups of three.) The auth-hostname must match exactly, no wildcards are used.

ssl = <hostname:[port]> | none |
    A proxy server that should be used for Secure Socket Layer (SSL) connections e.g. https (default = none).

None of the entries in the LocalHost or LocalNet section are fetched using a proxy.

The DontIndex section contains a way of recognising URLs not to be indexed.

outgoing = URL-SPECIFICATION
    Do not index any URLs that match URL-SPECIFICATION in the outgoing index.

latest = URL-SPECIFICATION
    Do not index any URLs that match URL-SPECIFICATION in the lasttime/prevtime/latest indexes.

monitor = URL-SPECIFICATION
    Do not index any URLs that match URL-SPECIFICATION in the monitor index.

host = URL-SPECIFICATION
    Do not index any URLs that match URL-SPECIFICATION in the host indexes.

URL-SPECIFICATION
    Do not index any URLs that match URL-SPECIFICATION in any of the indexes.

See the URL-SPECIFICATION section for details of the URL-SPECIFICATION option.

The Alias A list of aliases that are used to replace the server name and path with another server name and path. Also for servers known by two names.

URL-SPECIFICATION1 = URL-SPECIFICATION2
    When a request matching URL-SPECIFICATION1 is used the request is modified into a request for URL-SPECIFICATION2 , the two are also considered identical for the purposes of indexing, purging and recursive fetching. The URL-SPECIFICATION must not be a wildcard match and the URL arguments are ignored.

The Purge section controls how the cache is purged. The method to determine which pages to purge, the default age, the host specific maximum age of the pages in days, and a maximum allowed cache size. An age of zero means to always delete when a purge is done, a negative age means never purge. The maximum cache size and minimum free space include the files that are from hosts that are marked never to be purged but will not purge them.

use-mtime = yes | no
    The decision of which pages to purge can be made on last access time (atime) or last modification time (mtime) (default=no).

max-size = <size>
    The maximum size of the cache in MB after purging, excluding the hosts that are never to be purged, if this is zero then it does not apply (default=0).

min-free = <size>
    The minimum amount of free disk space in MB after purging, excluding the hosts that are never to be purged, if this is zero then it does not apply (default=0).

use-url = yes | no
    If true then use the URL to decide on the purge age, otherwise use the protocol and host only (default=no).

del-dontget = yes | no
    If true then delete the files from hosts that are in the DontGet section (default=no).

del-dontcache = yes | no

If true then delete the files from hosts that are in the DontCache section
    (default=no). default = <age> The age to purge hosts that are not otherwise specified here (default=28).

URL-SPECIFICATION = ...
    The age to purge hosts with URLs that match URL-SPECIFICATION this does not include the path and extension part which are ignored. See the URL-SPECIFICATION section for details of the URL-SPECIFICATION option.

WILDCARD
A wildcard match is one that uses the '*' character to represent any group of characters.

This is basically the same as the command line file matching expressions in DOS or the UNIX shell, except that the '*' can match the '/' character. A maximum of 2 '*' characters can be used in any wildcard.

For example

 *.gif matches foo.gif and bar.gif
 *.foo.com matches www.foo.com and ftp.foo.com
 /foo/* matches /foo/bar.html and /foo/bar/foobar.html

URL-SPECIFICATION
When specifying a host and protocol and pathname in many of the sections a URL-SPECIFICATION can be used, this is a way of recognising a URL.

For the purposes of this explanation a URL is considered to be made up of five parts.

proto
    The protocol that is used (e.g. 'http', 'ftp')

host
    The server hostname (e.g. 'www.gedanken.demon.co.uk').

port
    The port number on the host (e.g. default of 80 for HTTP).

path
    The pathname on the host (e.g. '/bar.html') or a directory name (e.g. '/foo/').

args
    Optional arguments with the URL used for CGI scripts etc. (e.g. 'search=foo').

For example the WWWOFFLE homepage: http://www.gedanken.demon.co.uk/wwwoffle/ The protocol is 'http', the host is 'www.gedanken.demon.co.uk', the port is the default (in this case 80), and the pathname is '/wwwoffle/'.

In general this is written as <proto>://<host>[:<port>]/<path>[?<args>]

Where [] indicates an optional feature, and <> indicate a user supplied name or number.

Some example URL-SPECIFICATION options are the following:

*://*
    Any protocol, Any host, Any port, Any path, Any args (This is that same as saying 'default').

*://*/<path>
    Any protocol, Any host, Any port, Named path, Any args

*://*/*.<ext>
    Any protocol, Any host, Any port, Named path, Any args

*://*/*?
    Any protocol, Any host, Any port, Any path, No args

*://<path>?*
    Any protocol, Any host, Any port, Named path, Any args

*://<host>
    Any protocol, Named host, Any port, Any path, Any args

<proto>://
    Named protocol, Any host, Any port, Any path, Any args

<proto>://<host>
    Named protocol, Named host, Any port, Any path, Any args

<proto>://<host>:
    Named protocol, Named host, Default port, Any path Any args

*://<host>:<port>
    Any protocol, Named host, Named port, Any path, Any args

The matching of the host, the path and the args use the wildcard matching that is described above.

In some sections that accept URL-SPECIFICATIONs they can be negated by appending the '!' character to the start. This will mean that the comparison of a URL with the URL-SPECIFICATION will return the logically opposite value to what would be returned without the '!'. If all of the URL-SPECIFICATIONs in a section are negated and '*://*/*' is added to the end then the sense of the whole section is negated.

EXAMPLE
 StartUp
 {
  http-port = 8080
  wwwoffle-port = 8081
  spool-dir = /var/spool/wwwoffle
  use-syslog = yes
  password =
 }
 Options
 {
  index-latest-days = 14
  add-info-refresh = no
  request-changed = 3600
 }
 FetchOptions
 {
  images = yes
  frames = yes
 }
 LocalHost
 {
  wwwoffle.foo.com
  localhost
  127.0.0.1
 }
 DontGet
 [
  wwwoffle.DontGet.conf
 ]
 LocalNet
 {
  *.foo.com
 }
 AllowedConnectHosts
 {
  *.foo.com
 }
 Proxy
 {
  http://foo.com/* = www-cache.foo.com:8080
 }
 Purge
 {
  default = 28
  max-size = 10
  http://*.bar.com/* = 7
 }

FILES
/var/spool/wwwoffle/wwwoffle.conf The wwwoffled(8) configuration file.

/var/spool/wwwoffle The WWWOFFLE spool directory.

SEE ALSO
wwwoffle(1), wwwoffled(8).

AUTHOR
Andrew M. Bishop 1996,1997,1998,1999 (amb@gedanken.demon.co.uk)