swish-e/pod/SWISH-CONFIG.pod

=head1 NAME

SWISH-CONFIG - Configuration File Directives

=head1 Swish-e CONFIGURATION FILE

What files Swish-e indexes and how they are indexed, and where the index
is written can be controlled by a configuration file.

The configuration file is a text file composed of comments, blank
lines, and B<configuration directives>.  The order of the directives
is not important.  Some directives may be used more than once in the
configuration file, while others can only be used once (e.g. additional
directives will overwrite preceding directives).  Case of the directive
is not important -- you may use upper, lower, or mixed case.

Comments are any line that begin with a "#".

    # This is a comment

Directives may take more than one parameter.  Enclose single parameters
that include whitespace in quotes (single or double).  Inside of quotes
the backslash escapes the next character.

    ReplaceRules append "foo bar"   <- define "foo bar" as a single parameter

If you need to include a quote character in the value either use a
backslash to escape it, or enclose it in quotes of the other type.

For example, under unix you can use quotes to include white space in a
single paramter.  Here, to protect against path names (%p) that might
have white space embedded use single quotes (this also protects against
shell expansion or metacharacters):

    FileFilter .foo foofilter "'%p'"  <- parameter passed through the shell in single quotes
    FileFilter .foo foofilter '"%p"'  <- windows uses double-quotes
    FileFilter .foo foofilter '\'%p\''<- silly example
    

Backslashes also have special meaning in regular expressions.

    FileFilterMatch pdftotext "'%p' -" /\.pdf$/

This says that the dot is a real dot (instead of matching any character).
If you place the regular expression in quotes then you must use
double-backslashes.

    FileFilterMatch pdftotext "'%p' -" "/\\.pdf$/"

Swish-e will convert the double backslash into a single backslash before
passing the parameter to the regular expression compiler.

Commented example configuration files are included in the F<conf>
directory of the Swish-e distribution.

Some command line arguments can override directives specified in the
configuration file.  Please see also the L<SWISH-RUN|SWISH-RUN> for
instructions on running Swish-e, and the L<SWISH-SEARCH|SWISH-SEARCH>
page for information and examples on how to search your index.

The configuration file is specified to Swish-e by the C<-c> switch.
For example,

    swish-e -c myconfig.conf

You may also split your directives up into different configuration files.
This allows you to have a master configuration file used for many
different indexes, and smaller configuration files for each separate
index.  You can specify the different configuration files when running
from the command line with the C<-c> switch (see L<SWISH-RUN|SWISH-RUN>),
or you may include other Configuration file with the B<IncludeConfigFile>
directive below.

Typically, in a configuration file the directives are grouped together in
some logical order -- that is, directives that control the source of the
documents would be grouped together first, and directives that control
how each document is filtered or its words index in another group of
directives. (The directives listed below are grouped in this order).

The configuration file directives are listed below in these groups:    

=over 4

=item *

L<Administrative Headers Directives|/"Administrative Headers Directives">
-- You may add administrative information to the header of the index file.

=item *

L<Document Source Directives|/"Document Source Directives"> -- Directives
for selecting the source documents and the location of the index file.

=item *

L<Document Contents Directives|/"Document Contents Directives"> --
Directives that control how a document content is indexed.

=item *

L<Directives for the File Access method only|/"Directives for the File
Access method only"> -- These directives are only applicable to the File
Access indexing method.

=item *

L<Directives for the HTTP Access Method Only|/"Directives for the HTTP
Access Method Only"> -- Likewise, these only apply to the HTTP Access
method.

=item *

L<Directives for the prog Access Method Only|/"Directives for the prog
Access Method Only"> -- These only apply to the prog Access method.

=item *

L<Document Filter Directives|/"Document Filter Directives"> -- This is
a special section that describes using document filters with Swish-e.

=back

=head2 Alphabetical Listing of Directives

=over 4

=item *

L<AbsoluteLinks|/"item_AbsoluteLinks"> [yes|NO]

=item *

L<BeginCharacters|/"item_BeginCharacters"> *string of characters*

=item *

L<BumpPositionCounterCharacters|/"item_BumpPositionCounterCharacters"> *string*

=item *

L<Buzzwords|/"item_Buzzwords"> [*list of buzzwords*|File: path]


=item *

L<ConvertHTMLEntities|/"item_ConvertHTMLEntities"> [YES|no]

=item *

L<DefaultContents|/"item_DefaultContents"> [TXT|HTML|XML|WML]

=item *

L<Delay|/"item_Delay"> *seconds*

=item *

L<DontBumpPositionOnEndTags|/"item_DontBumpPositionOnEndTags"> *list of names*

=item *

L<DontBumpPositionOnStartTags|/"item_DontBumpPositionOnStartTags"> *list of names*

=item *

L<EnableAltSearchSyntax|/"item_EnableAltSearchSyntax">  [yes|NO]

=item *

L<EndCharacter|/"item_EndCharacters"> *string of characters*

=item *

L<EquivalentServer|/"item_EquivalentServer"> *server alias*

=item *

L<ExtractPath|/"item_ExtractPath"> *metaname* [replace|remove|prepend|append|regex]

=item *

L<FileFilter|/"item_FileFilter"> *suffix* *program* [options]

=item *

L<FileFilterMatch|/"item_FileFilterMatch"> *program* *options* *regex* [*regex* ...]

=item *

L<FileInfoCompression|/"item_FileInfoCompression"> [yes|NO]

=item *

L<FileMatch|/"item_FileMatch"> [contains|is|regex] *regular expression*

=item *

L<FileRules|/"item_FileRules"> [contains|is|regex] *regular expression*

=item *

L<FuzzyIndexingMode|/"item_FuzzyIndexingMode"> [NONE|Stemming|Soundex|Metaphone|DoubleMetaphone]

=item *

L<FollowSymLinks|/"item_FollowSymLinks"> [yes|NO]

=item *

L<HTMLLinksMetaName|/"item_HTMLLinksMetaName"> *metaname*

=item *

L<IgnoreFirstChar|/"item_IgnoreFirstChar"> *string of characters*

=item *

L<IgnoreLastChar|/"item_IgnoreLastChar"> *string of characters*

=item *

L<IgnoreLimit|/"item_IgnoreLimit"> *integer integer*

=item *

L<IgnoreMetaTags|/"item_IgnoreMetaTags"> *list of names*

=item *

L<IgnoreNumberChars|/"item_IgnoreNumberChars"> *list of characters*

=item *

L<IgnoreTotalWordCountWhenRanking|/"item_IgnoreTotalWordCountWhenRanking"> [YES|no]

=item *

L<IgnoreWords|/"item_IgnoreWords"> [*list of stop words*|File: path]

=item *

L<ImageLinksMetaName|/"item_ImageLinksMetaName"> *metaname*

=item *

L<IncludeConfigFile|/"item_IncludeConfigFile">

=item *

L<IndexAdmin|/"item_IndexAdmin"> *text*

=item *

L<IndexAltTagMetaName|/"item_IndexAltTagMetaName"> *tagname*|as-text

=item *

L<IndexComments|/"item_IndexComments"> [YES|no]

=item *

L<IndexContents|/"item_IndexContents"> [TXT|HTML|XML|WML|TXT2|HTML2|XML2]  *file extensions*

=item *

L<IndexDescription|/"item_IndexDescription"> *text*

=item *

L<IndexDir|/"item_IndexDir"> [URL|directories or files]

=item *

L<IndexFile|/"item_IndexFile"> *path*

=item *

L<IndexName|/"item_IndexName"> *text*

=item *

L<IndexOnly|/"item_IndexOnly"> *list of file suffixes*

=item *

L<IndexPointer|/"item_IndexPointer"> *text*

=item *

L<IndexReport|/"item_IndexReport"> [0|1|2|3]

=item *

L<MaxDepth|/"item_MaxDepth"> *integer*

=item *

L<MaxWordLimit|/"item_MaxWordLimit"> *integer*

=item *

L<MetaNameAlias|/"item_MetaNameAlias"> *meta name* *list of aliases*

=item *

L<MetaNames|/"item_MetaNames"> *list of names*

=item *

L<MinWordLimit|/"item_MinWordLimit"> *integer*

=item *

L<NoContents|/"item_NoContents"> *list of file suffixes*

=item *

L<obeyRobotsNoIndex|/"item_obeyRobotsNoIndex"> [yes|NO]

=item *

L<ParserWarnLevel|/"item_ParserWarnLevel"> [0|1|2|3]

=item *

L<PreSortedIndex|/"item_PreSortedIndex"> *list of property names*

=item *

L<PropCompressionLevel|/"item_PropCompressionLevel"> [0-9]

=item *

L<PropertyNameAlias|/"item_PropertyNameAlias"> *property name* *list of aliases*

=item *

L<PropertyNames|/"item_PropertyNames"> *list of meta names*

=item *

L<PropertyNamesCompareCase|/"item_PropertyNamesCompareCase"> *list of meta names*

=item *

L<PropertyNamesIgnoreCase|/"item_PropertyNamesIgnoreCase"> *list of meta names*

=item *

L<PropertyNamesDate|/"item_PropertyNamesDate"> *list of meta names*

=item *

L<PropertyNamesNumeric|/"item_PropertyNamesNumeric"> *list of meta names*

=item *

L<PropertyNamesMaxLength|/"item_PropertyNamesMaxLength"> integer *list of meta names*

=item *

L<ReplaceRules|/"item_ReplaceRules"> [replace|remove|prepend|append|regex]

=item *

L<ResultExtFormatName|/"item_ResultExtFormatName">  name -x format string

=item *

L<SpiderDirectory|/"item_SpiderDirectory"> *path*

=item *

L<StoreDescription|/"item_StoreDescription"> [XML <tag>|HTML <meta>|TXT size]

=item *

L<SwishProgParameters|/"item_SwishProgParameters> *list of parameters*

=item *

L<SwishSearchDefaultRule|/"item_SwishSearchDefaultRule">   [<AND-WORD>|<or-word>]

=item *

L<SwishSearchOperators|/"item_SwishSearchOperators"> <and-word> <or-word> <not-word>

=item *

L<TmpDir|/"item_TmpDir"> *path*

=item *

L<TranslateCharacters|/"item_TranslateCharacters"> [*string1 string2*|:ascii7:]

=item *

L<TruncateDocSize|/"item_TruncateDocSize">
*number of characters*

=item * 

L<UndefinedMetaTags|/"item_UndefinedMetaTags"> [error|ignore|INDEX|auto]

=item *

L<UndefinedXMLAttributes|/"item_UndefinedXMLAttributes"> [DISABLE| error|ignore|index|auto]

=item *

L<UseStemming|/"item_UseStemming"> [yes|NO]

=item *

L<UseSoundex|/"item_UseSoundex"> [yes|NO]

=item *

L<UseWords|/"item_UseWords"> [*list of words*|File: path]

=item *

L<WordCharacters|/"item_WordCharacters"> *string of characters*

=item *

L<XMLClassAttributes|/"item_XMLClassAttributes"> *list of XML attribute names*

=back

=head2 Directives that Control Swish

These configuration directives control the general behavior of Swish-e.

=over 4

=item IncludeConfigFile *path to config file*

This directive can be used to include configuration directives located
in another file.

    IncludeConfigFile /usr/local/swish/conf/site_config.config

=item IndexReport [0|1|2|3]

This is how detailed you want reporting while indexing. You can specify
numbers 0 to 3.  0 is totally silent, 3 is the most verbose.   The default
is 1.

This may be overridden from the command line via the C<-v> switch (see
L<SWISH-RUN|SWISH-RUN>).

=item ParserWarnLevel [0|1|2|3]

Sets the error level when using the libxml2 parser for XML and HTML.
libxml2 will point out structural errors in your documents.

    0 = no report
    1 = fatal errors
    2 = errors
    3 = warnings

The exception to this is UTF-8 to Latin-1 coversion errors are reported at
level 1.  This is because words may be indexed incorrecty in these cases.

Note that unlike other errors generated by Swish-e, these errors are
sent to stderr.

=item IndexFile *path*

Index file specifies the location of the generated index file.  If not
specified, Swish-e will create the file F<index.swish-e> in the current
directory.

    IndexFile /usr/local/swish/site.index

=item obeyRobotsNoIndex [yes|NO]

When enabled, Swish-e will not index any HTML file that contains:

    <meta name="robots" content="noindex">

The default is to ignore these meta tags and index the document.
This tag is described at http://www.robotstxt.org/wc/exclusion.html.

Note: This feature is only available with the libxml2 HTML parser.

Also, if you are using the libxml2 parser (HTML2 and XML2) then you can use the the following
comments in your documents to prevent indexing:

       <!-- SwishCommand noindex -->
       <!-- SwishCommand index -->

and/or these may be used also:

       <!-- noindex -->
       <!-- index -->

For example, these are very helpful to prevent indexing of common headers, footers, and menus.


=back

B<NOTE>: This following items are currently not available.  These items
require Swish-e to parse the configuration file while searching.


=over 4

=item EnableAltSearchSyntax [yes|NO]

B<NOTE>: This following item is currently not available.

Enable alternate search syntax.  Allows the usage of a basic
"Altavista(c)", "Lycos(c)", etc. like search syntax.  This means a search
query can contain "+" and "-" as syntax parameter.

Example:

    swish-e -w "+word1 +word2 -word3  word4 word5"
    "+"  = following word has to be in all found documents
    "-"  = following word may not be in any document found
    " "  = following word will be searched in documents

=item SwishSearchOperators <and-word> <or-word> <not-word>

B<NOTE>: This following item is currently not available.

Using this config directive you can change the boolean search operators of
Swish-e, e.g. to adapt these to your language.
The default is:    AND  OR  NOT

Example (german):

    SwishSearchOperators   UND  ODER  NICHT

=item SwishSearchDefaultRule   [<AND-WORD>|<or-word>]

B<NOTE>: This following item is currently not available.

C<SwishSearchDefaultRule> defines the default Boolean operator to use
if none is specified between words or phrases.  The default is C<AND>.

The word you specify must match one of the available
C<SwishSearchOperators>.

Example:

    SwishSearchOperators   UND  ODER  NICHT
    # Make it act like a web search engine
    SwishSearchDefaultRule ODER

=item ResultExtFormatName name -x format string

B<NOTE>: This following item is currently not available.

The output of Swish-e can be defined by specifying a format string with
the C<-x> command line argument.  Using C<ResultExtFormatName> you can
assign a predefined format string to a name.

Examples:

    ResultExtFormatName  moreinfo   "%c|%r|%t|%p|<author>|<publishyear>\n"

Then when searching you can specify the the format string's name
   
    swish-e   ...  -x moreinfo  ...

See the C<-x> switch in L<SWISH-RUN|SWISH-RUN> for more information
about output formats.

=back


=head2 Administrative Headers Directives

Swish-e stores configuration information in the header of the index file.
This information can be retrieved while searching or by functions in
the Swish-e C library.  There are a number of fields available for your
own use.  None of these fields are required:

=over 4

=item IndexName *text*

=item IndexDescription *text*

=item IndexPointer *text*

=item IndexAdmin *text*

These variables specify information that goes into index files to help
users and administrators.  IndexName should be the name of your index,
like a book title.  IndexDescription is a short description of the index
or a URL pointing to a more full description.  IndexPointer should be
a pointer to the original information, most likely a URL.  IndexAdmin
should be the name of the index maintainer and can include name and email
information.  These values should not be more than 70 or so characters
and should be contained in quotes.  Note that the automatically generated
date in index files is in D/M/Y and 24-hour format.

Examples:

    IndexName "Linux Documentation"
    IndexDescription "This is an index of /usr/doc on our Linux machine." 
    IndexPointer http://localhost/swish/linux/index.html
    IndexAdmin webmaster


=back

=head2 Document Source Directives

These directives control I<what> documents are indexed and I<how>
they are accessed.  See also L<Directives for the File Access method
only|/"Directives for the File Access method only"> and L<Directives for
the HTTP Access Method Only|/"Directives for the HTTP Access Method Only">
for directives that are specific to those access methods.


=over 4

=item IndexDir [directories or files|URL|external program]

IndexDir defines the source of the documents for Swish-e.  Swish-e
currently supports three file access methods: B<File system>, B<HTTP>
(also called B<spidering>), and B<prog> for reading files from an
external program.

The C<-S> command line argument is used to select the file access method.

    swish-e -c swish.config -S fs    - file system
    swish-e -c swish.config -S http  - internal http spider
    swish-e -c swish.config -S prog  - external program of any type

For the B<fs> method of access B<IndexDir> is a space-separated
list of files and directories to index.  Use a forward slash as the path
separator in MS Windows.

For the B<http> method the B<IndexDir> setting is a list of space-separated
URLs.

For the B<prog> method the B<IndexDir> setting is a list of space-separated
programs to run (which generate documents for swish to index).

You may specify more than one B<IndexDir> directive.

Any sub-directories of any listed directory will also be indexed.

Note: While I<processing> directories, Swish-e will ignore any files
or directories that begin with a dot (".").  You may index files
or directories that begin with a dot by specifying their name with
C<IndexDir> or C<-i>.

Examples:

    # Index this directory an any subdirectories
    IndexDir /usr/local/home/http

    # Index the docs directory in current directory
    IndexDir ./docs

    # Index these files in the current directory
    IndexDir ./index.html ./page1.html ./page2.html
    # and index this directory, too
    IndexDir ../public_html

For the B<HTTP> method of access specify the URL's from which
you want the spidering to begin.

Example:

    IndexDir http://www.my-site.com/index.html
    IndexDir http://localhost/index.html

Obviously, using the B<HTTP> method to index is B<much> slower than
indexing local files.  Be well aware that some sites do not appreciate
spidering and may block your IP address.  You may wish to contact the
remote site before spidering their web site.  More information about
spidering can be found in L<Directives for the HTTP Access Method
Only|/"Directives for the HTTP Access Method Only"> below.

For the L<prog|SWISH-RUN/"item_prog"> method of access B<IndexDir>
specifies the path to the program(s) to execute.  The external program
must correctly format the documents being passed back to Swish-e.
Examples of external programs are provided in the F<prog-bin> directory.

    IndexDir ./myprogram.pl

See L<prog|SWISH-RUN/"item_prog"> for details.


Note: Not all directives work with all methods.

=item NoContents *list of file suffixes*

Files with these suffixes will B<not> have their contents indexed.

If the file's type is HTML (as set by C<IndexContents> or
C<DefaultContents>) then the file will be parsed for a HTML title and
that title will be indexed.  Note that you must set the file's type:
C<.html> and C<.htm> are NOT type HTML by default.

If a title is found, it will still be checked for C<FileRules title>,
and the file will be skipped if a match is found.  See C<FileRules>.

If the file's type is not HTML, or it is HTML and no title is found,
then the file's path will be indexed.  For example, you might wish to
search for image files by file name.

Example:

    NoContents .gif .xbm .au .mov .mpg .pdf .ps

Note: Using this directive will not cause files with those suffixes
to be indexed.  That is, if you use C<IndexOnly> to limit the types of
files that are indexed, then you must specify in C<IndexOnly> the same
suffixes listed in C<NoContents>.

A C<-S prog> program may set the C<No-Contents:> header (to anything)
to enable this feature for a specific document (althought it would be
smarter for the C<-S prog> program to simply only send the pathname or
title to be indexed.

=item ReplaceRules [replace|remove|prepend|append|regex]

ReplaceRules allows you to make changes to file pathnames before
they're indexed.  These changed file names or URLs will be returned in
search results.

For example, you may index your files locally (with the File system
indexing method), yet return a URL in search results.  This directive can
be used to map the file names to their respective URLs on your web server.

There are five operations you can specify: B<replace>, B<append>,
B<remove>, B<prepend>, and B<regex> They will parse the pathname in the
order you've typed these commands.

This directive uses C library regex.h regular expressions.

   replace "the string you want replaced" "what to change it to"
   remove "a string to remove"   
   prepend "a string to add before the result"
   append "a string to add after the result"
   regex  "/search string/replace string/options"

Remember, quotes are needed if an expression contains white space,
and backslashes have special meaning.

Regex is an Extended Regular Expression.  The first character found is 
the delimiter (but it's not smart enough to use matched chars such as [],
(), and {}).

The B<replace> string may use substitution variables:

    $0      the entire matched (sub)string
    $1-$9   returns patterns captured in "(" ")" pairs
    $`      the string before the matched pattern
    $'      the string after the matched pattern

The B<options> change the behavior of expression:

    i       ignore the case when matching
    g       repeat the substitution for the entire pattern

Examples:

    ReplaceRules replace testdir/ anotherdir/
    ReplaceRules replace [a-z_0-9]*_m.*\.html index.html

    ReplaceRules remove testdir/

    ReplaceRules prepend http://localhost/
    ReplaceRules append .html

    ReplaceRules regex  !^/web/(.+)/!http://$1.domain.com/!
    replaces a file path:
        /web/search/foo/index.html
    with
        http://search.domain.com/foo/index.html

    ReplaceRules regex  #^#http://localhost/www#
    ReplaceRules prepend http://localhost/www  (same thing)

    # Remove all extensions from C source files
    ReplaceRules remove .c     # ERROR! That "." is *any char*
    ReplaceRules remove \.c    # much better...

    ReplaceRules remove "\\.c" # if in quotes you need double-backslash!  
    ReplaceRules remove "\.c"  # ERROR! "\." -> "." and is *any char*
    

=item IndexContents [TXT|HTML|XML|WML|TXT2|HTML2|XML2]  *file extensions*

The C<IndexContents> directive assigns one of Swish-e's document parsers
to a document, based on the its extension.  Swish-e currently knows how
to parse TXT, HTML, and XML documents.

The XML2, HTML2, and TXT2 parsers are currently only available when
Swish-e is configured to use libxml2.

Documents that are not assigned a parser with C<IndexContents> will, by
default, use the HTML parser.  The C<DefaultContents> directive may be
used to assign a parser to documents that do not match a file extension
defined with the C<IndexContents> directive.

Example:

    IndexContents HTML .htm .html .shtml
    IndexContents TXT  .txt .log .text
    IndexContents XML  .xml

HTML is the default type for all files, unless otherwise specified
(and this default can be changed by the B<DefaultContents> directive.
Swish-e parses titles from HTML files, if available, and keeps track
of the context of the text for context searching (see C<-t> in
L<SWISH-RUN|SWISH-RUN>).  HTML and XML files use different tag formats
for B<MetaNames> and B<PropertyNames>.

If using filters to convert documents you should include those extensions,
too.  For example, if using a filter to conver .pdf to .html, you need
to tell Swish-e that .pdf should be indexed by the internal HTML parser:

    FileFilter  .pdf   pdf2html
    IndexContent  HTML  .pdf

See also L<Document Filter Directives|/"Document Filter Directives">.

B<Note:> Some of this may be changed in the future to use content-types
instead of file extensions.  See L<SWISH-3.0|SWISH-3.0>

=item DefaultContents [TXT|HTML|XML|WML|TXT2|HTML2|XML2]

This sets the default parser for documents that are not specified in
B<IndexContents>. If not specified the default is HTML.

The XML2, HTML2, and TXT2 parsers are currently only available when
Swish-e is configured to use libxml2.


Example:

    DefaultContents HTML

The C<DefaultContents> directive I<should> be used when spidering,
as HTML files may be returned without a file extension (such as when
requesting a directory and the default index.html is returned).


=item FileInfoCompression [yes|NO]

** This directive is currently not supported **

Setting B<FileInfoCompression> to C<yes> will compress the index file to
save disk space.  This may result in longer indexing times.  The default
is C<no>.

Also see the C<-e> switch in L<SWISH-RUN|SWISH-RUN> for saving RAM
during indexing.


=back

=head2 Document Contents Directives

These directives control what information is extracted from your source
documents, and how that information is made available during searching.

=over 4

=item ConvertHTMLEntities [YES|no]

ASCII I<entities> can be converted automatically while indexing documents
of type HTML.  For performance reasons you may wish to set this to C<no>
if your documents do not contain HTML entities.  The default is C<yes>.

If C<ConvertHTMLEntities> is set C<no> the entities will be indexed
without conversion.

B<NOTE:> Entities within XML files and files parsed with libxml2 are
converted regardless of this setting.

=item MetaNames *list of names*

META names are a way to define "fields" in your XML and HTML documents.
You can use the META names in your queries to limit the search to just
the words contained in that META name of your document.  For example,
you might have a META tagged field in your documents called C<subjects>
and then you can search your documents for the word "foo" but only return
documents where "foo" is within the C<subjects> META tag.

    swish-e -w subjects=foo

(See also the C<-t> switch in L<SWISH-RUN|SWISH-RUN> for information
about I<context> searching in HTML documents.)

The B<MetaNames> directive is a space separated list.  For example:

    MetaNames meta1 meta2 keywords subjects

You may also use L<UndefinedMetaTags|/"item_UndefinedMetaTags"> to specify
automatic extraction of meta names from your HTML and XML documents,
and also to ignore indexing content of meta tags.

META tags can have two formats in your B<HTML> source documents:

    <META NAME="meta1" CONTENT="some content">

and (if using the HTML2/libxml2 parser)

    <meta1>
        some content
    </meta1>

But this second version is invalid HTML, and will generate a warning if
ParserWarningLevel is set (libxml2 only).

And in B<XML> documents, use the format:

    <meta1>
        Some Content
    </meta1>

Then you can limit your search to just META B<meta1> like this:

    swish-e -w 'meta1=(apples or oranges)'

You may nest the XML and the start/end tag versions:

    <keywords>
        <tag1>
            some content
        </tag1>
        <tag2>
            some other content
        </tag2>
    <keywords>

Then you can search in both tag2 and tag2 with:

    swish-e -w 'keywords=(query words)'

Swish-e indexes all text as some metaname.  The default is
C<swishdefault>, so these two queries are the same:

    swish-e -w foo
    swish-e -w swishdefault=foo

When indexing HTML Swish-e indexes the HTML title as default text, so
when searching Swish-e will find matches in both the HTML body and the
HTML title.  Swish also, by default, indexes content of meta tags.  So:

    swish-e -w foo

will find "foo" in the body, the title, or any meta tags.

Currently, there's no way to prevent Swish-e from indexing
the title contents along with the body contents, but see
L<UndefinedMetaTags|/"item_UndefinedMetaTags"> for how to control the
indexing of meta tags.

If you would like to search just the title text, you may use:

    MetaNames swishtitle

This will index the title text separately under the built-in swish
internal meta name "swishtitle".  You may then search like

    swish-e -w foo  -- search for "foo" in title, body (and undefined meta tags)
    swish-e -w swishtitle=foo -- search for "foo" in title only

In addition to swishtitle, you can limit searches to documents' path with:

   MetaNames swishdocpath

Then to search for "foo" but also limit searches to documents that include
"manual" or "tutorial" in thier path:

   swish-e -w foo swishdocpath=(manual or tutorial)

See also L<ExtractPath|/"item_ExtractPath">.


=item MetaNameAlias *meta name* *list of aliases*

MetaNameAlias assigns aliases for a meta name.  For example, if your
documents contain meta tags "description", "summary", and "overview"
that all give a summary of your documents you could do this:

    MetaNames summary
    MetaNameAlias summary description overview

Then all three tags will get indexed as meta tag "summary".  You can
then search all the fields as:

    -w summary=foo

The Alias work at search time, too.  So these will also limit the searh
to the "summary" meta name.

    -w description=foo
    -w overview=foo

=item MetaNamesRank integer *list of meta names*

    * Not implemented yet *

You can assign a bias to metanames that will affect how ranking is
calculated.  The range of values is from -10 to +10, with zero being
no bias.

    MetaNamesRank 4 subject
    MetaNamesRank 3 swishdefault
    MetaNamesRank 2 author publisher
    MetaNamesRank -5 wrongwords

This feature is not implemented yet

=item HTMLLinksMetaName *metaname*

Allows indexing of HTML links.  Normally, HTML links (href tags) are
not indexed by Swish-e.  This directive defines a metaname, and links
will be indexed under this meta name.

Example:

    HTMLLinksMetaName links

Now, to limit searches to files with a link to "home.html" do this:

    -w links='"home.html"'

The double quotes force a phrase search.    

To make Swish-e index links as normal text, you may use:

    HTMLLinksMetaName swishdefault

This feature is only available with the libxml2 HTML parser.    

=item ImageLinksMetaName *metaname*

Allows indexing of image links under a metaname.  Normally, image URLs
are not indexed.

Example:

    ImagesLinksMetaName images

Now, if you would like to find pages that include a nice image of a beach:

    -w images='beach'

To make Swish-e index links as normal text, you may use:

    ImageLinksMetaName swishdefault

This feature is only available with the libxml2 HTML parser.


=item IndexAltTagMetaName *tagname*|as-text

Allows indexing of images <IMG> ALT tag text.  Specify either a tag name which will be
used as a metaname, or the special text "as-text" which says to index the ALT text as
if it were plain text at the current location.

For example, by specifying a tag name:

   IndexAltTagMetaName bar

would make this markup:   

    <foo>
        <img src="/someimage.png" alt="Alt text here">
    </foo>

appear like

    <foo>
        <bar>Alt text here</bar>
    </foo>

Then the normal rules (C<MetaNames> and C<PropertyNames>) apply to how that text is indexed.

If you use the special tag "as-text" then
      
    <foo>
        <img src="/someimage.png" alt="Alt text here">
    </foo>

simply becomes

    <foo>
        Alt text here
    </foo>

This feature is only available when using the libxml2 parser (HTML2 and XML2).    


=item AbsoluteLinks [yes|NO]

If this is set true then Swish-e will attempt to convert relative URIs
extracted from HTML documents for use with C<HTMLLinksMetaName> and
C<ImageLinksMetaName> into absolute URIs.  Swish-e will use any <BASE>
tag found in the document, otherwise it will use the file's pathname.
The pathname used will be the pathname *after* C<ReplaceRules> has been
applied to the document's pathname.

For example, say you wish to index image links under the metaname
"images".

    ImageLinksMetaName images

If an image is located in http://localhost/vacations/france/index.html
and C<AbsoluteLinks> is set to no, then a image within that document:

     <img src="beach.jpeg">

will only index "beach.jpeg".

But, if you want more deatil when searching, you can enable
C<AbsoluteLinks> and Swish-e will index
"http://localhost/vacations/france/beach.jpeg".  You can then look for
images of beaches, but only in France:

    -w images=(beach and france)

This also means you can search for any images within France:

    -w images=(france)

This feature is only available with the libxml2 HTML parser.    

=item UndefinedMetaTags [error|ignore|INDEX|auto]

This directive defines the behavior of Swish-e during indexing when a
meta name is found but is B<not> listed in B<MetaNames>.  There are
four choices:


=over 2

=item error

If a meta name is found that is not listed in B<MetaNames>
then indexing will be halted and an error reported.

=item ignore

The contents of the meta tag are ignored and B<not> indexed
unless a metaname has been defined with the C<MetaNames> directive.

=item index

The contents of the meta tag are indexed, but placed in the
main index unless there's an enclosing metatag already in force. This
is the default.

=item auto

This method create meta tags automatically for HTML meta names
and XML elements.  Using this is the same as specifying all the meta
names explicitly in a B<MetaNames> dirictive.

=back

=item UndefinedXMLAttributes [DISABLE|error|index|auto]

This is similar to C<UndefinedMetaTags>, but only applies to XML documents (parsed with libxml2).
This allows indexing of attribute content, and provides a way to index the content under a
metaname.  For example, C<UndefinedXMLAttributes> can make

    <person age="23">
          John Doe
    </person>

look like the following to swish:

    <person>
        <person.age>
            23
        </person.age>
        John Doe
    </person>

What happens to the text "23" will depend on the setting of C<UndefinedXMLAttributes>:    

=over 2

=item disable

XML attributes are not parsed and not indexed.  This is the default.

=item error

If the concatenated meta name (e.g. person.age) is not listed in
B<MetaNames> then indexing will be halted and an error reported.

=item ignore

The contents of the meta tag are ignored and B<not> indexed unless a
metaname has been defined with the C<MetaNames> directive.

=item index

The contents of the meta tag are indexed, but placed in the main index
unless there's an enclosing metatag already in force.

=item auto

This method will create meta tags from the combined element and attributes
(and XML Class name) This options should be used with caution as it can
generate a lot of metaname entries.

See also the example below C<XMLClassAttribues>.


=back

=item XMLClassAttributes *list of XML attribute names*

Combines an XML class name with the element name to make up a metaname.
For example:

    XMLClassAttributes class

    <person class="first">
        John
    </person>
    <person class="last">
        Doe
    </person>

Will appear to Swish-e as:

    <person>
        <person.first>
        John
        </person.first>
    </person>
    <person>
        <person.last>
        Doe
        </person.last>
    </person>

How the data is indexed depends on C<MetaNames> and C<UndefinedMetaTags>.

Here's an example using the following configuation which combines the
two directives C<XMLClassAttributes> and C<UndefinedXMLAttributes>.

    XMLClassAttributes class
    UndefinedMetaTags auto
    UndefinedXMLAttributes auto
    IndexContents XML2 .xml
    
The source XML file looks like:    

    <xml>
    <person class="student" phone="555-1212" age="102">
       John
    </person>
    <person greeting="howdy">Bill</person>
    </xml>

Swish-e parses as:

    ./swish-e -c 2 -i 1.xml -T parsed_tags  parsed_text  -v 0
    Indexing Data Source: "File-System"
    
    <xml> (MetaName)
    
        <person> (MetaName)
            <person.student> (MetaName)
                <person.student.phone> (MetaName)
                    555-1212
                </person.student.phone> 
                <person.student.age> (MetaName)
                    102
                </person.student.age> 
                John
        </person> 
    
        <person> (MetaName)
            <person.greeting> (MetaName)
                howdy
            </person.greeting> 
            Bill
        </person> 
    
    </xml> 
    Indexing done!

One thing to note is that the first <person> block finds a class name
"student" so all metanames that are created from attributes use the
combined name "person.student".  The second <person> block doesn't contain
a "class" so, the attribute name is combinded directly with the element
name (e.g. "person.greeting").

=item ExtractPath *metaname* [replace|remove|prepend|append|regex]

This directive can be used to index extracted parts of a document's path.
A common use would to to limit searches to specific areas of your
file tree.

The extracted string will be indexed under the specified meta name.

See C<ReplaceRules> for a description of the various pattern replacement
methods, but you will use the I<regex> method.

For example, say your file system (or web tree) was organized into departments:

    /web/sales/foo...
    /web/parts/foo...
    /web/accounting/foo...

And you wanted a way to limit searches to just documents under "sales".

    ExtractPath department regex !^/web/([^/]+)/.*$!$1!

Which says, extract out the department name (as substring $1) and index
it as meta name C<department>.  Then to limit a search to the sales
department:

    swish-e -w foo AND department=sales

Note that the C<regex> method uses a substitution pattern, so to index
only a sub-string match the I<entire> document path in the regular
expression, as shown above.

See the C<ExtractPathDefault> option for a way to set a value if not
patterns match.

Although unlikely, you may use more than one C<ExtractPath> direcive.
More than one directive of the I<same> meta name will operate successively
(in order listed in the configuration file) on the path.  This allows
you to use regular expressions on the results of the previous pattern
substitution (as if piping the output from one expression to the patter
of the next).

    ExtractPath foo regex !^(...).+$!$1!
    ExtractPath foo regex !^.+(.)$!$1!

So, the third letter is indexed as meta name "foo" if both patterns match.    

    ExtractPath foo regex !^X(...).+$!$1!
    ExtractPath foo regex !^.+(.)$!$1!

Now (not the "X"), if the first pattern doesn't match, the last character
of the path name is indexed.  You must be clear on this behavior if you
are using more than one C<ExtractPath> directive with the same metaname.

The document path operated on is the real path swish used to access
the document.  That is, the C<ReplaceRules> directive has no effect on
the path used with C<ExtractPath>.

The full path is used for each meta name if more than one C<ExtractPath>
directive is used.  That is, changes to the path used in C<ExtractPath
foo> do not affect the path used by C<ExtractPath bar>.

=item ExtractPathDefault *metaname* default_value

This can be used with C<ExtractPath> to set a default string to index
under the given metaname if none of the C<ExtractPath> patterns match.

For example, say your want to index each document with a metaname
"department" based on the following path examples:

    /web/sales/foo...
    /web/parts/foo...
    /web/accounting/foo...

But you are also indexing documents that do not follow that pattern and you want to search those
seperately, too.

    ExtractPath department regex !^/web/([^/]+)/.*$!$1!
    ExtractPathDefault department other

Now, you may search like this:

    -w foo department=(sales)      - limit searches to the sales documents
    -w foo department=(parts)      - limit searches to the parts documents
    -w foo department=(accounting) - limit searches to the accounting documents
    -w foo department=(other)      - everything but sales, parts, and accounting.

This basically is a shortcut for:

    -w foo not department=(sales or parts or accounting)

but you don't need to keep track of what was extracted.    

=item PropertyNames *list of meta names*

=item PropertyNamesCompareCase *list of meta names*

=item PropertyNamesIgnoreCase *list of meta names*

Swish-e allows you to specify certain META tags that can be used as
B<document properties>.  The contents of any META tag that has been
identified as a document property can be returned as part of the search
results along with the rank, file name, title, and document size (see
the C<-p> and C<-x> switches in L<SWISH-RUN|SWISH-RUN>).

Properties are useful for returning additional data from documents in
search results -- this saves the effort of reading and parsing the source
files while reading Swish-e search results, and is especially useful
when the source documents are no longer available or slow to access
(e.g. over http).

Another feature of properties is that Swish-e can use the PropertyNames
for sorting the search results (see the C<-s> switch).

    PropertyNames author subjects

Two variations are available.  C<PropertyNamesCompareCase> and
C<PropertyNamesIgnoreCase>.  These tell Swish-e to either ignore or
compare case when sorting results.  The default for C<PropertyNames>
is to ignore the case.

    PropertyNamesIgnoreCase subject
    PropertyNamesCompareCase keyword

The defaults for "internal" properties are:

    swishtitle          --  ignore the case
    swishdocpath        --  compare case
    swishdescription    --  compare case

These can be overridden with C<PropertyNamesCompareCase> and
C<PropertyNamesIgnoreCase>.

    PropertyNamesCompareCase swishtitle    

Use of PropertyNames will increase the size of your index files,
sometimes significantly.  Properties will be compressed if Swish-e is
compiled with zlib as described in the L<INSTALL|INSTALL> manual page.

If Swish-e finds more than one property of the same name in a document
the property's contents will be concatinated for strings, and a warning
issues for numeric (or date) properties.


=item PropertyNamesNumeric

This directive is similar to C<PropertyNames>, but it flags the property
as being a string of digits (integer value) that will be stored as binary data instead
of a string.  This allows sorting with C<-s> and limiting with C<-L>
to sort and limit the property correctly.

Swish-e uses C<strtoul(3)> to convert the string into an unsigned long
integer.  Therefore, only positive integers can be stored.

Future versions of Swish-e may be able to store different property types
(such as negative integers and real numbers).  This directive may change
in future releases of Swish.

=item PropertyNamesDate

This directive is exactly like C<PropertyNamesNumeric>, but it also
flags the number as a machine timestamp (seconds since Epoch), and
will print a formatted date when returning this property.  See C<-x>
in L<SWISH-RUN|SWISH-RUN>.

Swish-e will not parse dates when indexing; you must use a timestamp.

=item PropertyNameAlias  *property name* *list of aliases*

This allows aliases for a property name.  For example, if you are indexing
HTML files, plus XML files that are written in English, German, and
Spanish and thus use the tags "title", "titel", and "t�tulo" you can use:

    PropertyNameAlias swishtitle title titel t�tulo titulo

Note that "swishtitle" is the built-in property used to store the title of
a document, and therefore you do not need to specify it as a PropertyName
before use.

=item PropertyNamesMaxLength  integer *list of meta names*

This option will set the max length of the text stored in a property.
You must specify a number between 0 and the max integer size on your
platform, and a list of properties.  The properties specified must not
be aliases.

If any of the property names do not exist they will be created (e.g. you
do not need to define the property with PropertyNames first).

In general, this feature will only be useful when parsing HTML or XML
with the libxml2 parser.

For example:

    PropertyNamesMaxLength 1000 swishdescription
    PropertyNameAlias swishdescription body

Is somewhat like

    StoreDescription HTML <body> 1000
    StoreDescription XML <body> 1000
    StoreDescription HTML2 <body> 1000
    StoreDescription XML2 <body> 1000

but StoreDescription allows setting the tag for each parser type.

    PropertyNamesMaxLength 1000 headings
    PropertyNameAlias headings h1 h2 h3 h4

collects all the heading text into a single property called "headings", not
to exceed 1000 characters.


=item PreSortedIndex *list of property names*

By default Swish-e generates presorted tables while indexing for each
property name.  This allows faster sorting when generating results.
On large document collections this presorting may add to the indexing
time, and also adds to the total size of the index.  This directive can
be used to customize exactly which properties will be presorted.

If C<PreSortedIndex> it is I<not> present in the config file (default
action), all the properties will be presorted at indexing time.  If it
is present without any parameter, no properties will be presorted.
Otherwise, only the property names specified will be presorted.

For example, if you only wish to sort results by a property called
C<title>:

    PropertyNames title age time
    PreSortedIndex  title


=item StoreDescription [XML <tag> size|HTML <meta> size|TXT size]

B<StoreDescription> allows you to store a document description in the
index file, and this description can be returned in your search results
when the C<-x> switch is used to include the I<swishdescription> for
extended results.

For text documents you specify the type C<TXT> and the number of I<characters> to capture.

    StoreDescription TXT 20

The above stores only the first twenty characters from the text file in the Swish-e index
file.

For HTML, and XML file types, specify the the tag to use for the
description, and optionally the number of characters to capture.  If not
specified will capture the entire contents of the tag.

    StoreDescription HTML <body> 20000
    StoreDescription XML  <desc> 40

Note that documents must be assigned a document type with C<IndexContents>
or C<DefaultContents> to use this feature.

Swish-e will compress the descriptions (or any other large property)
if compiled to use zlib (see L<INSTALL|INSTALL>).  This is recommended when using
StoreDescription and a large number of documents.  Compression of 30% to 50% is
not uncomon with HTML files.

=item PropCompressionLevel [0-9]

This directive sets the compression level used when storing properties
to disk.  A setting of zero is no compression, and a setting of nine is
the most compression.

The default depends on the default setting compiled with zlib, but is
typicaly six.

This option is useful when using C<StoreDescription> to store a large
amount text in properties (or if using C<PropertyNames> with large
property sizes).

Properties must be over a value defined in F<config.h> (100 is the
default) before compression will be attempted.  Swish-e will never store
the results of the compression if the compressed data is larger than
the original data.

This option is only available when Swish-e is compiled with zlib support.


=item TruncateDocSize *number of characters*

TruncateDocSize limits the size of a document while indexing documents
and/or using filters.  This config directive truncates the numbers of
read bytes of a document to the specified size.  This means: if a document
is larger, read only the specified numbers of bytes of the document.

Example:

    TruncateDocSize    10000000

The default is zero, which means read all data.


Warning: If you use TruncateDocSize, use it with care!  TruncateDocSize
is a safty belt only, to limit e.g.  filteroutput, when accessing
databases, or to limit "runnaway" filters.  Truncating doc input may
destroy document structures for Swish-e (e.g.  swish may miss closing
tags for XML or HTML documents).

TruncateDocSize does not currently work with the C<prog> input source
method.

=item FuzzyIndexingMode NONE|Stemming|Soundex|Metaphone|DoubleMetaphone

Selects the type of index to create.  Only one type of index may be created.

It's a good idea to create both a normal index and a fuzzy index and
allow your search interface select which index to use.  Many people find the
fuzzy searches to be too fuzzy.

The available fuzzy indexing options are:

=over 4

=item None

Words are stored in the index without any conversion.  This is the default.

=item Stemming

Words are converted using the Porter stemming algorithm.

From: http://www.tartarus.org/~martin/PorterStemmer/

    The Porter stemming algorithm (or �Porter stemmer�) is a
    process for removing the commoner morphological and inflexional
    endings from words in English. Its main use is as part of a
    term normalisation process that is usually done when setting up
    Information Retrieval systems.


This will help a search for "running" to also find "run" and "runs", for example.

The stemming function does not convert words to their root, rather
programmatically removes endings on words in an attempt to make similar
words with different endings stem to the same string of characters.
It's not a perfect system, and searches on stemmed indexes often return
curious results.  For example, two entirely different words may stem to
the same word.

Stemming also can be confusing when used with a wildcard (truncation).
For example, you might expect to find the word "running" by searching for
"runn*".  But this fails when using a stemmed index, as "running" stems to
"run", yet searching for "runn*" looks for words that start with "runn".

=item Soundex

Soundex was developed in the 1880s so records for people with similar
sounding names could be found more readily.  Soundex is a coded surname
based on the way a surname sounds rather than spelling.  Surnames that
sound similar, like Smith and Smyth, are filed together under the same
Soundex code.  This is mostly useful for US English.

Soundex should not be used to search for sound-alike words.  Metaphone
would be more appropriate for generic sound matching of words.  Soundex
should only be used where you need to search multiple documents for
proper names which sound similar.  This is primarily used for indexing
genealogical records.  This may be useful for indexing other collections
of data consisting mostly of names.  Many common name variations are
matched by Soundex.  The only notable exception is the first letter of
the name.  The first letter is not matched for sound.

=item Metaphone and DoubleMetaphone

Words are transformed into a short series of letters representing the sound of the word (in English).
Metaphone algorithms are often used for looking up mis-spelled words in dictionary programs.

From: http://aspell.sourceforge.net/metaphone/

    Lawrence Philips' Metaphone Algorithm is an algorithm which returns
    the rough approximation of how an English word sounds.

The C<DoubleMetaphone> mode will sometimes generate two different metaphones for the same word.
This is supposed to be useful when a word may be pronounced more than one way.

A metaphone index should give results somewhere in between Soundex and Stemming.    

=back

=item UseStemming [yes|NO]

Put yes to apply word stemming algorithm during indexing, else no.

    UseStemming no
    UseStemming yes

When UseStemming is set to C<yes> every word is stemmed before placing
it in to the index.

This option is depreciated.  It has been superceded by C<FuzzyIndexingMode>.

=item UseSoundex [yes|NO]

When UseSoundex is set to C<yes> every word is converted to a Soundex
code before placing it in to the index.

This option is depreciated.  It has been superceded by C<FuzzyIndexingMode>.

=item IgnoreTotalWordCountWhenRanking [YES|no]

Put yes to ignore the total number of words in the file when calculating
ranking. Often better with merges and small files. Default is yes.

    IgnoreTotalWordCountWhenRanking no

The default was changed from no to yes in version 2.2.

=item MinWordLimit *integer*

Set the minimum length of an word. Shorter words will not be indexed.
The default is 1 (as defined in F<src/config.h>).

    MinWordLimit 5

=item MaxWordLimit *integer*

Set the maximum length of an indexable word. Every longer word will not
be indexed.  The Default is 40 (as defined in F<src/config.h>).

=item WordCharacters *string of characters*

=item IgnoreFirstChar *string of characters*

=item IgnoreLastChar *string of characters*

=item BeginCharacters *string of characters*

=item EndCharacter *string of characters*


These settings define what a word consists of to the Swish-e indexing engine.
Compiled in defaults are in F<src/config.h>.

When indexing Swish-e uses B<WordCharacters> to split up the document
into words.  Words are defined by any string of non-blank characters
that contain only the characters listed in WordCharacters.  If a string
of characters includes a character that is not in WordCharacters then
the word will be spit into two or more separate words.

For example:

    WordCharacters abde

Would turn "abcde" into two words "ab" and "de".

Next, of these words, any characters defined in B<IgnoreFirstChar> are
stripped off the start of the word, and B<IgnoreLastChar> characters
are stripped off the end of the word.  This allows, for example,
periods within a word (www.slashdot.com), but not at the end of
a word.  Characters in IgnoreFirstChar and IgnoreLastChar must be in
WordCharacters.

Finally, the resulting words MUST begin with one of the characters
listed in B<BeginCharacters> and end with one of the characters listed in
B<EndCharacters>.  BeginCharacters and EndCharacters must be a subset of
the characters in WordCharacters.  Often, WordCharacters, BeginCharacters
and EndCharacters will all be the same.

Note that the same process applies to the query while searching.

Getting these settings correct will take careful consideration and
practice.  It's helpful to create an index of a single test file, and
then look at the words that are placed in the index (see the C<-v 4>,
C<-D> and C<-k> searching switches).

Currently there is only support for eight-bit characters.

Example:

    WordCharacters  .abcdefghijklmnopqrstuvwxyz
    BeginCharacters abcdefghijklmnopqrstuvwxyz
    EndCharacters   abcdefghijklmnopqrstuvwxyz
    IgnoreFirstChar .
    IgnoreLastChar  .

So the string

    Please visit http://www.example.com/path/to/file.html.

will be indexed as the following words:

    please
    visit
    http
    www.example.com
    path
    to
    file.html

Which means that you can search for C<www.example.com> as a single word,
but searching for just C<example> will not find the document.

Note: when indexing HTML documents HTML entities are converted to their
character equivalents before being processed with these directives.
This is a change from previous versions of Swish-e where you were
required to include the characters C<0123456789&#;> to index entities.
See also L<ConvertHTMLEntities|/"item_ConvertHTMLEntities">

=item Buzzwords [*list of buzzwords*|File: path]

The Buzzwords option allows you to specify words that will be indexed
regardless of WordCharacters, BeginCharacters, EndCharacters, stemming,
soundex and many of the other checks do on words while indexing.

Buzzwords are case insensitive.

Buzzwords should be separated by spaces and may span multiple directives.
If the special format C<File:filename> is used then the Buzzwords will
be read from an external file during indexing.

Examples:

    Buzzwords C++ TCP/IP

    Buzzwords File: ./buzzwords.lst

If a Buzzword contains search operator characters they must be backslashed
when searching.  For example:

    Buzzwords C++ TCP/IP web=http

    ./swish-e -w 'web\=http'

Buzzwords are found by splitting the text on whitespace, removing
C<IgnoreFirstChar> and C<IgnoreLastChar> characters from the word,
and then comparing with the list of C<Buzzwords>.  Therefore, if
adding C<Buzzwords> to an index you will probably want to define
C<IgnoreFirstChar> and C<IgnoreLastChar> settings.

Note: Buzzwords specific settings for C<IgnoreFirstChar> and
C<IgnoreLastChar> may be used in the future.


=item IgnoreWords [*list of stop words*|File: path]

The IgnoreWords option allows you to specify words to ignore, called
I<stopwords>.  The default is to not use any stopwords.

Words should be separated by spaces and may span multiple directives.
If the special format C<File:filename> is used then the stop words will
be read from an external file during indexing.

In previous versions of Swish-e you could use the directive

    IgnoreWords swishdefault - obsolete!

to include a default list of compiled in stopwords.  This keyword is no
longer supported.

Examples:

    IgnoreWords www http a an the of and or

    IgnoreWords File: ./stopwords.de

=item UseWords [*list of words*|File: path]

UseWords defines the words that Swish-e will index.  B<Only> the words
listed will be indexed.

You can specify a list of words following the directive (you may specify
more than one C<UseWords> directive in a config file), and/or use the
C<File:> form to specify a path to a file containing the words:

    UseWords perl python pascal fortran basic cobal php
    UseWords File: /path/to/my/wordlist

Please drop the Swish-e list a note if you actually use this feature.
It may be removed from future versions.

=item IgnoreLimit *integer integer*

This automatically omits words that appear too often in the files (these
words are called stopwords). Specify a whole percentage and a number,
such as "80 256". This omits words that occur in over 80% of the files
and appear in over 256 files. Comment out to turn off auto-stopwording.

    IgnoreLimit 50 1000

Swish-e must do extra processing to adjust the entire index when this
feature is used.  It is recommended that instead of using this feature
that you decided what words are stopwords and add them to B<IngoreWords>
in your configuration file.  To do this, use IgnoreLimit one time and
note the stop words that are found while indexing.  Add this list to
IgnoreWords, and then remove IgnoreLimit from the configuration file.

=item IgnoreMetaTags *list of names*

C<IgnoreMetaTags> defines a list of metantags to ignore while indexing
XML files (and HTML files if using libxml2 for parsing HTML).  All text
within the tags will be ignored -- both for indexing (C<MetaNames>)
and properties (C<PropertyNames>).  To still parse properties, yet do
not index the text, see L<UndefinedMetaTags|/"item_UndefinedMetaTags">.

This option is useful to avoid indexing specific data from a file.
For example:

    <person>
        <first_name>
            William
        </first_name> <last_name>
            Shakespeare
        </last_name> <updated_date>
            April 25, 1999
        </updated_date>
    </person>

In the above example you might B<not> want to index the updated date,
and therefore prevent finding this record by searching

    -w 'person=(April)'

This is solved by:

    IgnoreMetaTags updated_date


See also L<UndefinedMetaTags|/"item_UndefinedMetaTags">.

=item IgnoreNumberChars *list of characters*

Experimental Feature

This experimental feature can be used to define a set of characters
that describe a number.  If a word is found to contain only those
characters it will not be indexed.  The characters listed must be part
of C<WordCharacters> settings.  In other words, the "word" checked is
a word that Swish-e would otherwise index.

For example,

    IgnoreNumberChars 0123456789$.,

Then Swish-e would not index the following:

    123
    123,456.78
    $123.45
    
You might be tempted to avoid indexing hex numbers with:

    IgnoreNumberChars 0123456789abcdef

which will not index 0D31, but will also not index the word "bad".

This is an experimental feature that may change in future versions.
One possible change is to use regular expressions instead.


=item IndexComments [NO|yes]

This option allows the user decide if to index the contents of HTML
comments.  Default is no. Set to yes if comment indexing is required.

    IndexComments yes

Note: This is a change in the default behavior prior to version 2.2.

=item TranslateCharacters [*string1 string2*|:ascii7:]

The TranslateCharacters directive maps the characters in string1 to the
characters listed in string2.

For example:

    # This will index a_b as a-b and �mo as amo
    TranslateCharacters _� -a

C<TranslateCharacters :ascii7:> is a predefined set of characters that
will translate eight bit characters to ascii7 characters.  Using the
:ascii7: rule will translate "���" to "aac". This means: searching
"�elik", "�elik" or "celik" will all match the same word.

TranslateCharacters is done early in the indexing process, after
converting HTML entities but before splitting the input text into words
based on B<WordCharacters>.  So characterters you are translating I<from>
do not need to be listed in word characters.

The same character translations take place when searching.

=item BumpPositionCounterCharacters *string*

When indexing Swish-e assigns a word position to each word.  This enables
phrase searching.  There may be cases where you would like to prevent
phrase matching.  The BumpPositionCounterCharacters directive allows
you to specify a set of characters that when found in the text will
increment the word position -- effectively preventing phrase matches
across that character.

For example, if you have a tag:

    <subjects>
        computer programming | apple computers
    </subjects>

You might want to prevent matching "programming apple" in that meta name.

    BumpPositionCounterCharacters |

There is no default, and you may list a string of characters.

=item DontBumpPositionOnEndTags *list of names*

=item DontBumpPositionOnStartTags *list of names*

Since metatags are typically separate data fields, the word position
counter is automatically bumped between metatags (actally, bumpted when a
start tag is found and when an end tag is found).  This prevents matching
a phrase that spans more than one metaname.  C<DontBumpPositionOnEndTags>
and C<DontBumpPositionOnStartTags> disables this feature for the listed
metanames.

For example,

    <person>
        <first_name>
            William
        </first_name>
        <last_name>
            Shakespeare
        </last_name>
        <updated_date>
            April 25, 1999
        </updated_date>
    </person>

In the conifuration file:

    DontBumpPositionOnEndTags first_name
    DontBumpPositionOnStartTags last_name

This configuration allows this phrase search

    -w 'person=("william shakespeare")'

but this phrase search will fail

    -w 'person=("shakespeare april")'
    

=back


=head2 Directives for the File Access method only

Some directives have different uses depending on the source of the
documents.  These directives are only valid when using the B<File system>
method of indexing.

=over 4

=item IndexOnly *list of file suffixes*

This directive specifies the allowable file suffixes (extensions) while
indexing.  The default is to index all files specified in B<IndexDir>.

    # Only index .html .htm and .q files
    IndexOnly .html .htm .q

C<IndexOnly> checks that the file end in the characters listed.  It does
not check "extensions".  C<IndexOnly> is tested right before C<FileRules>
is processed.

=item FollowSymLinks [yes|NO]

Put "yes" to follow symbolic links in indexing, else "no".  Default is no.

    FollowSymLinks no
    FollowSymLinks yes

Note that when set to C<no> extra stat(2) system calls must be made for
each file.  For large number of files you may see a small reduction in
indexing time by setting this to C<yes>.

See also the C<-l> switch in L<SWISH-RUN|SWISH-RUN>.

=item FileRules [type] [contains|is|regex] *regular expression*

=item FileMatch [type] [contains|is|regex] *regular expression*

FileRules and FileMatch are used to, respectively, exclude and include
files and directories to index.  Since, by default, Swish-e indexes all
files and recurses all directories (but see also C<FollowSymLinks>) you
will typically only use C<FileRules> to exclude files or directories.
C<FileMatch> is useful in a few cases, for example, to override the
behavior of C<IndexOnly>.  Some examples are included below.

Except for C<FileRules title ...>, this feature is only available for
file access method (-S fs), which is the default indexing mode.  Also,
any pathname modification with C<ReplaceRules> happens after the check
for C<FileRules>.  (It's unlikly that you would exclude files with
C<FileRules> based on text you added with C<ReplaceRules>!)

The regular expression is a C regex.h extended regular expression.
You may supply more than one regular expression per line, or use
separate directives.  Preceeding the regular expression with the word
"not" negates the match.

The regular expression is compared against B<[type]> as described below.

For historical reasons, you can specify C<contains> or C<is>.  C<is>
simply forces the regular expression to match at the start and end
of the string (by internally prepending "^" and appending "$" to the
regular expression).

The C<regex> option requires delimiter characters:

    FileRules title regex /^private/i

The only advantage of C<regex> is if you want to do case insensitive
matches, or simply like your regular expressions to look like perl
regular expressions.  You must use matching delimiters; (), {}, and [],
are not currently supported for no good reason other than laziness.

Use quotes (" or ') around a pattern if it contains any white space.
Note that the backslash character becomes the escape character within
quotes.

For example, these sets generate the same regular expressions.

    FileRules title is hello
    FileRules title contains ^hello$
    FileRules title regex /^hello$/

These all need quotes due to the included space character

    FileRules title is "hello there"
    FileRules title contains "^hello there$"
    FileRules title regex "!^hello there$!"

These show how the backslash must be doubled inside of quotes.
Swish-e converts a double-backslash into a single backslash, and then
passes that single onto the regular expression compiler.

    FileRules filename regex /\.pdf/
    FileRules filename regex "/\\.pdf/"

    FileRules filename regex !hello\\there!     # need double for real backslash 
    FileRules filename regex "!hello\\\\there!" # need double-double inside of quotes
    

B<Matching Types>

The following types of match strings my be supplied:

    FileRules pathname
    FileRules dirname
    FileRules filename
    FileRules directory
    FileRules title
    
    FileMatch pathname
    FileMatch filename
    FileMatch dirname
    FileMatch directory

B<pathname> matches the regular expression against the current pathname.
The pathname may or may not be absolute depending on what you supplied
to C<IndexDir>.

Example:

    # Don't index paths that contain private or hidden
    FileRules pathname contains (private|hidden)

    # Same thing
    FileRules pathname regex /(private|hidden)/

    # Don't index exe files
    FileRules pathname contains \.exe$

B<dirname> and B<filename> split the path name by the last delimiter
character into a directory name, and a file name.  Then these are compared
against the patterns supplied.  Directory names do B<not> have a trailing
slash.  All path names use the forward slash as a delimiter within Swish-e.

Example:

    # Same as last example - don't index *.exe files.
    FileRules filename contains \.exe$

    # Don't index any file called test.html files
    FileRules filename contains ^test\.html$

    # Same thing
    FileRules filename is test\.html

    # Don't index any directoires that contain "old"  (/usr/local/myold/docs)
    FileRules dirname contains old

    # Don't index any directories that contain the path segment "old" (/usr/local/old/foo)
    FileRules dirname contains /old/  

    # Index only .htm, .html, plus any all-digit file names
    IndexOnly .htm .html
    FileMatch filename contains ^\d+$

    # Same as previous, but maybe a little slower
    FileRules filename regex not !\.(htm|html)$!
    FileMatch filename contains ^\d+$

Swish-e checks these settings in the order of C<pathname>, C<dirname>, and
C<filename>, and C<FileMatch> patterns are checked before C<FileRules>,
in general.  This allows you to exclude most files with C<FileRules>,
yet allow in a few special cases with C<FileMatch>. For example:

    # Exclude all files of .exe, .bin, and .bat
    FileRules filename contains \.(exe|bin|bat)$
    # But, let these two in
    FileMatch filename is baseball\.bat incoming_mail\.bin

    # Same, but as a single pattern
    FileMatch filename is (baseball\.bat|incoming_mail\.bin)

The C<directory> type is somewhat unique. When Swish-e recurses into a
directory it will compare all the I<files> in the directory with the
pattern and then decide if that entire directory should or should not
be indexed (or recursed).  Note that you are matching against file names
in a directory -- and some of those names may be directory names.

A C<FileRules directory> match will cause Swish-e to ignore all files and
sub-directories in the current directory.

Warning: A match with C<FileMatch directory> says to index B<everything>
in the *current* directory and B<ignore> any FileRules for this directory.


Example:

    # Don't index any directories (and sub directories) that contain
    # a file (or sub-directory) called "index.skip"
    FileRules directory contains ^index\.skip$

    # Don't index directories that contain a .htaccess file.
    FileRules directory contains ^\.htaccess

Note: While I<processing> directories, Swish-e will ignore any files
or directories that begin with a dot (".").  You may index files
or directories that begin with a dot by specifying their name with
C<IndexDir> or C<-i>.

C<title> checks for a pattern match in an HTML title.

Example:

    FileRules title contains construction example pointers

    # This example says to ignore case
    FileRules title regex "/^Internal document/i"

Note: C<FileRules title> works for any input method (fs, prog, or http)
that is parsed as HTML, and where a title was found in the document.

In case all this seems a bit confusing, processing a directory happens
in the following order.

First the directory name is checked:

    FileRules dirname - reject entire directory if matches

Next the directory is scanned and each file name (which might be the
name of a sub-directory) is checked:

    FileRules directory - reject entire dir if any files match FileMatch
    directory - accept *entire* dir if any files match

Then, unless C<FileMatch directory> matched, each file is tested with
FileMatch.  A match says to index the file without further testing
(i.e. overrides FileRules and IndexOnly):

    FileMatch pathname  \
    FileMatch dirname   - file is accepted if any match
    FileMatch filename  /

otherwise    

    IndexOnly - file is checked for the correct file extension

    FileRules pathname  \
    FileRules dirname   - file is rejected if any match
    FileRules filename  /

finally, the file is indexed.

Files (not directories) listed with C<IndexDir> or C<-i> are processed
in a similar way:

    FileMatch pathname  \
    FileMatch dirname   - file is accepted if any match
    FileMatch filename  /

otherwise, the file is rejected if it doesn't have the correct extension
or a FileRules matches.

    IndexOnly - file is checked for the correct file extension

    FileRules pathname  \
    FileRules dirname   - file is rejected if any match
    FileRules filename  /

Note:  If things are not indexing as you expect, create a directory
with some test files and use the C<-T regex> trace option to see how
file names are checked.  Start with very simple tests!


=back    

=head2 Directives for the HTTP Access Method Only

These directives are available when using the HTTP Access Method of indexing.

=over 4

=item MaxDepth *integer*

MaxDepth defines how many links the spider should follow before stopping.
A value of 0 configures the spider to traverse all links.  The default
is MaxDepth 5.

    MaxDepth 5

=item Delay *seconds*

The number of seconds to wait between issuing requests to a server.
This setting allows for more friendly spidering of remote sites.
The default is 60 seconds.

    Delay 1

=item TmpDir *path*

The location of a writable temp directory on your system.  The HTTP
access method tells the Perl helper to place its files in this location,
and the C<-e> switch causes Swish-e to use this directory while indexing.
There is no default.

    TmpDir /tmp/swish

If this directory does not exist or is not writable Swish-e will fail
with an error during indexing.

Note, the environment variables of C<TMPDIR>, C<TMP>, and C<TEMP>
(in that order) will B<override> this setting.

=item SpiderDirectory *path*

The location of the Perl helper script called F<swishspider>.  If you
use a relative directory, it is relative to your directory when you run
Swish-e, not to the directory that Swish-e is in.  The default is C<./>

    SpiderDirectory /usr/local/swish

=item EquivalentServer *server alias*

Often times the same site may be referred to by different names.
A common example is that often http://www.some-server.com and
http://some-server.com are the same.  Each line should have a list of
all the method/names that should be considered equivalent.  Multiple
EquivalentServer directives may be used.  Each directive defines its
own set of equivalent servers.

    EquivalentServer http://library.berkeley.edu http://www.lib.berkeley.edu
    EquivalentServer http://sunsite.berkeley.edu:2000 http://sunsite.berkeley.edu

=back

=head2 Directives for the prog Access Method Only

This section details the directives that are only available for the
"prog" document source feature of Swish-e.  The "prog" access method runs
an external program that "feeds" documents to Swish-e.  This allows indexing
and filtering of documents from any source.

See L<prog - general purpose access method|SWISH-RUN/"item_prog"> in
the SWISH-RUN man page for more information.


A number of example programs for use with the "prog" access method are
provided in the F<prog-bin> directory.  Please see those example if you
have questions about implementing a "prog" input program.

=over 4

=item SwishProgParameters *list of parameters*

This is a list of parameters that will be sent to the external program
when running with the "prog" document source method.

    SwishProgParameters /path/to/config hello there
    IndexDir /path/to/program.pl

Then running:

    swish-e -c config -S prog

Swish-e will execute C</path/to/program.pl> and pass C</path/to/config
hello there> as three command line arguments to the program.  This
directive makes it easy to pass settings from the Swish-e configuration
file to the external program.

For example, the C<spider.pl> program (included in the C<prog-bin>
directory) uses the C<SwishProgParameters> to specify what file to read
for configuation information.

    SwishProgParameters spider.config
    IndexDir ./spider.pl

The C<spider.pl> program also has a default action so you can avoid
using a configuration file:

    SwishProgParameters default http://www.swishe.org/ http://some.other.site/
    IndexDir ./spider.pl

And the spider program will use default settings for spidering those sites.

=back

B<Notes when using MS Windows>

You should use unix style path separators to specify your external
program.  Swish will convert forward slashes to backslashes before
calling the external program.  This is only true for the program name
specified with C<IndexDir> or the C<-i> command line option.

In addition, Swish-e will make sure the program specified actually exists,
which means you need to use the full name of the program.

For example, to run the perl spider program F<spider.pl> you would need
a Swish-e configuration file such as:

    IndexDir e:/perl/bin/perl.exe
    SwishProgParameters prog-bin/spider.pl default http://swish-e.org

and run indexing with the command:

    swish-e -c swish.cfg -S prog -v 9

The C<IndexDir> command tells Swish-e the name of the program to run.
Under unix you can just specify the name of the script, since unix will
figure out the program from the first line of the script.

The C<SwishProgParameters> are the parameters passed to the program
specified by C<IndexDir> (perl.exe in this case).  The first parameter
is the perl script to run (F<prog-bin/spider.pl>).  Perl passes the rest
of the parameters directly to the perl script.  The second parameter
F<default> tells the F<spider.pl> program to use default settings for
spidering (or you could specify a spider config file -- see C<perldoc
spider.pl> for details), and lastly, the URL is passed into the spider
program.


=head2 Document Filter Directives

Internally, Swish-e knows how to parse only text, HTML, and XML documents.
With Swish-e filters you can index other types of documents.  For example,
if all your web pages are in gzip format a filter can uncompress these
on the fly for indexing.

A filter is an external program that Swish-e executes while processing
a document of a given type.  Swish-e will execute the filter program
for each file that matches the file suffix (extension) set in the
B<FileFilter> or B<FileFilterMatch> directives.  B<FileFilterMatch>
matches using regular expressions and is described below.

Swish-e calls the external program passing as B<default> arguments:

=over 4

=item $0 

the name of the filter program

=item $1

the physical path name of the file to read.  This may be a temporary
file location if indexing by the http method.

=item $2

When indexing under the file system this will be the same as $1 (the
path to the source file), but when indexing under the http method this
will be the URL of the source document.

=back

Swish-e can also pass other parameters to the filter program.  These
parameters can be defined using the B<FileFilter> or B<FileFilterMatch>
directives.  See Filter Options below.

The filter program must open the file, process its contents, and return
it to Swish-e by printing to STDOUT.

Note that this can add a significant amount of time to the indexing
process if your external program is a perl or shell script.  If you
have many files to filter you should consider writing your filter in C
instead of a shell or perl script, or using the "prog" Access Method.

=over 4

=item FilterDir  *path-to-directory*

This is the path to a directory where the filter programs are stored.
Swish-e looks in this directory to find the filter specified in the
B<FileFilter> directive. If this directive is omitted, you have to
specify the full path to the filterscript on each FileFilter directive.

This feature does *not* apply to the C<FileFilterMatch> directive.

Example:

    FilterDir /usr/local/swish/filters

=item FileFilter   *suffix*   "filter-prog"   ["filter-options"]

This maps file suffixe (extension) to a filter program.  If I<filter-prog>
starts with a directory delimiter (absolute path), Swish-e doesn't use
the FilterDir settings, but uses the given I<filter-prog> path directly.

Filter options:

Filter options are a string passed as arguments to the I<filter-prog>.
Filter options can contain variables, replaced by Swish-e.  If you ommit
I<filter-options> Swish-e will use default parameters for the options
listed above.

    Default:      "'%p' '%P'"
    Which means:  pass   "workfile path" and "documentfile path" to filter (each quoted).

Variables in filter options:

    %%   =  %
    %P   =  Full document pathname (e.g. URL, or path on filesystem)  
    %p   =  Full pathname to work file (maybe a tmpfile or the real document path on filesystem)
    %F   =  Filename stripped from full document pathname
    %f   =  Filename stripped from "work" pathname
    %D   =  Directoryname stripped from full document pathname
    %d   =  Directoryname stripped from full "work" pathname

Examples of strings passed:

    %P =  document pathname:  http://myserver/path1/mydoc.txt
    %p =  work pathname:      /tmp/tmp.1234.mydoc.txt
    %F =     mydoc.txt
    %f =     tmp.1234.mydoc.txt
    %D =     http://myserver/path1
    %d =     /tmp

Important hint for security:

When using variable substitution, use quotes to ensure filename integrity.

        e.g. "'%f'"  -->  'file name with spaces.doc'.

If you don't use this, your system security may be compromised, or
filtering may not work for these files.

B<Notes when using MS Windows>

Windows uses double quotes to escape shell metacharacters, so reverse
the quotes in the examples above.  e.g.:

        '"%f"' --> "file name with spaced.doc"

You can specify the filter program using forward slashes (unix style).
Swish will convert the slashes to backslashes before running your program.

    FileFilter .mydoc     c:/some/path/mydocfilter.exe  '-d "%d" -example -url "%P" "%f"'


Examples of filters:

    FileFilter .doc       /usr/local/bin/catdoc "-s8859-1 -d8859-1 '%p'"
    FileFilter .pdf       pdftotext   "'%p' -"
    FileFilter .html.gz   gzip  "-c '%p'"
    FileFilter .mydoc     "/some/path/mydocfilter"  "-d '%d' -example -url '%P' '%f'"

The above examples are running a I<binary> filter program.  For more
complicated filtering needs you may use a scripting language such as
Perl or a shell script.  Here's some examples of calling a shell and
perl script:

    FileFilter .pdf       pdf2html.sh
    FileFilter .ps        ghostscript-filter.pl

Using a scripting language (or any language that has a large startup
cost) can B<greatly increase the indexing time>.  For small indexing
jobs, this may not be an issue, but for large collections of files that
require processing by a scripting language, you may be better off using
the C<-S prog> access method where the script will only be compiled once,
instead of for each document.

Filters are probably easier to write than a C<-S prog> program.  Which you
decide to use depends on your requirements.  Examples of filter scripts
can be found in the F<filter-bin> directory, and examples of C<-S prog>
programs can be found in the F<prog-bin> directory.

=item FileFilterMatch   *filter-prog*   *filter-options*  *regex* [*regex* ...]

This is similar to C<FileMatch> except uses regular expressions to
match against the file name.  *filter-prog* is the path to the program.
Unlike C<FileFilter> this does B<not> use the C<FilterDir> option.
Also unlike C<FileFilter> you B<must> specify the *filter-options*.

Examples:

    FileFilterMatch ./pdftotext "'%p' -" /\.pdf$/

Note that will also match a file called ".pdf", so you may want to use
something that requires a filename that has more than just an extension.
For example:

    FileFilterMatch ./pdftotext "'%p' -" /.\.pdf$/

To specify more than one extension:

    FileFilterMatch ./check_title.pl "%p" /\.html$/  /\.htm$/
    
Or a few ways to do the same thing:

    FileFilterMatch ./check_title.pl %p /\.(html|html)$/
    FileFilterMatch ./check_title.pl %p /\.html?$/

And to ignore case:    

    FileFilterMatch ./check_title.pl %p /\.html?$/i

You may also precede an expression with "not" to negate regular expression
that follow.  For example, to match files that do not have an extension:

    FileFilterMatch ./convert "%p %P" not /\..+$/

=back

=head1 Document Info

$Id: SWISH-CONFIG.pod,v 1.60 2002/08/28 14:30:23 whmoseley Exp $

.