swish-e/pod/SWISH-SEARCH.pod

=head1 NAME

SWISH-SEARCH - Swish-e Searching Instructions

=head1 OVERVIEW

This page describes the process of searching with Swish-e.  Please see
the L<SWISH-CONFIG|SWISH-CONFIG> page for information the Swish-e
configuration file directives, and L<SWISH-RUN|SWISH-RUN> for a complete
list of command line arguments.

Searching a Swish-e index involves passing L<command line
arguments|SWISH-RUN> to it that specify the index file to use, and
the L<query|/"Searching Syntax and Operations"> (or search words) to
locate in the index.  Swish-e returns a list of file names (or URLs)
that contain the matched search words.  L<Perl|/"Searching with Perl">
is often used as a front-end to Swish-e such as in CGI applications,
and L<perl modules|/"Perl Modules"> exist to for interfacing with Swish-e.

=head1 Searching Syntax and Operations

The C<-w> command line argument (switch) is used specify the search
query to Swish-e.  When running Swish-e from a shell prompt, be careful
to protect your query from shell metacharacters and shell expansions.
This often means placing single or double quotes around your query.
See L<Searching with Perl> if you plan to use Perl as a front end
to Swish-e.

The following section describes various aspects of searching with Swish-e.

=head2 Boolean Operators

You can use the Boolean operators B<and>, B<or>, or B<not> in searching.
Without these Boolean operators, Swish-e will assume you're B<and>ing
the words together.  The operators are not case sensitive.

[Note: you can change the default to B<or>ing by changing the variable
DEFAULT_RULE in the config.h file and recompiling Swish-e.]

Evaluation takes place from B<left to right> only, although you can use
parentheses to force the order of evaluation.

Examples:

     swish-e -w "smilla or snow" -f myIndex

Retrieves files containing either the words "smilla" or "snow".

     swish-e -w "smilla and snow not sense" -f myIndex 
     swish-e -w "(smilla and snow) and not sense" -f myIndex (same thing)

retrieves first the files that contain both the words "smilla" and
"snow"; then among those the ones that do not contain the word "sense".


=head2 Truncation

The wildcard (*) is available, however it can only be used at the end
of a word: otherwise is is considerd a normal character (i.e. can be
searched for if included in the WordCharacters directive).

     swish-e -w "librarian" -f myIndex

this query only retrieves files which contain the given word.

On the other hand:

     swish-e -w "librarian*" -f myIndex

retrieves "librarians", "librarianship", etc. along with "librarian".

Note that wildcard searches combined with word stemming can lead
to unexpected results.  If stemming is enabled, a search term with a
wildcard will be stemmed internally before searching.  So searching for
C<running*> will actually be a search for C<run*>, so C<running*> would
find C<runway>.  Also, searching for C<runn*> will not find C<running>
as you might expect, since C<running> stems to C<run> in the index,
and thus C<runn*> will not find C<run>.


=head2 Order of Evaluation

Expressions are always evaluated left to right: 

    swish -w "juliet not ophelia and pac" -f myIndex 

retrieves files which contain "juliet" and "pac" but not "ophelia" 

However it is always possible to force the order of evaluation by using
parenthesis.  For example:

    swish-e -w "juliet not (ophelia and pac)" -f myIndex 

retrieves files with "juliet" and containing neither "ophelia" nor "pac".

=head2 Meta Tags

MetaNames are used to represent I<fields> (called I<columns> in a
database) and provide a way to search in only parts of a document.
See L<SWISH-CONFIG|SWISH-CONFIG/"Document Contents Directives"> for
a description of MetaNames, and how they are specified in the source
document.

To limit a search to words found in a meta tag you prefix the keywords
with the name of the meta tag, followed by the equal sign:

    metaname = word
    metaname = (this or that)
    metatname = ( (this or that) or "this phrase" )

It is not necessary to have spaces at either side of the "=", consequently
the following are equivalent:

    swish-e -w "metaName=word"
    swish-e -w "metaName = word" 
    swish-e -w "metaName= word" 

To search on a word that contains a "=", precede the "=" with a "\"
(backslash).

    swish-e -w "test\=3 = x\=4 or y\=5" -f <index.file> 

this query returns the files where the word "x=4" is associated with
the metaName "test=3" or that contains the word "y=5" not associated
with any metaName.

Queries can be also constructed using any of the usual search features,
moreover metaName and plain search can be mixed in a single query.

     swish-e -w "metaName1 = (a1 or a4) not (a3 and a7)" -f index.swish-e

This query will retrieve all the files in which "a1" or "a2" are found
in the META tag "metaName1" and that do not contain the words "a3" and
"a7", where "a3" and "a7" are not associated to any meta name.

=head2 Phrase Searching

To search for a phrase in a document use double-quotes to delimit your
search terms.  (The phrase delimiter is set in src/swish.h.)

You must protect the quotes from the shell.

For example, under Unix:

    swish-e -w '"this is a pharase" or (this and that)'
    swish-e -w 'meta1=("this is a pharase") or (this and that)'

Or under Windows:

    swish-e -w \"this is a pharase\" or (this and that)

You can use the C<-P> switch to set the phrase delimiter character.
See L<SWISH-RUN|SWISH-RUN> for examples.

  
=head2 Context

At times you might not want to search for a word in every part of
your files since you know that the word(s) are present in a particular
tag. The ability to seach according to context greatly increases the
chances that your hits will be relevant, and Swish-e provides a mechanism
to do just that.

The -t option in the search command line allows you to search for words
that exist only in specific HTML tags. Each character in the string
you specify in the argument to this option represents a different tag
in which the word is searched; that is you can use any combinations of
the following characters:

    H means all<HEAD> tags 
    B stands for <BODY> tags 
    t is all <TITLE> tags 
    h is <H1> to <H6> (header) tags 
    e is emphasized tags (this may be <B>, <I>, <EM>, or <STRONG>) 
    c is HTML comment tags (<!-- ... -->)

    # This search will look for files with these two words in their titles only.
    swish-e -w "apples oranges" -t t -f myIndex

    # This search will look for files with these words in comments only.
    swish-e -w "keywords draft release" -t c -f myIndex

    This search will look for words in titles, headers, and emphasized tags.
    swish-e -w "world wide web" -t the -f myIndex

=head1 Searching with Perl

Perl ( http://www.perl.com/ ) is probably the most common programming
language used with Swish-e, especially in CGI interfaces.  Perl makes
searching and parsing results with Swish-e easy, but if not done properly
can leave your server vulnerable to attacks.

When designing your CGI scripts you should carefully screen user input,
and include features such as paged results and a timer to limit time
required for a search to complete.  These are to protect your web site
against a denial of service (DoS) attack.

Included with every distribution of Perl is a document called perlsec --
Perl Security.  I<Please> take time to read and understand that document
before writing CGI scripts in perl.

Type at your shell/command prompt:

    perldoc perlsec

If nothing else, start every CGI program in perl as such:

    #!/usr/local/bin/perl -wT
    use strict;

That alone won't make your script secure, but may help you find insecure
code.

=head2 CGI Danger!

There are many examples of CGI scripts on the Internet.  Many are poorly
written and insecure.  A commonly seen way to execute Swish-e from a
perl CGI script is with a I<piped open>.  For example, it is common to
see this type of C<open()>:

    open(SWISH, "$swish -w $query -f $index|");

This C<open()> gives shell access to the entire Internet!  Often an
attempt is made to strip C<$query> of I<bad> characters.  But, this
often fails since it's hard to guess what every I<bad> character is.
Would you have thought about a null?  A better approach is to only allow
I<in> known safe characters.

Even if you can be sure that any user supplied data is safe, this
I<piped open> still passes the command parameters through the shell.
If nothing else, it's just an extra unnecessary step to running Swish-e.

Therefore, the recommended approach is to fork and exec C<swish-e> directly
without passing through the shell.  This process is described in the
perl man page C<perlipc> under the appropriate heading B<Safe Pipe Opens>.

Type:

    perldoc perlipc

If all this sounds complicated you may wish to use a Perl module that
does all the hard work for you.

=head2 Perl Modules

There are a couple of Perl modules for accessing Swish-e.  One of the
modules is included with the distribution, and the other module (or set
of modules) is located on CPAN.  The included module provides a way to
embed Swish-e into your perl program, while the modules on CPAN provide an
abstracted interface to it.  Hopefully, they make using Swish-e easier.

B<The Included SWISHE Perl Module>

When compiling Swish-e from source the build process creates a C library
(see the L<Swish-e INSTALL|INSTALL/"Installing_the_SWISH_E_C_Library">
documentation).  The Swish-e distribution includes a F<perl> directory
with files required to create the F<SWISHE.pm> module.  This module
will I<embed> Swish-e into your perl program so that searching does not
require running an external program.  Embedding the Swish-e program into
your perl program results in faster Swish-e searches since it avoids the
cost of forking and exec'ing a separate program and opening the index
file for each request.

You will probably B<not> want to embed Swish-e into perl if running under
mod_perl as you will end up with very large Apache processes.

Building and usage instructions for the F<SWISHE.pm> module can be found
in the L<SWISH-PERL|SWISH-PERL> man page.

Here's an edited snip from that man page:

    my $handle = SwishOpen( $indexfiles )
        or die "Failed to open '$indexfiles'";

    my $num_results = SwishSearch($handle, $query, 1, $props, $sort);

    unless ( $num_results ) {
        print "No Results\n";

        my $error = SwishError( $handle );
        print "Error number: $error\n" if $error;

        return;  # or next.
    }

    while( my($rank,$index,$file,$title,$start,$size,@props)
        = SwishNext( $handle ))
    {
        print join( ' ',
              $rank,
              $index,
              $file,
              qq["$title"],
              $start,
              $size,
              map{ qq["$_"] } @props,
              ),"\n";
    }


B<SWISH Modules on CPAN>

The Comprehensive Perl Archive Network, or CPAN, is a collection of
modules for use with Perl.  Always search CPAN (http://search.cpan.org/)
before starting any new program.  Chances are someone has written just
what you need.

On CPAN are also modules for searching with Swish-e.  They can be found
at http://search.cpan.org/search?mode=module&query=SWISH  The main
SWISH module (different from the SWISHE<E> module included with the
Swish-e distribution) provides a high-level Object Oriented interface
to Swish-e, and the same interface can be used to used to either fork
and exec the Swish-e binary, or use the Swish-e C Library if installed
by just changing one line of code.  A server interface will be written
when a Swish-e server is written.

The main idea is that you can write a program to search with Swish-e,
but not have to change your code (much) when you wish to change to a
new way of accessing Swish-e.

Here's an example of SWISH module usage from the synopsis:

    use SWISH;

    $sh = SWISH->connect('Fork',
       prog     => '/usr/local/bin/swish-e',
       indexes  => 'index.swish-e',
       results  => sub { print $_[1]->as_string,"\n" },
    ) or die $SWISH::errstr unless $sh;

    $hits = $sh->query('metaname=(foo or bar)');

This takes care of running Swish-e in a secure way, parsing the output
from it, and providing OO methods of accessing the resulting data.

=head1 Document Info

$Id: SWISH-SEARCH.pod,v 1.4 2002/04/15 02:34:43 whmoseley Exp $

.
1	=head1 NAME
2
3	SWISH-SEARCH - Swish-e Searching Instructions
4
5	=head1 OVERVIEW
6
7	This page describes the process of searching with Swish-e. Please see
8	the L<SWISH-CONFIG\|SWISH-CONFIG> page for information the Swish-e
9	configuration file directives, and L<SWISH-RUN\|SWISH-RUN> for a complete
10	list of command line arguments.
11
12	Searching a Swish-e index involves passing L<command line
13	arguments\|SWISH-RUN> to it that specify the index file to use, and
14	the L<query\|/"Searching Syntax and Operations"> (or search words) to
15	locate in the index. Swish-e returns a list of file names (or URLs)
16	that contain the matched search words. L<Perl\|/"Searching with Perl">
17	is often used as a front-end to Swish-e such as in CGI applications,
18	and L<perl modules\|/"Perl Modules"> exist to for interfacing with Swish-e.
19
20	=head1 Searching Syntax and Operations
21
22	The C<-w> command line argument (switch) is used specify the search
23	query to Swish-e. When running Swish-e from a shell prompt, be careful
24	to protect your query from shell metacharacters and shell expansions.
25	This often means placing single or double quotes around your query.
26	See L<Searching with Perl> if you plan to use Perl as a front end
27	to Swish-e.
28
29	The following section describes various aspects of searching with Swish-e.
30
31	=head2 Boolean Operators
32
33	You can use the Boolean operators B<and>, B<or>, or B<not> in searching.
34	Without these Boolean operators, Swish-e will assume you're B<and>ing
35	the words together. The operators are not case sensitive.
36
37	[Note: you can change the default to B<or>ing by changing the variable
38	DEFAULT_RULE in the config.h file and recompiling Swish-e.]
39
40	Evaluation takes place from B<left to right> only, although you can use
41	parentheses to force the order of evaluation.
42
43	Examples:
44
45	swish-e -w "smilla or snow" -f myIndex
46
47	Retrieves files containing either the words "smilla" or "snow".
48
49	swish-e -w "smilla and snow not sense" -f myIndex
50	swish-e -w "(smilla and snow) and not sense" -f myIndex (same thing)
51
52	retrieves first the files that contain both the words "smilla" and
53	"snow"; then among those the ones that do not contain the word "sense".
54
55
56	=head2 Truncation
57
58	The wildcard (*) is available, however it can only be used at the end
59	of a word: otherwise is is considerd a normal character (i.e. can be
60	searched for if included in the WordCharacters directive).
61
62	swish-e -w "librarian" -f myIndex
63
64	this query only retrieves files which contain the given word.
65
66	On the other hand:
67
68	swish-e -w "librarian*" -f myIndex
69
70	retrieves "librarians", "librarianship", etc. along with "librarian".
71
72	Note that wildcard searches combined with word stemming can lead
73	to unexpected results. If stemming is enabled, a search term with a
74	wildcard will be stemmed internally before searching. So searching for
75	C<running> will actually be a search for C<run>, so C<running*> would
76	find C<runway>. Also, searching for C<runn*> will not find C<running>
77	as you might expect, since C<running> stems to C<run> in the index,
78	and thus C<runn*> will not find C<run>.
79
80
81	=head2 Order of Evaluation
82
83	Expressions are always evaluated left to right:
84
85	swish -w "juliet not ophelia and pac" -f myIndex
86
87	retrieves files which contain "juliet" and "pac" but not "ophelia"
88
89	However it is always possible to force the order of evaluation by using
90	parenthesis. For example:
91
92	swish-e -w "juliet not (ophelia and pac)" -f myIndex
93
94	retrieves files with "juliet" and containing neither "ophelia" nor "pac".
95
96	=head2 Meta Tags
97
98	MetaNames are used to represent I<fields> (called I<columns> in a
99	database) and provide a way to search in only parts of a document.
100	See L<SWISH-CONFIG\|SWISH-CONFIG/"Document Contents Directives"> for
101	a description of MetaNames, and how they are specified in the source
102	document.
103
104	To limit a search to words found in a meta tag you prefix the keywords
105	with the name of the meta tag, followed by the equal sign:
106
107	metaname = word
108	metaname = (this or that)
109	metatname = ( (this or that) or "this phrase" )
110
111	It is not necessary to have spaces at either side of the "=", consequently
112	the following are equivalent:
113
114	swish-e -w "metaName=word"
115	swish-e -w "metaName = word"
116	swish-e -w "metaName= word"
117
118	To search on a word that contains a "=", precede the "=" with a "\"
119	(backslash).
120
121	swish-e -w "test\=3 = x\=4 or y\=5" -f <index.file>
122
123	this query returns the files where the word "x=4" is associated with
124	the metaName "test=3" or that contains the word "y=5" not associated
125	with any metaName.
126
127	Queries can be also constructed using any of the usual search features,
128	moreover metaName and plain search can be mixed in a single query.
129
130	swish-e -w "metaName1 = (a1 or a4) not (a3 and a7)" -f index.swish-e
131
132	This query will retrieve all the files in which "a1" or "a2" are found
133	in the META tag "metaName1" and that do not contain the words "a3" and
134	"a7", where "a3" and "a7" are not associated to any meta name.
135
136	=head2 Phrase Searching
137
138	To search for a phrase in a document use double-quotes to delimit your
139	search terms. (The phrase delimiter is set in src/swish.h.)
140
141	You must protect the quotes from the shell.
142
143	For example, under Unix:
144
145	swish-e -w '"this is a pharase" or (this and that)'
146	swish-e -w 'meta1=("this is a pharase") or (this and that)'
147
148	Or under Windows:
149
150	swish-e -w \"this is a pharase\" or (this and that)
151
152	You can use the C<-P> switch to set the phrase delimiter character.
153	See L<SWISH-RUN\|SWISH-RUN> for examples.
154
155
156	=head2 Context
157
158	At times you might not want to search for a word in every part of
159	your files since you know that the word(s) are present in a particular
160	tag. The ability to seach according to context greatly increases the
161	chances that your hits will be relevant, and Swish-e provides a mechanism
162	to do just that.
163
164	The -t option in the search command line allows you to search for words
165	that exist only in specific HTML tags. Each character in the string
166	you specify in the argument to this option represents a different tag
167	in which the word is searched; that is you can use any combinations of
168	the following characters:
169
170	H means all<HEAD> tags
171	B stands for <BODY> tags
172	t is all <TITLE> tags
173	h is <H1> to <H6> (header) tags
174	e is emphasized tags (this may be <B>, <I>, <EM>, or <STRONG>)
175	c is HTML comment tags (<!-- ... -->)
176
177	# This search will look for files with these two words in their titles only.
178	swish-e -w "apples oranges" -t t -f myIndex
179
180	# This search will look for files with these words in comments only.
181	swish-e -w "keywords draft release" -t c -f myIndex
182
183	This search will look for words in titles, headers, and emphasized tags.
184	swish-e -w "world wide web" -t the -f myIndex
185
186	=head1 Searching with Perl
187
188	Perl ( http://www.perl.com/ ) is probably the most common programming
189	language used with Swish-e, especially in CGI interfaces. Perl makes
190	searching and parsing results with Swish-e easy, but if not done properly
191	can leave your server vulnerable to attacks.
192
193	When designing your CGI scripts you should carefully screen user input,
194	and include features such as paged results and a timer to limit time
195	required for a search to complete. These are to protect your web site
196	against a denial of service (DoS) attack.
197
198	Included with every distribution of Perl is a document called perlsec --
199	Perl Security. I<Please> take time to read and understand that document
200	before writing CGI scripts in perl.
201
202	Type at your shell/command prompt:
203
204	perldoc perlsec
205
206	If nothing else, start every CGI program in perl as such:
207
208	#!/usr/local/bin/perl -wT
209	use strict;
210
211	That alone won't make your script secure, but may help you find insecure
212	code.
213
214	=head2 CGI Danger!
215
216	There are many examples of CGI scripts on the Internet. Many are poorly
217	written and insecure. A commonly seen way to execute Swish-e from a
218	perl CGI script is with a I<piped open>. For example, it is common to
219	see this type of C<open()>:
220
221	open(SWISH, "$swish -w $query -f $index\|");
222
223	This C<open()> gives shell access to the entire Internet! Often an
224	attempt is made to strip C<$query> of I<bad> characters. But, this
225	often fails since it's hard to guess what every I<bad> character is.
226	Would you have thought about a null? A better approach is to only allow
227	I<in> known safe characters.
228
229	Even if you can be sure that any user supplied data is safe, this
230	I<piped open> still passes the command parameters through the shell.
231	If nothing else, it's just an extra unnecessary step to running Swish-e.
232
233	Therefore, the recommended approach is to fork and exec C<swish-e> directly
234	without passing through the shell. This process is described in the
235	perl man page C<perlipc> under the appropriate heading B<Safe Pipe Opens>.
236
237	Type:
238
239	perldoc perlipc
240
241	If all this sounds complicated you may wish to use a Perl module that
242	does all the hard work for you.
243
244	=head2 Perl Modules
245
246	There are a couple of Perl modules for accessing Swish-e. One of the
247	modules is included with the distribution, and the other module (or set
248	of modules) is located on CPAN. The included module provides a way to
249	embed Swish-e into your perl program, while the modules on CPAN provide an
250	abstracted interface to it. Hopefully, they make using Swish-e easier.
251
252	B<The Included SWISHE Perl Module>
253
254	When compiling Swish-e from source the build process creates a C library
255	(see the L<Swish-e INSTALL\|INSTALL/"Installing_the_SWISH_E_C_Library">
256	documentation). The Swish-e distribution includes a F<perl> directory
257	with files required to create the F<SWISHE.pm> module. This module
258	will I<embed> Swish-e into your perl program so that searching does not
259	require running an external program. Embedding the Swish-e program into
260	your perl program results in faster Swish-e searches since it avoids the
261	cost of forking and exec'ing a separate program and opening the index
262	file for each request.
263
264	You will probably B<not> want to embed Swish-e into perl if running under
265	mod_perl as you will end up with very large Apache processes.
266
267	Building and usage instructions for the F<SWISHE.pm> module can be found
268	in the L<SWISH-PERL\|SWISH-PERL> man page.
269
270	Here's an edited snip from that man page:
271
272	my $handle = SwishOpen( $indexfiles )
273	or die "Failed to open '$indexfiles'";
274
275	my $num_results = SwishSearch($handle, $query, 1, $props, $sort);
276
277	unless ( $num_results ) {
278	print "No Results\n";
279
280	my $error = SwishError( $handle );
281	print "Error number: $error\n" if $error;
282
283	return; # or next.
284	}
285
286	while( my($rank,$index,$file,$title,$start,$size,@props)
287	= SwishNext( $handle ))
288	{
289	print join( ' ',
290	$rank,
291	$index,
292	$file,
293	qq["$title"],
294	$start,
295	$size,
296	map{ qq["$_"] } @props,
297	),"\n";
298	}
299
300
301	B<SWISH Modules on CPAN>
302
303	The Comprehensive Perl Archive Network, or CPAN, is a collection of
304	modules for use with Perl. Always search CPAN (http://search.cpan.org/)
305	before starting any new program. Chances are someone has written just
306	what you need.
307
308	On CPAN are also modules for searching with Swish-e. They can be found
309	at http://search.cpan.org/search?mode=module&query=SWISH The main
310	SWISH module (different from the SWISHE<E> module included with the
311	Swish-e distribution) provides a high-level Object Oriented interface
312	to Swish-e, and the same interface can be used to used to either fork
313	and exec the Swish-e binary, or use the Swish-e C Library if installed
314	by just changing one line of code. A server interface will be written
315	when a Swish-e server is written.
316
317	The main idea is that you can write a program to search with Swish-e,
318	but not have to change your code (much) when you wish to change to a
319	new way of accessing Swish-e.
320
321	Here's an example of SWISH module usage from the synopsis:
322
323	use SWISH;
324
325	$sh = SWISH->connect('Fork',
326	prog => '/usr/local/bin/swish-e',
327	indexes => 'index.swish-e',
328	results => sub { print $_[1]->as_string,"\n" },
329	) or die $SWISH::errstr unless $sh;
330
331	$hits = $sh->query('metaname=(foo or bar)');
332
333	This takes care of running Swish-e in a secure way, parsing the output
334	from it, and providing OO methods of accessing the resulting data.
335
336	=head1 Document Info
337
338	$Id: SWISH-SEARCH.pod,v 1.4 2002/04/15 02:34:43 whmoseley Exp $
339
340	.