swish-e/perl/SWISHE.pm

package SWISHE;

require Exporter;
require DynaLoader;

@ISA = qw(Exporter DynaLoader);

# Probably shouldn't export everything 
@EXPORT = qw(
    SwishOpen
    SwishSearch
    SwishClose
    SwishNext
    SwishSeek
    SwishError
    SwishHeaderParameter
    SwishStopWords
    SwishWords
    SwishStem
    SwishErrorString
    SwishHeaders
    SetLimitParameter
    ClearLimitParameter
);

$VERSION = '0.02';

bootstrap SWISHE $VERSION;

# Preloaded methods go here.

# Autoload methods go after __END__, and are processed by the autosplit program.

1;
__END__

=head1 NAME

SWISH-PERL - Perl Interface to the SWISH-E Library

=head1 SYNOPSIS

    use SWISHE;

    my $indexfilename1 = '/path/to/index1.swish';
    my $indexfilename2 = '/path/to/index2.swish';

    # To search for several indexes just put them together
    $indexfiles = "$indexfilename1 $indexfilename2";

    my $handle = SwishOpen( $indexfiles )
        or die "Failed to open '$indexfiles'";

    # Get a few headers from the index files
    my @headers = qw/WordCharacters BeginCharacters EndCharacters/;
    for ( @headers ) {
        my @h = SwishHeaderParameter( $handle, $_ );
        print "$_ for index 0 is $h[0]\n",
              "$_ for index 1 is $h[1]\n\n";
    }


    # Now search
    @standard = ('Rank', 'File Name', 'Title', 'Document Size');
    @props = qw/prop1 prop2 prop3/;
    
    $props = join ' ', @props;
    $sort  = 'prop1 asc prop2 desc';
    $query = 'meta1=metatest1';

    my $num_results = SwishSearch($handle, $query, 1, $props, $sort);

    if ( $num_results <= 0 ) {
        print ($num_results ? SwishErrorString( $handle ) : 'No Results');

        my $error = SwishError( $handle );
        print "\nError number: $error\n" if $error;

        return;  # or next.
    }

    my %results; # place to store the return values by name
    
    while(  @results{ @standard, @props } = SwishNext( $handle )) {
        print "\n";
        printf("%20s -> '%s'\n", $_, $results{$_}) for @standard, @props;
    }

    # No more queries on these indexes
    SwishClose( $handle );


=head1 ABSTRACT

Swish-e version 2.1.x creates an archive library of the internal Swish-e C functions.
This perl module provides access to those functions by embedding the Swish-e code in
your application.  The benefits are faster searches (no need to fork/execute an external program)
and avoids commonly used unsafe system calls.

This module provides direct access to the Swish-e C library functions.  For a higher level, object
oriented interface to SWISH visit http://search.cpan.org/search?mode=module&query=SWISH

=head2 Warnings and Gotchas

The Swish-e library was created from the executable program (instead of the other way around).
The Swish-e executable program handles errors by printing them to STDOUT and then aborting.
Not exactly desired behavior from a library.  
This means you should carefully screen your input data before calling the library.

Another minor issue is that the Swish-e library currently includes all code -- both searching and
indexing code.  Your program will be larger than needed due to this, but hopefully your OS will
be smart and share this code across processes and the impact will be minimal.

The library is often the last code to get checked during Swish-e development.  Test
carefully, and watch for memory leaks if running in a persistent environment.


=head1 INSTALLATION

Before you can build the perl module you must build and install SWISH-E.  Please read the
B<INSTALL> documentation included in the SWISHE distribution package.

    perldoc INSTALL

After building the SWISHE executable and successfully running make test, you will need to install
the SWISHE archive library.  This is done while in the top-level directory of the SWISHE distribution.

    % su root
    % make install-lib
    % exit

This will install the archive library (F<libswish-e.a>) into /usr/local/lib by default.

Next, build the perl module.

    % cd perl
    % perl Makefile.PL
    % make
    % make test
    % su root
    % make install
    % exit

If you do not have root access you can instead use

    % perl Makefile.PL PREFIX=/path/to/my/local/perl/library

And then in your perl script:

    use lib '/path/to/my/local/perl/library';


To test it you can run the test.pl script.  Type "./test.pl" at your command prompt.
This perl script uses the index file built by the "make test" command used during the build
of SWISHE as described in the B<INSTALL> document.

B<NOTE> Currently swish will exit the running program on some fatal errors.  In general
this should not happen, but it is something to think about if running under mod_perl
as an error will kill off the web server process.  Apache, for example, should recover
by starting up a new child process.  But, it's not a very graceful way to handle errors.

=head1 FUNCTION DESCRIPTIONS

The following describes the perl interface to the SWISHE C library.

=over 4

=item B<$handle = SwishOpen( $IndexFiles );>

Open one or more index files and returns a handle.

    Examples:

        $handle = SwishOpen( 'index_file.idx' );

        # open two indexes
        $handle = SwishOpen( 'index1.idx index2.idx' );

Returns undefined on an error, but the only errors are typically fatal, so
will most likely exit the running program.

If running under a persistent framework (such as mod_perl) you may run many queries
against the $handle.  This results in much faster searches.  For example, on searching
an index with 24,000 files and returning (the same) 400 results, opening and closing the index
for each search resulted in about 15 queries per second.  Leaving the index open resulted
in about 150 queries per second.


=item B<SwishClose( $handle );>

Closes the handle returned by SwishOpen.  
Closes all the opened files and frees the used memory.

=item B<$num_results = SwishSearch($handle, $search, $structure, $properties, $sortspec);>

Returns the number of hits, zero for no results, or a negative number.
If negative SwishErrorString( $handle ) will return the error message.

The values passed are:

=over 2

=item *

$handle is the handle returned by SwishOpen

=item *

$search is the search string.

Examples:
    my $query = 'title="this a is phrase"';
    my $query = '(title="this phrase") or (description=(these words))';

=item *

$structure is an integer value only applicable for an html search.  It defines
where in an html search to look.
It can be IN_FILE, IN_TITLE, IN_HEAD, IN_BODY, IN_COMMENTS, IN_HEADER or IN_EMPHASIZED or
or'ed combinations of them (e.g.: IN_HEAD | IN_BODY).
Use IN_FILE (a value of 1) if your documents are not html.
The numerical values for these constants are in src/swish.h

You can define these in your code with:

    # Set bits
    use constant IN_FILE  => 1;
    use constant IN_TITLE => 2;
    use constant IN_HEAD  => 4;

Not many people use the structure feature.    

=item *

$properties is a string with the properties to be returned separated by spaces.  Properties must
be defined during indexing.  See B<README-SWISHE> for more information.

Example:

    my $properties = 'subject description';

You may also use the swish internal properties:

    my $properties = 'subject description swishrank swishlastmodified';


=item *

$sortspec is the sort spec if different from relevance.

Examples:
    my $sortspec = ''  # sort by relevance

    # sort first in ascending order by title,
    # then by other fields in descending order
    my $sortspec = 'title asc category desc category desc';

=back

=item B<SetLimitParameter( $handle, $property, $low, $high )>

This experimental feature allows limiting results to a range of values
for a given property.  For example, to limit Titles:

    SetLimitParameter( $handle, 'swishtitle', 'a', 'zzzzzzzzzz' );

Limits titles to the range specified.

Note, if you do not call SwishOpen() and SwishClose() for every search
you must clear the limit selection with C<ClearLimitParameter>.


=item B<ClearLimitParameter( $handle )>

This function must be called before calling SetLimitParameter() again while
making repeated queries on an open index.

Once a limit is set on an open index, that limit stays in effect.  If you want to
change the limit, or just remove the limit you must call ClearLimitParameter().
For example:

    $handle = SwishOpen( $index );
    while ( $query = next_query() ) {
        ClearLimitParameter( $handle );

        my @limits = $query->limits;
        SetLimitParameter( $handle, @limits} ) if @limits;

        SwishSearch($handle, $query->search, 1 );

        ...
    }

    SwishClose( $handle );

There's no harm in calling ClearLimitParameter() if a limit is not going to be used.

This behavior may change in the future.


=item B<SwishNext( $handle )>

($rank, $filename, $title, $size, @properties) = SwishNext( $handle );

This function returns the next hit of a search. Must be executed after SwishSearch to read the results.

=over 2

=item *

$rank - An integer from 1 to 1000 indicating the relevance of the result

=item *

$filename - The source filename

=item *

$title - The title as indexed (as found in the HTML E<lt>TITLEE<gt> section)

=item *

$size - The length of the source document

=item *

@properties - The list of properties returned for this result.

=back

See the SYNOPSIS above for an example.


=item B<$rc=SwishSeek($handle, $num);>

Repositions the pointer in the result list to the element pointed by num.
It is useful when you want to read only the results starting at $num (e.g. for showing
results one page at a time).

=item B<$error_number=SwishError($handle);>

Returns the last error if any (always a negative value).
If there is not an error it will return 0.

=item B<$error_string=SwishErrorString( $handle );>

Returns the error string for the number supplied.

    print 'Error: ', SwishErrorString( $handle ), "\n";

=item B<@ParameterArray=SwishHeaderParameter($handle,$HeaderParameterName);>

This function is useful to access the header data of the index files
Returns the contents of the requested header parameter of all index files
opened by SwishOpen in an array.

Example:

    @wordchars = SwishHeaderParameter( $handle, 'WordCharacters' );
    print "WordCharacters for index 0 = $wordchars[0]\n";
    print "WordCharacters for index 1 = $wordchars[1]\n";


Valid values for HeaderParameterName are:

    WordCharacters
    BeginCharacters
    EndCharacters
    IgnoreFirstChar
    IgnoreLastChar
    Indexed on
    Description
    IndexPointer
    IndexAdmin
    Stemming
    Soundex

Note that this list may be incomplete.  Check the source code or the swish-e
discussion list for more info.

=item B<@stopwords = SwishStopWords( $handle, $indexfilename );>

Returns an array containing all the stopwords stored in the index file pointed by $indexfilename
where $indexfilename must match one of the names used in SwishOpen.

Example:
    @stopwords = SwishStopWords( $handle, $indexfilename );
    print 'Stopwords: ',
          join(', ', @stopwords),
          "\n";

=item B<@keywords = SwishWords( $handle, $indexfilename, $c);>

Returns an array containing all the keywords stored in the index file pointed by
$indexfilename ($indexfilename must match one of the names used in SwishOpen)
and starting with the character $c.

Example:
    my $letter = 't';
    @keywords = SwishWords( $handle, $indexfilename, $letter);

    print "List of keywords that start with the letter '$letter':\n",
          join("\n", @keywords),
          "\n";

=item B<$stemword=SwishStem( $word );>

Returns the stemmed word preserving the original one.

Example:
    my $stemword = SwishStem( 'parking' );
    print $stem_word;     # prints park

=back

=head1 SUPPORT

Questions about this module and SWISHE should be posted to the SWISHE mailing list.
See http://swish-e.org


=head1 AUTHOR

Jose Ruiz -- jmruiz@boe.es  (Documentation by Bill Moseley)


=head1 SEE ALSO

http://swish-e.org

SWISH, SWISH::Library at your local CPAN site.


=head1 Document Info

$Id: SWISHE.pm,v 1.9 2002/08/22 22:58:38 whmoseley Exp $


1	package SWISHE;
2
3	require Exporter;
4	require DynaLoader;
5
6	@ISA = qw(Exporter DynaLoader);
7
8	# Probably shouldn't export everything
9	@EXPORT = qw(
10	SwishOpen
11	SwishSearch
12	SwishClose
13	SwishNext
14	SwishSeek
15	SwishError
16	SwishHeaderParameter
17	SwishStopWords
18	SwishWords
19	SwishStem
20	SwishErrorString
21	SwishHeaders
22	SetLimitParameter
23	ClearLimitParameter
24	);
25
26	$VERSION = '0.02';
27
28	bootstrap SWISHE $VERSION;
29
30	# Preloaded methods go here.
31
32	# Autoload methods go after __END__, and are processed by the autosplit program.
33
34	1;
35	__END__
36
37	=head1 NAME
38
39	SWISH-PERL - Perl Interface to the SWISH-E Library
40
41	=head1 SYNOPSIS
42
43	use SWISHE;
44
45	my $indexfilename1 = '/path/to/index1.swish';
46	my $indexfilename2 = '/path/to/index2.swish';
47
48	# To search for several indexes just put them together
49	$indexfiles = "$indexfilename1 $indexfilename2";
50
51	my $handle = SwishOpen( $indexfiles )
52	or die "Failed to open '$indexfiles'";
53
54	# Get a few headers from the index files
55	my @headers = qw/WordCharacters BeginCharacters EndCharacters/;
56	for ( @headers ) {
57	my @h = SwishHeaderParameter( $handle, $_ );
58	print "$_ for index 0 is $h[0]\n",
59	"$_ for index 1 is $h[1]\n\n";
60	}
61
62
63	# Now search
64	@standard = ('Rank', 'File Name', 'Title', 'Document Size');
65	@props = qw/prop1 prop2 prop3/;
66
67	$props = join ' ', @props;
68	$sort = 'prop1 asc prop2 desc';
69	$query = 'meta1=metatest1';
70
71	my $num_results = SwishSearch($handle, $query, 1, $props, $sort);
72
73	if ( $num_results <= 0 ) {
74	print ($num_results ? SwishErrorString( $handle ) : 'No Results');
75
76	my $error = SwishError( $handle );
77	print "\nError number: $error\n" if $error;
78
79	return; # or next.
80	}
81
82	my %results; # place to store the return values by name
83
84	while( @results{ @standard, @props } = SwishNext( $handle )) {
85	print "\n";
86	printf("%20s -> '%s'\n", $_, $results{$_}) for @standard, @props;
87	}
88
89	# No more queries on these indexes
90	SwishClose( $handle );
91
92
93	=head1 ABSTRACT
94
95	Swish-e version 2.1.x creates an archive library of the internal Swish-e C functions.
96	This perl module provides access to those functions by embedding the Swish-e code in
97	your application. The benefits are faster searches (no need to fork/execute an external program)
98	and avoids commonly used unsafe system calls.
99
100	This module provides direct access to the Swish-e C library functions. For a higher level, object
101	oriented interface to SWISH visit http://search.cpan.org/search?mode=module&query=SWISH
102
103	=head2 Warnings and Gotchas
104
105	The Swish-e library was created from the executable program (instead of the other way around).
106	The Swish-e executable program handles errors by printing them to STDOUT and then aborting.
107	Not exactly desired behavior from a library.
108	This means you should carefully screen your input data before calling the library.
109
110	Another minor issue is that the Swish-e library currently includes all code -- both searching and
111	indexing code. Your program will be larger than needed due to this, but hopefully your OS will
112	be smart and share this code across processes and the impact will be minimal.
113
114	The library is often the last code to get checked during Swish-e development. Test
115	carefully, and watch for memory leaks if running in a persistent environment.
116
117
118	=head1 INSTALLATION
119
120	Before you can build the perl module you must build and install SWISH-E. Please read the
121	B<INSTALL> documentation included in the SWISHE distribution package.
122
123	perldoc INSTALL
124
125	After building the SWISHE executable and successfully running make test, you will need to install
126	the SWISHE archive library. This is done while in the top-level directory of the SWISHE distribution.
127
128	% su root
129	% make install-lib
130	% exit
131
132	This will install the archive library (F<libswish-e.a>) into /usr/local/lib by default.
133
134	Next, build the perl module.
135
136	% cd perl
137	% perl Makefile.PL
138	% make
139	% make test
140	% su root
141	% make install
142	% exit
143
144	If you do not have root access you can instead use
145
146	% perl Makefile.PL PREFIX=/path/to/my/local/perl/library
147
148	And then in your perl script:
149
150	use lib '/path/to/my/local/perl/library';
151
152
153	To test it you can run the test.pl script. Type "./test.pl" at your command prompt.
154	This perl script uses the index file built by the "make test" command used during the build
155	of SWISHE as described in the B<INSTALL> document.
156
157	B<NOTE> Currently swish will exit the running program on some fatal errors. In general
158	this should not happen, but it is something to think about if running under mod_perl
159	as an error will kill off the web server process. Apache, for example, should recover
160	by starting up a new child process. But, it's not a very graceful way to handle errors.
161
162	=head1 FUNCTION DESCRIPTIONS
163
164	The following describes the perl interface to the SWISHE C library.
165
166	=over 4
167
168	=item B<$handle = SwishOpen( $IndexFiles );>
169
170	Open one or more index files and returns a handle.
171
172	Examples:
173
174	$handle = SwishOpen( 'index_file.idx' );
175
176	# open two indexes
177	$handle = SwishOpen( 'index1.idx index2.idx' );
178
179	Returns undefined on an error, but the only errors are typically fatal, so
180	will most likely exit the running program.
181
182	If running under a persistent framework (such as mod_perl) you may run many queries
183	against the $handle. This results in much faster searches. For example, on searching
184	an index with 24,000 files and returning (the same) 400 results, opening and closing the index
185	for each search resulted in about 15 queries per second. Leaving the index open resulted
186	in about 150 queries per second.
187
188
189	=item B<SwishClose( $handle );>
190
191	Closes the handle returned by SwishOpen.
192	Closes all the opened files and frees the used memory.
193
194	=item B<$num_results = SwishSearch($handle, $search, $structure, $properties, $sortspec);>
195
196	Returns the number of hits, zero for no results, or a negative number.
197	If negative SwishErrorString( $handle ) will return the error message.
198
199	The values passed are:
200
201	=over 2
202
203	=item *
204
205	$handle is the handle returned by SwishOpen
206
207	=item *
208
209	$search is the search string.
210
211	Examples:
212	my $query = 'title="this a is phrase"';
213	my $query = '(title="this phrase") or (description=(these words))';
214
215	=item *
216
217	$structure is an integer value only applicable for an html search. It defines
218	where in an html search to look.
219	It can be IN_FILE, IN_TITLE, IN_HEAD, IN_BODY, IN_COMMENTS, IN_HEADER or IN_EMPHASIZED or
220	or'ed combinations of them (e.g.: IN_HEAD \| IN_BODY).
221	Use IN_FILE (a value of 1) if your documents are not html.
222	The numerical values for these constants are in src/swish.h
223
224	You can define these in your code with:
225
226	# Set bits
227	use constant IN_FILE => 1;
228	use constant IN_TITLE => 2;
229	use constant IN_HEAD => 4;
230
231	Not many people use the structure feature.
232
233	=item *
234
235	$properties is a string with the properties to be returned separated by spaces. Properties must
236	be defined during indexing. See B<README-SWISHE> for more information.
237
238	Example:
239
240	my $properties = 'subject description';
241
242	You may also use the swish internal properties:
243
244	my $properties = 'subject description swishrank swishlastmodified';
245
246
247	=item *
248
249	$sortspec is the sort spec if different from relevance.
250
251	Examples:
252	my $sortspec = '' # sort by relevance
253
254	# sort first in ascending order by title,
255	# then by other fields in descending order
256	my $sortspec = 'title asc category desc category desc';
257
258	=back
259
260	=item B<SetLimitParameter( $handle, $property, $low, $high )>
261
262	This experimental feature allows limiting results to a range of values
263	for a given property. For example, to limit Titles:
264
265	SetLimitParameter( $handle, 'swishtitle', 'a', 'zzzzzzzzzz' );
266
267	Limits titles to the range specified.
268
269	Note, if you do not call SwishOpen() and SwishClose() for every search
270	you must clear the limit selection with C<ClearLimitParameter>.
271
272
273	=item B<ClearLimitParameter( $handle )>
274
275	This function must be called before calling SetLimitParameter() again while
276	making repeated queries on an open index.
277
278	Once a limit is set on an open index, that limit stays in effect. If you want to
279	change the limit, or just remove the limit you must call ClearLimitParameter().
280	For example:
281
282	$handle = SwishOpen( $index );
283	while ( $query = next_query() ) {
284	ClearLimitParameter( $handle );
285
286	my @limits = $query->limits;
287	SetLimitParameter( $handle, @limits} ) if @limits;
288
289	SwishSearch($handle, $query->search, 1 );
290
291	...
292	}
293
294	SwishClose( $handle );
295
296	There's no harm in calling ClearLimitParameter() if a limit is not going to be used.
297
298	This behavior may change in the future.
299
300
301
302	=item B<SwishNext( $handle )>
303
304	($rank, $filename, $title, $size, @properties) = SwishNext( $handle );
305
306	This function returns the next hit of a search. Must be executed after SwishSearch to read the results.
307
308	=over 2
309
310	=item *
311
312	$rank - An integer from 1 to 1000 indicating the relevance of the result
313
314	=item *
315
316	$filename - The source filename
317
318	=item *
319
320	$title - The title as indexed (as found in the HTML E<lt>TITLEE<gt> section)
321
322	=item *
323
324	$size - The length of the source document
325
326	=item *
327
328	@properties - The list of properties returned for this result.
329
330	=back
331
332	See the SYNOPSIS above for an example.
333
334
335	=item B<$rc=SwishSeek($handle, $num);>
336
337	Repositions the pointer in the result list to the element pointed by num.
338	It is useful when you want to read only the results starting at $num (e.g. for showing
339	results one page at a time).
340
341	=item B<$error_number=SwishError($handle);>
342
343	Returns the last error if any (always a negative value).
344	If there is not an error it will return 0.
345
346	=item B<$error_string=SwishErrorString( $handle );>
347
348	Returns the error string for the number supplied.
349
350	print 'Error: ', SwishErrorString( $handle ), "\n";
351
352	=item B<@ParameterArray=SwishHeaderParameter($handle,$HeaderParameterName);>
353
354	This function is useful to access the header data of the index files
355	Returns the contents of the requested header parameter of all index files
356	opened by SwishOpen in an array.
357
358	Example:
359
360	@wordchars = SwishHeaderParameter( $handle, 'WordCharacters' );
361	print "WordCharacters for index 0 = $wordchars[0]\n";
362	print "WordCharacters for index 1 = $wordchars[1]\n";
363
364
365	Valid values for HeaderParameterName are:
366
367	WordCharacters
368	BeginCharacters
369	EndCharacters
370	IgnoreFirstChar
371	IgnoreLastChar
372	Indexed on
373	Description
374	IndexPointer
375	IndexAdmin
376	Stemming
377	Soundex
378
379	Note that this list may be incomplete. Check the source code or the swish-e
380	discussion list for more info.
381
382	=item B<@stopwords = SwishStopWords( $handle, $indexfilename );>
383
384	Returns an array containing all the stopwords stored in the index file pointed by $indexfilename
385	where $indexfilename must match one of the names used in SwishOpen.
386
387	Example:
388	@stopwords = SwishStopWords( $handle, $indexfilename );
389	print 'Stopwords: ',
390	join(', ', @stopwords),
391	"\n";
392
393	=item B<@keywords = SwishWords( $handle, $indexfilename, $c);>
394
395	Returns an array containing all the keywords stored in the index file pointed by
396	$indexfilename ($indexfilename must match one of the names used in SwishOpen)
397	and starting with the character $c.
398
399	Example:
400	my $letter = 't';
401	@keywords = SwishWords( $handle, $indexfilename, $letter);
402
403	print "List of keywords that start with the letter '$letter':\n",
404	join("\n", @keywords),
405	"\n";
406
407	=item B<$stemword=SwishStem( $word );>
408
409	Returns the stemmed word preserving the original one.
410
411	Example:
412	my $stemword = SwishStem( 'parking' );
413	print $stem_word; # prints park
414
415	=back
416
417	=head1 SUPPORT
418
419	Questions about this module and SWISHE should be posted to the SWISHE mailing list.
420	See http://swish-e.org
421
422
423	=head1 AUTHOR
424
425	Jose Ruiz -- jmruiz@boe.es (Documentation by Bill Moseley)
426
427
428	=head1 SEE ALSO
429
430	http://swish-e.org
431
432	SWISH, SWISH::Library at your local CPAN site.
433
434
435	=head1 Document Info
436
437	$Id: SWISHE.pm,v 1.9 2002/08/22 22:58:38 whmoseley Exp $
438
439
440
441