lemur-announce Mailing List for The Lemur Project

This Lemur project release brings Indri 5.5, Galago 3.4  and
RankLib 2.1 updates as well as the standalone Krovetz stemmer package,
implemented in both Java and C++ (numbered 3.4).


Applications compiled with the Indri API require the following
libraries: z, iberty, pthread, and m on linux. Applications built in
Visual Studio require the additional library wsock32.lib.  The java jar
files were built with Java 6 (jdk 1.6.0). The java UIs require Java
6. We have tested using GCC 4.1.2 (CentOS 5.3 linux), 4.3.3 (Ubuntu
10.04 linux), 4.2.1 (OS/X), and Visual Studio 2008 (Windows Vista, WIN32
and x86_64).

See https://sourceforge.net/projects/lemur/files/lemur/indri-5.5/ for the full release notes.

--
David Fisher
df...@cs...

This Lemur project release brings Indri 5.4, Galago 3.3 updates and the first release of RankLib (see https://sourceforge.net/p/lemur/wiki/RankLib/), a learning to rank system.

Applications compiled with the Indri API require the following
libraries: z, iberty, pthread, and m on linux. Applications built in
Visual Studio require the additional library wsock32.lib.  The java jar
files were built with Java 6 (jdk 1.6.0). The java UIs require Java
6. We have tested using GCC 4.1.2 (CentOS 5.3 linux), 4.3.3 (Ubuntu
10.04 linux), 4.2.1 (OS/X), and Visual Studio 2008 (Windows Vista, WIN32
and x86_64).

Bugs Fixed (see https://sourceforge.net/p/lemur/bugs/ for the complete tickets):

BUG# 

198	collection length overflow
197	DataPoint parsing text error in its constructor
175	Parameters - Null pointer exceptions
183	Out of Memory Errors in parsing do not result in failure
185	Searcher not thread safe
190	Null exceptions when building an empty index 
191	dump functions fail for empty index files	 
192	ExtentRestrictionNode leaks	 
193	handleSearch: java.lang.NullPointerException
194	running in drmaa mode -> ExecutionException 
196	QueryEnvironment::~QueryEnvironment calls close() producing double delete on memory

Feature Requests (see https://sourceforge.net/p/lemur/feature-requests/ for the complete tickets):

FR# 
42	WarcRecord.java should allow access to WARC-Date et al
48	Parameter tuning	 
49	Cached Retrieval 
51	Lengths Iterators 
54	Extent Attribute Indexes	 
63	Universal Parser 
64	parallel app execution

--
David Fisher
df...@cs...

This Lemur project release brings Indri 5.3, Galago updates and QueryLogToolbar updates.

Applications compiled with the Indri API require the following
libraries: z, iberty, pthread, and m on linux. Applications built in
Visual Studio require the additional library wsock32.lib.  The java jar
files were built with Java 6 (jdk 1.6.0). The java UIs require Java
6. We have tested using GCC 4.1.2 (CentOS 5.3 linux), 4.3.3 (Ubuntu
10.04 linux), 4.2.1 (OS/X), and Visual Studio 2008 (Windows Vista, WIN32
and x86_64).

See https://sourceforge.net/projects/lemur/files/lemur/indri-5.3/ for the full release notes.

--
David Fisher
df...@cs...

This Lemur project release brings Indri 5.2, Galago updates and its
first binary release.

Enhancements:

Query log toolbar: Make the toolbar work with versions of Firefox above
4.x and install up to 8.x.

Galago:

The galago source distribution is available from mercurial on
SourceForge at http://lemur.hg.sourceforge.net/hgweb/lemur/galago/. The initial binary release, the pie release, marks the transition to the fully sourceforge hosted version 3 of galago.

--
David Fisher
df...@cs...

This Lemur project release brings Indri 5.1, improvements to the Lemur
CGI, Galago updates, ClueWeb09 derived data and utility updates, and bug fixes throughout.

Available at: https://sourceforge.net/projects/lemur/

--
David Fisher
df...@cs...

The 5.0 release of Indri marks major changes in the packaging of the Lemur Project software. All of the deprecations that were previously announced have been performed, with the Lemur Toolkit distribution retired. Users of the deprecated APIs SHOULD NOT upgrade from the Lemur Toolkit 4.12 to Indri 5.0. All of the lemur::api API elements have been removed.

Applications compiled with the Indri API require the following libraries: z, iberty, pthread, and m on linux. Applications built in Visual Studio require the additional library wsock32.lib.  The java jar files were built with Java 6 (jdk 1.6.0). The java UIs require Java 6. We have tested using GCC 4.1.2 (CentOS 5.3 linux), 4.3.3 (Ubuntu 10.04 linux), 4.2.1 (OS/X),  and Visual Studio 2008 (Windows Vista, WIN32 and x86_64). 

Please see https://sourceforge.net/projects/lemur/files/lemur/indri-5.0/ for the full release notes and file downloads.

--
David Fisher
df...@cs...

4.12 corrects various issues in the 4.11 distribution package, provides
improvements to indri, improvements to the Lemur CGI, improvements to
the query log toolbars and server, and more.

Applications compiled with the Lemur Toolkit require the following
libraries: z, iberty, pthread, and m on linux. Applications built in
Visual Studio require the additional library wsock32.lib.  The java jar
files were built with Java 6 (jdk 1.6.0). The java UIs require Java
6. We have tested using GCC 3.4.6 (CentOS 4.2 linux x86_64), 4.1.2
(CentOS 5.3 linux), 4.3.3 (Ubuntu 10.04 linux), 4.2.1 (OS/X), Visual
Studio 2005 (Windows XP), and Visual Studio 2008 (Windows Vista, WIN32
and x86_64).


--
David Fisher
df...@cs...

4.11 corrects various issues in the 4.10 distribution package, provides
improvements to indri for processing ClueWeb09, improvements to the
query log toolbars and server, PHP5 indri API wrappers, support for
Visual Studio 2005 and 2008, and more.

Applications compiled with the Lemur Toolkit require the following
libraries: z, iberty, pthread, and m on linux, and additionally socket
and nsl on solaris.  Applications built in Visual Studio require the
additional library wsock32.lib.  The java jar files were built with
Java 6 (jdk 1.6.0). The java UIs require Java 6. We have tested using GCC
3.4.6 (solaris), 3.4.6 (CentOS 4.2 linux x86_64), 4.1.2 (CentOS 5.3
linux), 4.3.3 (Ubuntu 9.10 linux), 4.2.1 (OS/X), VC++ .NET 7.1(Windows
XP), Visual Studio 2005 (Windows XP), and Visual Studio 2008 (Windows
Vista, WIN32 and x86_64).

Enhancements:

PHP5 wrappers replace the PHP4 wrappers from the previous versions. Note
that it is now necessary to include the file libindri_php.php, which is
installed in $prefix/share/lemur, to get the PHP class definitions. The
site-search component version, in $prefix/share/lemur/php/include is a
modified version that uses the value of $indri_param['library'],
specified in config.php, to determine the name of the shared library to
load. Users of the PHP wrappers will need to upgrade from php 4 to php
5.

Turn off automatic addition of the field named document. The field was
added for FR# 1989805. Users who desire a global document parent element
should specify a field named document in their indexing parameters.

Expose the IndexEnvironment::setStoreDocs method to the C# and Java APIS.

Add Visual Studio 2005 project files.

Add Visual Studio 2008 project files. Note that Visual Studio 2008 is
the supported version for Windows compilation.

Open files in "rb" mode to ensure windows compatibility.

Add Visual Studio 64-bit target (x86_64).

Windows 64-bit installer, enabling use of the 64-bit Java runtime.

Add the ability to perform baseline experiments using tfidf, tfidf with
PRF, and okapi (no PRF). This will allow for ClueWeb scale baselines.

Remove test for antlr include, as Ubuntu installs have the header, but
the lib is not in the right place.

Add style to the list of excluded tags for html documents indexed with indri.

Better tracking of scrollbar movement in Firefox querylog toolbar.

--
David Fisher
df...@cs...

4.10 corrects various issues in the 4.9 distribution package, provides  
improvements to indri for processing ClueWeb09, improvements to the  
query log toolbars and server, and more.

Applications compiled with the Lemur Toolkit require the following
libraries: z, iberty, pthread, and m on linux, and additionally socket
and nsl on solaris.  Applications built in Visual Studio require the
additional library wsock32.lib.  The java jar files were built with
Java 5 (jdk 1.5.0). The java UIs require Java 5. We have tested using  
GCC
3.2 (solaris), 3.2.2(linux), 3.4(linux), 3.4.3(linux x86_64),  
4.0.2(linux),
4.3.1 (OS/X), VC++ .NET 7.1(Windows XP), and Visual Studio 2005 (Windows
XP).

Enhancements:

For ClueWeb09 processing, input files for the warc file class  
environment  can now be in gzipped form. The unsegmented languages can  
now be processed with the warcchar file class environment, which  
indexes each individual character as a separate token.

The query log toolbar upload server responds more quickly to uploads,  
processing the files after the upload success has been returned to the  
client. The IE version of the toolbar will now check for updates. The  
IE version of the toolbar can now operate correctly in IE 8 on Vista  
running in protected mode.


--
David Fisher
df...@cs...

4.9 corrects various issues in the 4.8 distribution package, provides  
a new
FileClassEnvironment for WARC file input, various indexing speed
optimizations for indri; and more.

Applications compiled with the Lemur Toolkit require the following
libraries: z, iberty, pthread, and m on linux, and additionally socket
and nsl on solaris.  Applications built in Visual Studio require the
additional library wsock32.lib.  The java jar files were built with
Java 5 (jdk 1.5.0). The java UIs require Java 5. We have tested using  
GCC
3.2 (solaris), 3.2.2(linux), 3.4(linux), 3.4.3(linux x86_64),  
4.0.2(linux),
4.3.1 (OS/X), VC++ .NET 7.1(Windows XP), and Visual Studio 2005 (Windows
XP).

Enhancements:

The LayoutManager constraints have been modified to improve the resize
behavior of the components.

The Query Log Toolbar and server support the automatic uploading of log
files on a scheduled basis. This preference can be set by the user of  
the
toolbar to completely automatic, automatic with confirmation required
before upload, or manual upload only.

A new FileClassEnvironment, warc, has been added to indri. This  
environment
enables indexing of the ClueWeb09 corpus,
http://boston.lti.cs.cmu.eduData/clueweb09/

Indri indexing speed optimizations, providing 5-15% speedup for GOV2 (25
million documents) sized collections.

--
David Fisher
df...@cs...

A full listing of the current bug fixes and enhancements are in the
release notes.

4.8 corrects various issues in the 4.7 distribution package, adds a GUI
for the ireval package; an application, ModifyFields, to add or remove
field data from an indri Repository without reindexing, improvements to
the Query Log Toolbar; and more.

Applications compiled with the Lemur Toolkit require the following
libraries: z, iberty, pthread, and m on linux, and additionally socket
and nsl on solaris.  Applications built in Visual Studio require the
additional library wsock32.lib.  The java jar files were built with
Java 5 (jdk 1.5.0). The java UIs require Java 5. We have tested using  
GCC 3.2 (solaris), 3.2.2(linux), 3.4(linux), 3.4.3(linux x86_64),  
4.0.2(linux), 4.3.1 (OS/X), VC++ .NET 7.1(Windows XP), and Visual  
Studio 2005 (Windows XP).

--
David Fisher
df...@cs...

A full listing of the current bug fixes and enhancements are in the  
release notes.

4.7 corrects various issues in the 4.6 distribution package, adds
a relevance judgment UI to the Lemur Retrieval UI; a java-based
trec_eval alternative;
a PageRank (TM http://www.google.com/technology/)
application; a performance-enhanced harvestlinks; a Firefox
query log toolbar; a SOAP server for indri repositories; and
more.

Applications compiled with the Lemur Toolkit require the
following libraries: z, iberty, pthread, and m on linux, and
additionally socket and nsl on solaris.  Applications built in
Visual Studio require the additional library wsock32.lib.  The
java jar files were built with Java 5 (jdk 1.5.0). The java UIs
require Java 5. We have tested using GCC 3.2 (solaris),
3.2.2(linux), 3.4(linux), 3.4.3(linux x86_64), 4.0.2(linux),
VC++ .NET 7.1(Windows XP), and Visual Studio 2005 (Windows XP).

--
David Fisher
df...@cs...

For details about new functionality, API changes, and bug fixes, see  
the release notes for 4.6 at http://sourceforge.net/project/ 
shownotes.php?group_id=161383&release_id=563255

4.6 INCLUDES CHANGES TO THE KEYED FILE BTREE PACKAGE THAT ARE NOT
BACKWARDS COMPATIBLE WITH PREVIOUS VERSIONS. ALL KeyfileIncIndexes AND
indri INDEXES WILL NEED TO BE REBUILT.

4.6 corrects various issues in the 4.5 distribution package, adds
   updates to the Java GUIS; indri query language query evaluation
   speedups; an updated version of the heritrix web crawler; passage
   level pseudo-relevance feedback and more.

Applications compiled with the Lemur Toolkit require the following
libraries: z, iberty, pthread, and m on linux, and additionally socket
and nsl on solaris.  Applications built in Visual Studio require the
additional library wsock32.lib.  The java jar files were built with
Java 5 (jdk 1.5.0). We have tested using GCC 3.2 (solaris),
3.2.2(linux), 3.4(linux), 3.4.3(linux x86_64), 4.0.2(linux), VC++ .NET
7.1(Windows XP), and Visual Studio 2005 (Windows XP).

Visit https://sourceforge.net/projects/lemur to download the latest
release. Downloads for this and future releases will be hosted
exclusively on SourceForge. Older releases will remain available in
the download archives at http://www.lemurproject.org/download- 
archive.php

--
The Lemur Team

Due to an oversight, the configure script in the Lemur Toolkit source  
distribution in lemur-4.5.tar.gz does not properly set the command  
line defines necessary for compilation with large file support on  
some 32-bit platforms. This can cause failures when writing a file  
that is larger than 2GB. Only the source distribution is affected.

We have replaced lemur-4.5.tar.gz on SourceForge with  
lemur-4.5.0.tar.gz.

If you have already downloaded lemur-4.5.tar.gz, please visit
https://sourceforge.net/projects/lemur to download the replacement  
source tree.

We apologize for any inconvenience.

-- 
The Lemur Team

For details about new functionality, API changes, and bug fixes see the
release notes for 4.5 at http://www.lemurproject.org/lemur/releasenotes.php

Key changes include:

4.5 includes changes to the indri Repository structure that are not
backwards compatible with previous versions. All indri indexes will need
to be rebuilt. 

4.5 corrects various issues in the 4.4.x distribution package, adds a
date field annotator for encoding dates as numeric fields, document
deletion and index compaction for indri Repositories, efficiency
improvements to the OffsetAnnotationAnnotator, improvements to the CGI,
and other features described below. 

Applications compiled with the Lemur Toolkit require the following
libraries: z, iberty, pthread, and m on linux, and additionally socket
and nsl on solaris.  Applications built in Visual Studio require the
additional library wsock32.lib.  The java jar files were built with
Java 5 (jdk 1.5.0). We have tested using GCC 3.2 (solaris),
3.2.2(linux), 3.4(linux), 3.4.3(linux x86_64), 4.0.2(linux), VC++ .NET
7.1(Windows XP), and Visual Studio 2005 (Windows XP).

Visit https://sourceforge.net/projects/lemur to download the latest
release. Downloads for this and future releases will be hosted
exclusively on SourceForge. Older releases will remain available in
the download archives at http://www.lemurproject.org/download-archive.php

-- 
The Lemur Team

For details about new functionality, API changes, and bug fixes see the
release notes for 4.4 at http://www.lemurproject.org/lemur/ 
releasenotes.php

Key changes include:

Substantial improvements to the Lemur Project website at http:// 
www.lemurproject.org.

4.4 includes changes to the keyed file package that are not backwards  
compatible with previous versions. All indexes will need to be rebuilt.

4.4 removes support for the deprecated index classes, InvIndex and  
InvFPIndex. These should be replaced with a KeyfileIncIndex or an  
indri index.

4.4 corrects various issues in the 4.3.x distribution package, adds a  
wildcard operator to the Indri Query Language, a snippet builder for  
indri retrieval results and other features. See the release notes for  
details.


Visit http://sourceforge.net/projects/lemur to download the latest  
release. Downloads for this and future releases will be hosted  
exclusively on SourceForge. Older releases will remain available in  
the download archives at http://www.lemurproject.org/download- 
archive.php.

--
The Lemur Team

With the upcoming Lemur Toolkit tutorial session at SIG/IR 2006, we have 
decided to issue the release of Lemur version 4.3.2 and Indri 2.3.2 to 
the general public.
Over the past month, we have corrected over sixteen problems, added the 
ability to reference child, parent, and ancestor extents, and made the 
summary results from the CGI cleaner.

A full listing of the current bug fixes and enahancements can be in the 
release notes at:
http://www.lemurproject.org/lemur/releasenotes.html

The new distributions are available for download at:
http://www.lemurproject.org/download.html

or via sourceforge:
http://sourceforge.net/projects/lemur

- also - with the initial release of version 4.3, we have been 
transitioning to use the services on SourceForge.net. This mailing list 
(lem...@cs...) will be going away in the very near future. To 
ensure you keep receiving Lemur Toolkit related news and updates, please 
sign up on the lemur-announce mailing list at:
https://lists.sourceforge.net/lists/listinfo/lemur-announce

The Lemur Team

With the upcoming Lemur Toolkit tutorial session at SIG/IR 2006, we have 
decided to issue the release of Lemur version 4.3.2 and Indri 2.3.2 to 
the general public.
Over the past month, we have corrected over sixteen problems, added the 
ability to reference child, parent, and ancestor extents, and made the 
summary results from the CGI cleaner.

A full listing of the current bug fixes and enahancements can be in the 
release notes at:
http://www.lemurproject.org/lemur/releasenotes.html

The new distributions are available for download at:
http://www.lemurproject.org/download.html

or via sourceforge:
http://sourceforge.net/projects/lemur

The Lemur Team

Due to a packaging error, two out of date files were included in the Lemur
toolkit 4.3 distribution. One of these files is also included in the Indri
2.3 distribution. To correct this, we have prepared a new distribution. We
have also included one bug fix for the Indri query language parser.

The changes are:

Problem: The file site-search/crawl-index.in does not copy the file
excluded_hosts to the crawl directory, causing a crawl to fail. (Lemur 4.3,
Indri 2.3)
Solution: add excluded_hosts to the files to be copied.

Problem: The file site-search/cgi/Makefile contains a typo the causes make
install to fail. (Lemur 4.3)
Solution: Change REAME to README in the install target.

Problem: Indri query parser fails to parse #date operators correctly. (Lemur
4.3, Indri 2.3)
Solution: Update grammar rule for #date operators.

We apologize for any inconvenience the packaging errors may have caused.

The new distributions are available for download via
http://www.lemurproject.org/download.html 

The Lemur Team

The Lemur Toolkit is transitioning from lemurproject.org to SourceForge.net
for various project services. Visit http://sourceforge.net/projects/lemur to
see project's pages. The first item we will transition is this announcements
mailing list. 

Please go to https://lists.sourceforge.net/lists/listinfo/lemur-announce to
subscribe to the new mailing list in order to continue receiving Lemur
Toolkit announcements.

Subsequently we will transition the discussion forum, CVS for the Lemur
Toolkit and Indri, and the tracking system for bugs, feature requests, and
support.

Lemur toolkit version 4.3 indri indexes are not backwards compatible
with older versions. All collections using indri indexes must be
reindexed.

Indri version 2.3 is not backwards compatible with older
versions. All collections must be reindexed.

All custom applications built against the Lemur Toolkit or Indri
libraries will need to be recompiled.

The indri Java wrappers have been refactored to facilitate integrating with
the new Lemur Toolkit Java wrappers. The package edu.umass.cs.indri has been
renamed lemurproject.indri. The package edu.umass.cs.indri.ui has been
renamed lemurproject.indri.ui. Users of the indri Java wrappers will need to
refactor their code accordingly.

For details about new functionality, API changes, and bug fixes see the
release notes for 4.3 at
http://www.lemurproject.org/lemur/releasenotes.html.

Key new features include:

Improved documentation and tutorials.

Out of the box site search, complete with web crawler, cgi, and PHP UI. See
http://www.lemurproject.org/lemur/SiteSearch.html for details.

Improved JNI support for using the Lemur toolkit with Java applications. The
indri API Java wrappers that were previously only available in the indri
standalone distribution are now available within the Lemur Toolkit.

C# wrapper support for using the Lemur toolkit and indri with C#
applications.

PHP wrapper support for using indri with PHP applications.

Java retrieval and indexing GUIs are now part of the main Lemur Toolkit
distribution. The retrieval GUI has been rewritten to use the new Java
wrappers.

The Indri applications harvestlinks, dumpindex, and makeprior have been
added to the Lemur toolkit distribution.

And numerous bug fixes.

Go to the download page at http://www.lemurproject.org/download.html to
download the latest release.

The Lemur Team