dbacl - digramic Bayesian classifier Git

348 lines (346 with data), 12.8 kB

\" t
.TH MAILTOE 1 "Bayesian Text Classification Tools" "Version @VERSION@" ""
.SH NAME
mailtoe \- a train-on-error simulator for use with dbacl.
.SH SYNOPSIS
.HP
.B mailtoe 
.I command 
[
.I command_arguments 
]
.SH DESCRIPTION
.PP
.B mailtoe
automates the task of testing email filtering and classification
programs such as 
.BR dbacl (1).
Given a set of categorized documents, mailtoe initiates test runs 
to estimate the classification errors and thereby permit fine tuning 
of the parameters of the classifier. 
.PP
Train-on-error (TOE) is a learning method which is sometimes advocated for 
email classifiers. Given an incoming email stream, the method consists
in reusing a fixed set of category databases until the first misclassification
occurs. At that point, the offending email is used to relearn the relevant 
category, until the next misclassification. In this way, categories are
only updated when errors occur. This directly models the way that some email
classifiers are used in practice.
.PP
TOE's error rates depend directly on the order in which emails are seen.
A small change in ordering, as might happen due to networking delays,
can have a large impact on the number of misclassifications. 
Consequently, 
.B mailtoe
does not give meaningful results, unless the sample emails are chosen carefully.
However, as this method is commonly used by spam filters, it is still worth 
computing to foster comparisons. Other methods  (see
.BR mailcross (1), mailfoot (1))
attempt to capture the behaviour of classification errors in other ways.
.PP
To improve and stabilize the error rate calculation,
.B mailtoe
performs the TOE simulations several times on slightly reordered email streams, and 
averages the results. The reorderings occur by multiplexing the emails from each
category mailbox in random order. Thus if there are three categories, the first email
classified is chosen randomly from the front of the sample email streams of each type. 
The second email is also chosen randomly among the three types, from the front of the
 streams after the first email was removed. Simulation stops when all sample streams 
are exhausted.
.PP
.B mailtoe
uses the environment variable MAILTOE_FILTER when
executing, which permits the simulation of arbitrary filters, provided
these satisfy the compatibility conditions stated in the  
ENVIRONMENT section below.
.PP
For convenience, 
.B mailtoe
implements a 
.B testsuite 
framework with predefined wrappers for several open
source classifiers. This permits the direct comparison of 
.BR dbacl (1) 
with competing classifiers on the same set of email samples. See the USAGE section below.
.PP
During preparation, 
.B mailtoe
builds a subdirectory named mailtoe.d in the current working directory. 
All needed calculations are performed inside this subdirectory.
.SH EXIT STATUS
.B mailtoe
returns 0 on success, 1 if a problem occurred.
.SH COMMANDS
.PP
.PP
.IP "\fBprepare\fR \fIsize\fR"
Prepares a subdirectory named mailtoe.d in the current working directory, and
populates it with empty subdirectories for exactly 
.I size
subsets.
.IP "\fBadd\fR \fIcategory\fR [ \fIFILE\fR ]..."
Takes a set of emails from either \fIFILE\fR if specified, or STDIN, and 
associates them with 
.IR category .
The ordering of emails within \fIFILE\fR is preserved, and subsequent \fIFILE\fRs are appended
to the first in each category.
This command can be repeated several times, 
but should be executed at least once.
.IP "\fBclean\fR"
Deletes the directory mailtoe.d and all its contents.
.IP "\fBrun\fR"
Multiplexes randomly from the email streams added earlier, and relearns
categories only when a misclassification occurs. The simulation is repeated
.I size
times.
.IP "\fBsummarize\fR"
Prints average error rates for the simulations.
.IP "\fBplot\fR [ \fIps\fR | \fIlogscale\fR ]..."
Plots the number of errors over simulation time. The "ps" option, if present,
writes the plot to a postscript file in the directory mailtoe/plots, instead of 
being shown on-screen. The "logscale" option, if present, causes the plot to
be on the log scale for both ordinates.
.IP "\fBreview\fR \fItruecat\fR \fIpredcat\fR"
Scans the last run statistics and extracts all the messages which belong to category
.I truecat
but have been classified into category
.IR predcat .
The extracted messages are copied to the directory 
.I mailtoe.d/review 
for perusal.
.PP
.IP "\fBtestsuite list\fR"
Shows a list of available filters/wrapper scripts which can 
be selected.
.IP "\fBtestsuite select\fR [ \fIFILTER\fR ]..."
Prepares the filter(s) named 
.I FILTER
to be used for simulation. The filter name is the name of 
a wrapper script located in the directory 
.IR @PKGDATADIR@/testsuite .
Each filter has a rigid interface documented below, and the act of selecting
it copies it to the 
.I mailtoe.d/filters 
directory. Only filters located there
are used in the simulations.
.IP "\fBtestsuite deselect\fR [ \fIFILTER\fR ]..."
Removes the named filter(s) from the directory
.I mailtoe.d/filters
so that they are not used in the simulation.
.IP "\fBtestsuite run\fR [ \fIplots\fR ]" 
Invokes every selected filter on the datasets added previously, and 
calculates misclassification rates. If the "plots" option is present,
each filter simulation is plotted as a postscript file in the directory 
.IR mailtoe.d/plots .
.IP "\fBtestsuite status\fR"
Describes the scheduled simulations.
.IP "\fBtestsuite summarize\fR"
Shows the cross validation results for all filters. Only makes sense
after the 
.I run 
command.
.SH USAGE
.PP
The normal usage pattern is the following: first, you should separate your email
collection into several categories (manually or otherwise). Each category should
be associated with one or more folders, but each folder should not contain 
more than one category. Next, you should decide how many runs to use, say 10. 
The more runs you use, the better the predicted error rates. However, more runs take more time. 
Now you can type
.HP
.na
% mailtoe prepare 10
.ad
.PP
Next, for every category, you must add every folder associated with this
category. Suppose you have three categories named 
.IR spam , 
.IR work , 
and 
.IR play ,
which are associated with the mbox files 
.IR spam.mbox , 
.IR work.mbox , 
and 
.IR play.mbox 
respectively. You would type
.PP
.na
% mailtoe add spam spam.mbox
.br
% mailtoe add work work.mbox
.br
% mailtoe add play play.mbox
.ad
.PP
You should aim for a similar number of emails in each category, as the random 
multiplexing will be unbalanced otherwise. The ordering of the email messages
in each 
.I *.mbox
file is important, and is preserved during each simulation. If you repeatedly
add to the same category, the later mailboxes will be appended to the first, preserving
the implied ordering. 
.PP
You can now perform as many TOE simulations as desired. The multiplexed emails
are classified and learned one at a time, by executing the command given in the 
environment variable MAILTOE_FILTER. If not set, a default value is used. 
.PP
.na
% mailtoe run
.br
% mailtoe summarize
.ad
.PP
The testsuite commands are designed to simplify the above steps and allow comparison
of a wide range of email classifiers, including but not limited to 
.BR dbacl .
Classifiers are supported through wrapper scripts, which are located in the 
.I @PKGDATADIR@/testsuite 
directory. 
.PP
The first stage when using the testsuite is deciding which classifiers to compare.
You can view a list of available wrappers by typing:
.PP
.na
% mailtoe testsuite list
.ad
.PP
Note that the wrapper scripts are NOT the actual email classifiers, which must 
be installed separately by your system administrator or otherwise.
Once this is done, you can select one or more wrappers for the simulation
by typing, for example:
.PP
.na 
% mailtoe testsuite select dbaclA ifile
.ad
.PP
If some of the selected classifiers cannot be found on the system, they
are not selected. Note also that some wrappers
can have hard-coded category names, e.g. if the classifier only supports binary
classification. Heed the warning messages. 
.PP
It remains only to run the simulation. Beware, this can take a long time 
(several hours depending on the classifier). 
.PP
.na
% mailtoe testsuite run
.br
% mailtoe testsuite summarize
.ad
.PP
Once you are all done, you can delete the working files, log
files etc. by typing
.PP
.na
% mailtoe clean
.ad
.SH SCRIPT INTERFACE
.PP
.B mailtoe testsuite 
takes care of learning and classifying your prepared email corpora for each
selected classifier. Since classifiers have widely varying interfaces, this
is only possible by wrapping those interfaces individually into a standard 
form which can be used by 
.BR "mailtoe testsuite" .
.PP
Each wrapper script is a command line tool which accepts a single command 
followed by zero or more optional arguments, in the standard form:
.PP
.na
wrapper command [argument]...
.ad
.PP
Each wrapper script also makes use of STDIN and STDOUT in a well defined 
way. If no behaviour is described, then no output or input should be used.
The possible commands are described below:
.IP filter
In this case, a single email is expected on STDIN, and a list of 
category filenames is expected in $2, $3, etc. The script writes the 
category name corresponding to the input email on STDOUT. No trailing newline
is required or expected.
.IP learn
In this case, a standard mbox stream is expected on STDIN, while a
suitable category file name is expected in $2. No output is written to
STDOUT.
.IP clean
In this case, a directory is expected in $2, which is examined for old
database information. If any old databases are found, they are purged or
reset. No output is written to STDOUT.
.IP describe
IN this case, a single line of text is written to STDOUT, describing the filter's
functionality. The line should be kept short to prevent line wrapping on a terminal.
.IP bootstrap
In this case, a directory is expected in $2. The wrapper script first checks for
the existence of its associated classifier, and other prerequisites. If the 
check is successful, then the wrapper is cloned into the supplied directory.
A courtesy notification should be given on STDOUT to express success or failure.
It is also permissible to give longer descriptions caveats.
.IP toe
In this case, a list of categories is expected in $3, $4, etc. Every possible
category must be listed. Preceding this list, the true category is given in $2.
.IP foot
Used by 
.BR mailfoot (1).
.SH ENVIRONMENT
.PP
Right after loading, 
.B mailtoe 
reads the hidden file .mailtoerc in the $HOME directory, if it exists, so
this would be a good place to define custom values for environment variables.
.IP MAILTOE_FILTER
This variable contains a shell command to be executed repeatedly
during the running stage.
The command should accept an email message on STDIN and output a
resulting category name. On the command line, it should also accept
first the true category name, then a list of all possible category
file names.  If the output category does not match the true category,
then the relevant categories are assumed to have been silently
updated/relearned.
If MAILTOE_FILTER is undefined, 
.B mailtoe
uses a default value.
.IP TEMPDIR
This directory is exported for the benefit of wrapper scripts. Scripts which
need to create temporary files should place them a the location given in TEMPDIR.
.SH NOTES
.PP
The subdirectory mailtoe.d can grow quite large. It 
contains a full copy of the training corpora, as well as learning files for 
.I size 
times all the added categories, and various log files. 
.PP
While TOE simulations for 
.BR dbacl (1)
can be used to compare with other classifiers, TOE should not be used
for real world classifications. This is because, unlike many other filters,
.BR dbacl (1)
learns evidence weights in a nonlinear way, and does not preserve relative
weights between tokens, even if those tokens aren't seen in new emails.
.SH WARNING
.PP
Because the ordering of emails within the added mailboxes matters, the estimated 
error rates are not well defined or even meaningful in an objective sense. 
However, if the sample emails represent an actual snapshot of a user's incoming email,
then the error rates are somewhat meaningful. The simulations can then be interpreted 
as alternate realities where a given classifier would have intercepted the incoming mail.  
.SH SOURCE
.PP
The source code for the latest version of this program is available at the
following locations: 
.PP
.na
http://www.lbreyer.com/gpl.html
.br
http://dbacl.sourceforge.net
.ad
.SH AUTHOR
.PP
Laird A. Breyer <laird@lbreyer.com>
.SH SEE ALSO
.PP
.BR bayesol (1)
.BR dbacl (1), 
.BR mailinspect (1),
.BR mailcross (1),
.BR mailfoot (1),
.BR regex (7)