cunei-commits Mailing List for Cunei Machine Translation Platform (Page 31)

Revision: 76
          http://cunei.svn.sourceforge.net/cunei/?rev=76&view=rev
Author:   ralfbrown
Date:     2009-06-25 18:01:07 +0000 (Thu, 25 Jun 2009)

Log Message:
-----------
Tweaked hashCode() to hopefully get a minor speedup and slightly better distribution when there are many language models.  Added profiling code [PROFILE* variables and finalize()] to help determine whether it is worth updating a local variable when extending the hypothesis rather than computing the hash code from scratch each time it is requested.

Modified Paths:
--------------
    src/cunei/decode/HypothesisBuilder.java


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

Revision: 75
          http://cunei.svn.sourceforge.net/cunei/?rev=75&view=rev
Author:   aaronbphillips
Date:     2009-06-25 14:45:55 +0000 (Thu, 25 Jun 2009)

Log Message:
-----------
Annotations can now be indexed and loaded from input. Annotations are not yet being scored or analyzed during decoding.

Modified Paths:
--------------
    src/cunei/confusion/XMLConfusionReader.java
    src/cunei/corpus/AnnotationIndex.java
    src/cunei/corpus/MonolingualCorpus.java
    src/cunei/corpus/MultilingualCorpus.java
    src/cunei/document/Phrase.java
    src/cunei/type/Annotation.java
    src/cunei/type/AnnotationType.java
    src/cunei/type/DependentAnnotation.java
    src/cunei/type/SequenceType.java


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

Revision: 74
          http://cunei.svn.sourceforge.net/cunei/?rev=74&view=rev
Author:   ralfbrown
Date:     2009-06-25 14:25:25 +0000 (Thu, 25 Jun 2009)

Log Message:
-----------
Changed hashCode to reduce collisions between different matches in the same training example.

Modified Paths:
--------------
    src/cunei/corpus/Example.java


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

Revision: 73
          http://cunei.svn.sourceforge.net/cunei/?rev=73&view=rev
Author:   ralfbrown
Date:     2009-06-24 20:46:21 +0000 (Wed, 24 Jun 2009)

Log Message:
-----------
UNTESTED: switched CorpusSynonymFinder to take advantage of the fixed offsets between left and right context matches by simply probing the Set for the other context for existence of a specfic Example, making the synonym finding linear in the number of context matches and eliminating the additional storage used by the previous version.

Modified Paths:
--------------
    src/cunei/corpus/SequenceIndex.java
    src/cunei/synonym/CorpusSynonymFinder.java
    src/cunei/synonym/CorpusSynonymSerializer.java


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

Revision: 72
          http://cunei.svn.sourceforge.net/cunei/?rev=72&view=rev
Author:   ralfbrown
Date:     2009-06-24 19:53:41 +0000 (Wed, 24 Jun 2009)

Log Message:
-----------
UNTESTED: invoke CorpusSynonymSerializer.read to get synonym corpus in Decode, Evaluate, Optimizer, and Translate.  Updated CorpusSynonymSerializer to return null instead of an exception message when the corpus does not exist, since it is optional.

Modified Paths:
--------------
    src/cunei/cli/Decode.java
    src/cunei/cli/Evaluate.java
    src/cunei/cli/Translate.java
    src/cunei/decode/Decoder.java
    src/cunei/optimize/Optimizer.java
    src/cunei/synonym/CorpusSynonymSerializer.java


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

Revision: 71
          http://cunei.svn.sourceforge.net/cunei/?rev=71&view=rev
Author:   ralfbrown
Date:     2009-06-24 19:04:28 +0000 (Wed, 24 Jun 2009)

Log Message:
-----------
Fixed up calls to Translate.translate, added new class CorpusSynonymSerializer.

Modified Paths:
--------------
    src/cunei/cli/Decode.java
    src/cunei/cli/Evaluate.java
    src/cunei/optimize/Optimizer.java

Added Paths:
-----------
    src/cunei/synonym/CorpusSynonymSerializer.java


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

Revision: 70
          http://cunei.svn.sourceforge.net/cunei/?rev=70&view=rev
Author:   ralfbrown
Date:     2009-06-24 18:45:14 +0000 (Wed, 24 Jun 2009)

Log Message:
-----------
Stubbed in passing source-language corpus to Translator.

Modified Paths:
--------------
    src/cunei/cli/Translate.java


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

Revision: 69
          http://cunei.svn.sourceforge.net/cunei/?rev=69&view=rev
Author:   ralfbrown
Date:     2009-06-24 17:53:02 +0000 (Wed, 24 Jun 2009)

Log Message:
-----------
Added null checks on TypeSequence from Phrase and added x == RARE_OOV where there was an existing x == LEXICAL special case.

Modified Paths:
--------------
    src/cunei/corpus/Context.java
    src/cunei/corpus/MultilingualCorpus.java
    src/cunei/corpus/PanliteCorpusWriter.java
    src/cunei/decode/XMLDecoderWriter.java
    src/cunei/document/Phrase.java
    src/cunei/document/SimpleDocumentWriter.java
    src/cunei/synonym/CorpusSynonymBuilder.java
    src/cunei/synonym/CorpusSynonymFinder.java
    src/cunei/translate/Translator.java


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

Revision: 68
          http://cunei.svn.sourceforge.net/cunei/?rev=68&view=rev
Author:   ralfbrown
Date:     2009-06-24 17:20:05 +0000 (Wed, 24 Jun 2009)

Log Message:
-----------
UNTESTED: fixed up CorpusSynonymBuilder to place both original sequence and replacement sequence in the new confusion nodes.  Required storing original Phrase in CorpusSynonym.

Modified Paths:
--------------
    src/cunei/synonym/CorpusSynonym.java
    src/cunei/synonym/CorpusSynonymBuilder.java


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

Revision: 67
          http://cunei.svn.sourceforge.net/cunei/?rev=67&view=rev
Author:   ralfbrown
Date:     2009-06-24 17:11:54 +0000 (Wed, 24 Jun 2009)

Log Message:
-----------
UNTESTED: eliminated SuggestedTranslations from Phrase in favor of new sequence type; moved insertCorpusSynonyms from ConfusionNetwork into new class CorpusSynonymBuilder.

Modified Paths:
--------------
    src/cunei/confusion/ConfusionNetwork.java
    src/cunei/confusion/ConfusionNode.java
    src/cunei/document/Phrase.java
    src/cunei/translate/Translator.java
    src/cunei/type/SequenceType.java

Added Paths:
-----------
    src/cunei/synonym/CorpusSynonymBuilder.java


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

Revision: 66
          http://cunei.svn.sourceforge.net/cunei/?rev=66&view=rev
Author:   aaronbphillips
Date:     2009-06-24 14:57:32 +0000 (Wed, 24 Jun 2009)

Log Message:
-----------
Quick hack to NistDocumentReader to read references. Some more thought should go into this later.

Modified Paths:
--------------
    src/cunei/document/NISTDocumentReader.java


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

Revision: 64
          http://cunei.svn.sourceforge.net/cunei/?rev=64&view=rev
Author:   aaronbphillips
Date:     2009-06-24 14:05:20 +0000 (Wed, 24 Jun 2009)

Log Message:
-----------
Updated NistDocumentReader to handle empty segments

Modified Paths:
--------------
    src/cunei/document/NISTDocumentReader.java


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

Revision: 65
          http://cunei.svn.sourceforge.net/cunei/?rev=65&view=rev
Author:   ralfbrown
Date:     2009-06-24 14:07:21 +0000 (Wed, 24 Jun 2009)

Log Message:
-----------
UNTESTED: first cut at IndexMonolingualCorpus, based on the code for IndexCorpus and ProcessDocument.

Added Paths:
-----------
    src/cunei/cli/IndexMonolingualCorpus.java


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

Revision: 63
          http://cunei.svn.sourceforge.net/cunei/?rev=63&view=rev
Author:   aaronbphillips
Date:     2009-06-24 03:19:52 +0000 (Wed, 24 Jun 2009)

Log Message:
-----------
Initial preparation for Annotations. New classes Annotation and DependentAnnotatation that are stored in the Phrase. Moved CorpusIndex to SequenceIndex and added an AnnotationIndex class. Cleaned up recent changes to corpus reading code, verified it worked, and removed MonolingualCorpusReader (which was no longer needed). Added .settings directory with formatting preferences for Eclipse. Moving forward source code should be formatted automatically with these specs by Eclipse.

Modified Paths:
--------------
    launch/Decode.launch
    launch/Estimate Lexicon Alignment.launch
    launch/Index Corpus.launch
    launch/Translate.launch
    src/cunei/corpus/Context.java
    src/cunei/corpus/MonolingualCorpus.java
    src/cunei/corpus/MultiFileCorpusReader.java
    src/cunei/corpus/MultilingualCorpus.java
    src/cunei/corpus/SentencePair.java
    src/cunei/document/Phrase.java
    src/cunei/translate/Translator.java
    src/cunei/type/AnnotationType.java

Added Paths:
-----------
    .settings/
    .settings/org.eclipse.jdt.core.prefs
    .settings/org.eclipse.jdt.ui.prefs
    launch/Process Panlite Corpus.launch
    launch/Workbench.launch
    src/cunei/corpus/AnnotationIndex.java
    src/cunei/corpus/SequenceIndex.java
    src/cunei/corpus/SequenceIndexBuilder.java
    src/cunei/type/Annotation.java
    src/cunei/type/DependentAnnotation.java

Removed Paths:
-------------
    src/cunei/corpus/CorpusIndex.java
    src/cunei/corpus/CorpusIndexBuilder.java
    src/cunei/corpus/MonolingualCorpusReader.java
    src/cunei/corpus/StandoffCorpusReader.java


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

Revision: 62
          http://cunei.svn.sourceforge.net/cunei/?rev=62&view=rev
Author:   ralfbrown
Date:     2009-06-23 21:52:42 +0000 (Tue, 23 Jun 2009)

Log Message:
-----------
UNTESTED: Hooked insertCorpusSynonyms into Translator.load().

Modified Paths:
--------------
    src/cunei/confusion/ConfusionNetwork.java
    src/cunei/synonym/CorpusSynonymFinder.java
    src/cunei/translate/Translator.java


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

Revision: 61
          http://cunei.svn.sourceforge.net/cunei/?rev=61&view=rev
Author:   aaronbphillips
Date:     2009-06-23 20:27:35 +0000 (Tue, 23 Jun 2009)

Log Message:
-----------
Updated MonolingualCorpus to use a DocumentReader instead of the CorpusReader as there is no such thing as a SentencePair in a monolingual context

Modified Paths:
--------------
    src/cunei/corpus/MonolingualCorpus.java


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

Revision: 60
          http://cunei.svn.sourceforge.net/cunei/?rev=60&view=rev
Author:   ralfbrown
Date:     2009-06-23 20:18:48 +0000 (Tue, 23 Jun 2009)

Log Message:
-----------
UNTESTED: moved common code between MonolingualCorpusReader and MultiFileCorpusReader into new class StandoffCorpusReader.

Modified Paths:
--------------
    src/cunei/corpus/MonolingualCorpusReader.java
    src/cunei/corpus/MultiFileCorpusReader.java

Added Paths:
-----------
    src/cunei/corpus/StandoffCorpusReader.java


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

Revision: 59
          http://cunei.svn.sourceforge.net/cunei/?rev=59&view=rev
Author:   ralfbrown
Date:     2009-06-23 19:42:19 +0000 (Tue, 23 Jun 2009)

Log Message:
-----------
UNTESTED: changed around code in getPhrase / getSentencePair to properly pass back the read phrase(s).

Modified Paths:
--------------
    src/cunei/corpus/MonolingualCorpusReader.java
    src/cunei/corpus/MultiFileCorpusReader.java
    src/cunei/document/Phrase.java


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

Revision: 58
          http://cunei.svn.sourceforge.net/cunei/?rev=58&view=rev
Author:   ralfbrown
Date:     2009-06-23 19:07:24 +0000 (Tue, 23 Jun 2009)

Log Message:
-----------
UNTESTED: switched from Boolean args to throwing exceptions.

Modified Paths:
--------------
    src/cunei/corpus/MonolingualCorpusReader.java
    src/cunei/corpus/MultiFileCorpusReader.java


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

Revision: 57
          http://cunei.svn.sourceforge.net/cunei/?rev=57&view=rev
Author:   ralfbrown
Date:     2009-06-23 18:45:26 +0000 (Tue, 23 Jun 2009)

Log Message:
-----------
UNTESTED: changed boolean args to Boolean to make them reference args.

Modified Paths:
--------------
    src/cunei/corpus/MonolingualCorpusReader.java
    src/cunei/corpus/MultiFileCorpusReader.java


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

Revision: 56
          http://cunei.svn.sourceforge.net/cunei/?rev=56&view=rev
Author:   ralfbrown
Date:     2009-06-23 18:30:45 +0000 (Tue, 23 Jun 2009)

Log Message:
-----------
UNTESTED: forgot to initialize processor

Modified Paths:
--------------
    src/cunei/corpus/MonolingualCorpusReader.java


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

Revision: 55
          http://cunei.svn.sourceforge.net/cunei/?rev=55&view=rev
Author:   ralfbrown
Date:     2009-06-23 18:23:51 +0000 (Tue, 23 Jun 2009)

Log Message:
-----------
UNTESTED: Refactored MultiFileCorpusReader.next() into smaller functions.  Added class MonolingualCorpusReader.  Added backoff to smaller context window in ConfusionNetwork.insertCorpusSynonyms.

Modified Paths:
--------------
    src/cunei/confusion/ConfusionNetwork.java
    src/cunei/corpus/MultiFileCorpusReader.java
    src/cunei/corpus/SentencePair.java

Added Paths:
-----------
    src/cunei/confusion/ChangeLog
    src/cunei/corpus/MonolingualCorpusReader.java


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

Revision: 54
          http://cunei.svn.sourceforge.net/cunei/?rev=54&view=rev
Author:   ralfbrown
Date:     2009-06-23 15:44:21 +0000 (Tue, 23 Jun 2009)

Log Message:
-----------
UNTESTED: Added feature Model.Weights.Substitution, whose value is proportional to the frequency of the substituting word in the wanted context.

Modified Paths:
--------------
    src/cunei/confusion/ConfusionNetwork.java
    src/cunei/synonym/CorpusSynonymFinder.java


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

Revision: 53
          http://cunei.svn.sourceforge.net/cunei/?rev=53&view=rev
Author:   ralfbrown
Date:     2009-06-23 15:04:27 +0000 (Tue, 23 Jun 2009)

Log Message:
-----------
UNTESTED: switched CorpusSynonymFinder from linear scan to log-n check for validating that a left-context match has a corresponding right-context match.  Added config parameters CorpusSynonym.Lookup.{MinFreq,MaxSynonyms}.         

Modified Paths:
--------------
    src/cunei/confusion/ConfusionNetwork.java
    src/cunei/synonym/CorpusSynonymFinder.java

Added Paths:
-----------
    src/cunei/synonym/ExampleComparator.java


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.

Revision: 52
          http://cunei.svn.sourceforge.net/cunei/?rev=52&view=rev
Author:   aaronbphillips
Date:     2009-06-23 14:05:08 +0000 (Tue, 23 Jun 2009)

Log Message:
-----------
Print parameters at beginning of optimization output (so looking at an old log file is sufficient to re-create the experiment)

Modified Paths:
--------------
    src/cunei/cli/CommandLineInterface.java
    src/cunei/cli/Optimize.java


This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.