[go: up one dir, main page]

Menu

Commit [r1511]  Maximize  Restore  History

* added new method CStringIndexator::IndexTokenFixLongColumns() for fallback repair of "greedy" tokens at indexing time

+ implicitly called by CStringIndexator::IndexOneToken() for overlong lines if new bool param tryFixErrors is true
+ tryFixErrors param is populated from CConcIndexator::m_bResumeOnIndexErrors on IndexOneToken() calls
+ truncates only "greedy" columns (those using more than MaxLen/nCols bytes)
+ truncates each "greedy" column indepdently, proportionate to its "greediness"
(the number of bytes by which its length exceeds MaxLen/nCols)
+ emits console warnings about truncation of line and each truncated column when invoked

mukau 2020-07-10

changed /ddc/trunk/Changes
changed /ddc/trunk/src/CommonLib/Makefile.am
changed /ddc/trunk/src/ConcordLib/ConcIndexator.cpp
changed /ddc/trunk/src/ConcordLib/IndexMorphXml.cpp
changed /ddc/trunk/src/ConcordLib/StringIndexator.cpp
changed /ddc/trunk/src/ConcordLib/StringIndexator.h
changed /ddc/trunk/src/ConcordLib/TabFormatIndexator.cpp