DWDS/Dialing Concordance Code

+ tagged ddc-2.1.22

v2.1.22: bug-fixes for new prefix- and suffix-search introduced in v2.1.21 (mantis #33072 [bams|zeit])
* speed improvements too
- prefix searches are 7-26% faster than ddc-v2.1.18
- suffix search performance gains from v2.1.21 remain; especially for highly specific suffixes:
+ *haus : ca. 32x faster (S=32.07)
+ *erde : ca. 74x faster (S=74.49)

+ tagged ddc-2.1.21

+ ddc-2.1.21 release

+ tagged ddc-2.1.21-rc1

v2.1.21-rc1:
* added suffix-query optimization via new index files CORPUS._suffix_TOKATTR
- new low-level evaluation method CStringIndexSet::QueryTokenListWithRightTruncation()
- new suffix-sorted index CIndexSetForQueryingStage::m_rIndex
- can result in drastically improved suffix query-times compared to old regex-based vocabulary scan:
+ *en : 24% faster (Amdahl S=1.24)
+ *chen : 169% faster (Amdahl S=3.69)
+ *ber : 432% faster (Amdahl S=5.32)
* added ddc_(index|union|split) code to create CORPUS._suffix_TOKATTR for all indices at build-time
- very small memory and disk footprint (<1% total index size, growth O(NTypes))
* suffix-indices are still optional: if not present, old regex-based vocabulary scan will be used

+ suffix-index support for ddc_(index|union|split)

+ suffix index: documented and cleaned up testing code
+ TODO: ddc_(index|union|split) integration

+ suffix-index tests

+ first stab at auxiliary suffix-index
- no ddc_index-support yet, test build with perl script scripts/ddc-make-suffix-index.perl
+ can be up to 7x faster for small-ish sets of matched types (e.g. *ber)
+ only ca. 20% improvement for large match-sets (e.g. *en)
- todo: optimize in-place suffix-comparison struct, avoid temporaries