scidb-users Mailing List for Scidb

> Thank you for your very detailed and thoughtful answer to my previous
> question.

I have even forgotten another important fact:

i) The STL guarantees the complexity in time, but it does in general not 
guarantee the complexity in space. Example: some implementations for STL are 
using a hybrid for std::string: small strings until about 20 bytes will be 
copied into a buffer, and larger strings will be reference counted. Any object 
which contains such a kind of std::string will be expanded by more than 30 
bytes per instance. If you have many empty strings, or many instances with 
larger strings, every instance is wasting about 20 bytes. With many strings 
this may have a significant impact on memory behavior. With MSTL I know the 
complexity in time, and I know the complexity in space, and I'm not dependent 
upon the "black box" behavior of the STL.

> - There is a minor glitch with the editor dialog where tcl throws some
> errors.

An error box should be shown with the message "not yet implemented". Should be 
fixed in r1142.

> - In the list of games the eco codes are wrong. For the game I opened it
> shows B85 although the actual eco code in the board window seems correct
> (E62). This might be a glitch in the cbh codec.

I still have a problem in this codec. If the source of the games is ChessBase, 
the ECO code will be decoded correctly, but if the source of the games is not 
ChessBase, the decoding of the ECO code does not work properly. I couldn't 
figure this out yet.

> I will do more testing as I am actively using chess databases and engines
> for my preparation anyway. And I am getting sick of chessbase because it is
> full of bugs. I really appreciate the scidb codebase.

The current version of Scidb is still dedicated for smaller databases, until 
about 1.5 million of games. With larger databases the opening browser is too 
slow, it makes no fun. But with next big release I will integrate a faster 
algorithm, I'm working since about 3 years on it, and I hope that the opening 
tree will become fast even with databases containing 10 million games, or 
more.

Gregor

Hi Gregor,

Thank you for your very detailed and thoughtful answer to my previous
question. 

In the mean time I was able to run ScidB on windows 10 using the linux
subsystem (actually an Ubuntu 16.04 under the hood) and the free VcXsrv X
server for Windows. Everything was compiled from the svn source. Here are
some results: http://imgur.com/a/OO7Eo
- There is a minor glitch with the editor dialog where tcl throws some
errors. Probably it's just because I am not running it on a real linux
system.
- In the list of games the eco codes are wrong. For the game I opened it
shows B85 although the actual eco code in the board window seems correct
(E62). This might be a glitch in the cbh codec.

I will do more testing as I am actively using chess databases and engines
for my preparation anyway. And I am getting sick of chessbase because it is
full of bugs. I really appreciate the scidb codebase.

Best regards,
Bogdan


-----Original Message-----
From: Gregor Cramer [mailto:re...@gm...] 
Sent: Friday, April 7, 2017 15:11
To: sci...@li...
Cc: Bogdan Burlacu <fro...@gm...>
Subject: Re: [Scidb-users] Some questions about Scidb

Hello Bogdan,

thanks for your interest for Scidb. About your questions:

> 1)      It seems that your MSTL overlaps at least partially with the STL.
Is
> there a reason why you chose to implement things from zero? I 
> understand Scidb uses some very specialized data structures and
algorithms.

There are some reasons:

a) Speed. One example: stl::string is a fine thing, but it has to work in
every application under any conditions. So a string assignment either has to
copy the whole string content, or to manage reference count, and the latter
is in general not more efficient than string copy, under general conditions
reference counting is a bit complex. Scidb has his own conditions. A string
always belongs to a database, so the life time of the string is coupled with
the life time of the associated database. So mstl::string does not need to
copy strings, or to use reference counting, it is only copying the pointer
of the string, this is super-fast. Such a method in crashing under general
conditions, but not under the conditions of Scidb.

b) Speed. The mstl classes are implemented with special optimization
strategies, based on very special type traits. STL does not provide this
(and cannot provide such special application dependent things).

c) Library size. One example: sorting inside STL will be performed with
std::sort, and the use of std::sort is producing many instances of the same
sort algorithm for different class types. This is a big drawback of STL. In
MSTL I'm using wrappers to qsort in stdlib.h. This is reducing the library
size significantly.

d) Implementation details. One example: stl::map in general is using a red-
black-tree, but Scidb's map is using a sorted vector (binary search). In
general the approach of Scidb has a bad performance (slow insert), but not
under the special conditions of Scidb, here the sorted vector is better.
(And if I really need fast inserts, then I use the rbtree class of MSTL.)

e) The MSTL classes are using the special exception handling of Scidb, which
provides a backtrace. This is not possible with the use of STL, here the
backtrace cannot be provided. (This would require that every function/method
has to catch the STL exception.)

f) MSTL contains some special basic classes, not available in STL (and I
don't like to use the boost library for similar reasons).

g) Scidb is using many specialized classes: specialized (file) stream
classes, specialized bit sets, and more. The STL has too few classes which
are of interest (for use in Scidb).

h) The implementation of a specialized STL is not a big thing, compared to
the overall development time of Scidb this is insignificant.

> 2)      Is it possible to decouple database functionality from the GUI,
and
> use it as a library? Eg., to use custom GUIs or scripts (it would be 
> easy with CQL). Would the library have any chance of being portable? 
> What kind of work would it require to make it portable?

It is decoupled, the GUI interface is separated.The Android app "CBH to PGN"

is using the Scidb library. But the GUI interface is designed only for the
Tk library, I don't have a toolkit independent framework.

Scidb is designed to be portable to Windows and Mac OS X. But as long as
Scidb has not reached the alpha state (stable state) the porting will not be
finished (but is mostly already implemented). Of course porting is a big
task, it requires to test the whole application on the ported platform.

> 3)      Will Scidb include some any command line utilities for
manipulating
> databases? Eg copying games from one database to another, converting 
> between formats, this kind of stuff.

Scidb already has the command line utilities cbh2si4 (converting CBH to
si4), and cdb2sci (converting any supported database format to sci). More
applications are not yet demanded.

Cheers,
Gregor

I've checked in revision 1140. I've overworked the game editor control, now it 
provides the maximal performance. With my example game (many variations with 
long lines), I could reduce the loading time of this game from about 15 
minutes to about 4 seconds with the integration of the revised text widget. 
With the overworked game editor control it has been reduced to about 1 second. 
The scrolling behavior is fast and smooth, with old text widget (from Tk 
library) scrolling was nearly impossible after loading this game.

Slowly I'm coming to an end with the implementation of the fast opening 
position search algorithm, so I will concentrate my work onto the next big 
release: fast opening position search, position (ECO) browser, game links, new 
database format, Tip of the Day dialog.

Gregor

Hello Bogdan,

thanks for your interest for Scidb. About your questions:

> 1)      It seems that your MSTL overlaps at least partially with the STL. Is
> there a reason why you chose to implement things from zero? I understand
> Scidb uses some very specialized data structures and algorithms.

There are some reasons:

a) Speed. One example: stl::string is a fine thing, but it has to work in every 
application under any conditions. So a string assignment either has to copy 
the whole string content, or to manage reference count, and the latter is in 
general not more efficient than string copy, under general conditions reference 
counting is a bit complex. Scidb has his own conditions. A string always 
belongs to a database, so the life time of the string is coupled with the life 
time of the associated database. So mstl::string does not need to copy 
strings, or to use reference counting, it is only copying the pointer of the 
string, this is super-fast. Such a method in crashing under general 
conditions, but not under the conditions of Scidb.

b) Speed. The mstl classes are implemented with special optimization 
strategies, based on very special type traits. STL does not provide this (and 
cannot provide such special application dependent things).

c) Library size. One example: sorting inside STL will be performed with 
std::sort, and the use of std::sort is producing many instances of the same 
sort algorithm for different class types. This is a big drawback of STL. In 
MSTL I'm using wrappers to qsort in stdlib.h. This is reducing the library 
size significantly.

d) Implementation details. One example: stl::map in general is using a red-
black-tree, but Scidb's map is using a sorted vector (binary search). In 
general the approach of Scidb has a bad performance (slow insert), but not 
under the special conditions of Scidb, here the sorted vector is better. (And 
if I really need fast inserts, then I use the rbtree class of MSTL.)

e) The MSTL classes are using the special exception handling of Scidb, which 
provides a backtrace. This is not possible with the use of STL, here the 
backtrace cannot be provided. (This would require that every function/method 
has to catch the STL exception.)

f) MSTL contains some special basic classes, not available in STL (and I don't 
like to use the boost library for similar reasons).

g) Scidb is using many specialized classes: specialized (file) stream classes, 
specialized bit sets, and more. The STL has too few classes which are of 
interest (for use in Scidb).

h) The implementation of a specialized STL is not a big thing, compared to the 
overall development time of Scidb this is insignificant.

> 2)      Is it possible to decouple database functionality from the GUI, and
> use it as a library? Eg., to use custom GUIs or scripts (it would be easy
> with CQL). Would the library have any chance of being portable? What kind of
> work would it require to make it portable?

It is decoupled, the GUI interface is separated.The Android app "CBH to PGN" 
is using the Scidb library. But the GUI interface is designed only for the Tk 
library, I don't have a toolkit independent framework.

Scidb is designed to be portable to Windows and Mac OS X. But as long as Scidb 
has not reached the alpha state (stable state) the porting will not be finished 
(but is mostly already implemented). Of course porting is a big task, it 
requires to test the whole application on the ported platform.

> 3)      Will Scidb include some any command line utilities for manipulating
> databases? Eg copying games from one database to another, converting between
> formats, this kind of stuff.

Scidb already has the command line utilities cbh2si4 (converting CBH to si4), 
and cdb2sci (converting any supported database format to sci). More 
applications are not yet demanded.

Cheers,
Gregor

Hello,

 

First of all thanks for this great piece of software. Although I am
currently using Chessbase 14 (as I use Windows 10 most of the time), I
always check the progress of Scidb and try it in my VM. As a fellow
developer (and FM chess player), I have some questions:

 

1)      It seems that your MSTL overlaps at least partially with the STL. Is
there a reason why you chose to implement things from zero? I understand
Scidb uses some very specialized data structures and algorithms.

2)      Is it possible to decouple database functionality from the GUI, and
use it as a library? Eg., to use custom GUIs or scripts (it would be easy
with CQL). Would the library have any chance of being portable? What kind of
work would it require to make it portable? 

3)      Will Scidb include some any command line utilities for manipulating
databases? Eg copying games from one database to another, converting between
formats, this kind of stuff. 

 

Best regards,

Bogdan