scidb-users Mailing List for Scidb

--
No, I've meant, as soon as the library is finished.
--

I did not fully undestand your last message, but, anyway, I would like to implement the API in java-python-somethingelse, but after having seen the API (and if they convince me! :) )

>> I've planned to provide a C/C++ library, would be nice if some
>> volunteers will do the "translation" into other languages.
>> I cannot do all.

> So you have already done the language definition
> (a DTD I suppose), and the API ? Can I see them?

No, I've meant, as soon as the library is finished.

--

Gregor wrote

You don't see the final result, this will be exposed with the first
final version. Please keep in mind that Scidb is definitively not
ChessBase, nor is it Scid, although it's a chess database, of course.
--

I would say it is a Chess DBMS (Database Management System), but anyway what you want to say?
It has sense to a new format and "launch" it in a full working environment like Scidb, but in my opinion would be better do it completely standalone, to testify it is thinked for general chess data sharing.

--
Yes, that's a good formulation, a "semi-strcutured data (xml)",
that's what I like to do. A full-structured XML is not of
interest for this purpose.
--

That was not what I mean. XML is defined as "semi-structured" language in itself (SQL is a full structured one).
http://en.wikipedia.org/wiki/Semi-structured_data

--
I've planned to provide a C/C++ library, would be nice if some
volunteers will do the "translation" into other languages.
I cannot do all.
--

So you have already done the language definition (a DTD I suppose), and the API ? Can I see them?

> OK I see your point. It seems to me just scidb is getting a bit
> overloaded with this.

You don't see the final result, this will be exposed with the first
final version. Please keep in mind that Scidb is definitively not
ChessBase, nor is it Scid, although it's a chess database, of course.

> Anyway my opinion is semi-structured data (xml) is the best way
> to do it.

Yes, that's a good formulation, a "semi-strcutured data (xml)",
that's what I like to do. A full-structured XML is not of
interest for this purpose. A full structured XML is appropriate
for Web browser content, for example.

> Would be nice opening a new open project defining the new format
> and provide API and implementation for parsing in a lot of languages.

I've planned to provide a C/C++ library, would be nice if some
volunteers will do the "translation" into other languages.
I cannot do all.

>Gregor wrote:

>.sci is the very private format of Scidb, especially designed for
compactness, speed, and it contains very special data for >the acceleration of position search, and the game classification. In
fact .sci is readable for everyone, it's an open format, >but the parser for .sci is a bit difficult, and the special data for
acceleration and classification is useless for other >applications.
.sci is not useful for data interchange. Also .scg (game data) is
not useful for game interchange, .scg is very >compact and diffcult to parse. 


>I'm speaking about a format which is especially useful for data
interchange. Scidb has features not supported by PGN. The >new
format will provide an easy and clean interface for other
applications for the import of games without any loss of
>information. With the new format I will provide a small library
(a new project in Sourceforge) for import and export of game
>archives. With a simple and clean format the import/export will
also be much faster than with PGN. Parsing PGN is very >difficult
and slow.

OK I see your point. It seems to me just scidb is getting a bit overloaded with this.
Anyway my opinion is semi-structured data (xml) is the best way to do it.

Would be nice opening a new open project defining the new format and provide API and implementation for parsing in a lot of languages.

-------------------------------------------------------------------
Chris wrote:
-------------------------------------------------------------------
> But it would still be possible to export to PGN and import PGN games,
> though right? I'm thinking TWIC, and the human friendly aspects of PGN.

Of course, Chris, PGN support will still live with full functionality,
the new format will not be a replacement.

-------------------------------------------------------------------
Drew wrote:
-------------------------------------------------------------------
> Computer Algebraic Notation: where
>
> CAN = e1e2
> SAN = Ke2
> LAN = Ke1e2
>
> there are lots more it seems: MAN, MRAN, CRAN, etc

Ok, I've found a reference for these terms:
http://en.wikipedia.org/wiki/Chess_notation.
In Scidb the notation "e1e2" is called "Algebraic Notation" (AN),
this is the most common term, another common term is
"Coordinate Notation".

CAN is a very interesting alternative for the new format, it's
easier to parse, and the parser will be faster. LAN is not
needed in this case, LAN is of interest if someone want do
display a game without doing any further computation (computing
the piece of the origin field).

-------------------------------------------------------------------
Pietro wrote:
-------------------------------------------------------------------
> OK, still I do not understand what is different with .sci format.
> Or are you talking exactly of that?

.sci is the very private format of Scidb, especially designed for
compactness, speed, and it contains very special data for the
acceleration of position search, and the game classification. In
fact .sci is readable for everyone, it's an open format, but the
parser for .sci is a bit difficult, and the special data for
acceleration and classification is useless for other applications.
.sci is not useful for data interchange. Also .scg (game data) is
not useful for game interchange, .scg is very compact and diffcult
to parse.

I'm speaking about a format which is especially useful for data
interchange. Scidb has features not supported by PGN. The new
format will provide an easy and clean interface for other
applications for the import of games without any loss of
information. With the new format I will provide a small library
(a new project in Sourceforge) for import and export of game
archives. With a simple and clean format the import/export will
also be much faster than with PGN. Parsing PGN is very difficult
and slow.

On Tue, Mar 25, 2014 at 07:00:42PM +0000, Pietro wrote:

[A lot of stuff, but tl;dr] -- (too long, didn't read.)

You should turn off digest mode. Also, *please* don't post in HTML.

-- 
"If you're not careful, the newspapers will have you hating the people
who are being oppressed, and loving the people who are doing the 
oppressing." --- Malcolm X

On Tue, Mar 25, 2014 at 06:04:49PM +0100, Gregor Cramer wrote:
> Due to the fact that PGN (http://en.wikipedia.org/wiki/Portable_Game_Notation)
> is not sufficient for modern chess applications, I'm working on a
> different format, especially for the backup of unsaved games.

But it would still be possible to export to PGN and import PGN games,
though right? I'm thinking TWIC, and the human friendly aspects of PGN.

A separate converter program *could* be an option, but adds an extra step
a human has to take.

-- 
"If you're not careful, the newspapers will have you hating the people
who are being oppressed, and loving the people who are doing the 
oppressing." --- Malcolm X

On Tue, 25 Mar 2014 22:50:48 +0100
"Gregor Cramer" <re...@gm...> wrote:

> > For instance just say that the file is UTF-8 & uses
> > CAN and most of your problems go away.
> 
> What is CAN, I never heard this before.

Computer Algebraic Notation: where

 CAN = e1e2
 SAN = Ke2
 LAN = Ke1e2

there are lots more it seems: MAN, MRAN, CRAN, etc

-- 
Drew

>Gregor wrote:

>I cannot make this format a new standard. It will become a standard
>if many other applications will support this format, otherwise
>it will not become a standard. It's not a problem for me if
>someone else is offering a new standard which solves the problem
>with the interchangeability, but currently I cannot find any other
>tangible proposal.

OK, still I do not understand what is different with .sci format. Or are you talking exactly of that?

>>2. Interchanging data between chess applications. Of course this is
>>   possible only if this format will be widespread accepted. PGN
>>   cannot be used for this purpose, no way to import an exported
>>   game from ChessBase via PGN, PGN doesn't know anything about
>>   multilingual comments and different comment types like
>>   post-comments and pre-comments. 

> Yes, but the problem is exactly this: how do you make your new format a "standard"?

I cannot make this format a new standard. It will become a standard
if many other applications will support this format, otherwise
it will not become a standard. It's not a problem for me if
someone else is offering a new standard which solves the problem
with the interchangeability, but currently I cannot find any other
tangible proposal.

Thanks for your dream list, Fulvio. Possibly one day such a
format will exists. An open project has to do this, it seems
that the leader ChessBase is not interested in open standards.

Currently of special interest for me is:

> 4) a format where moves include both uci and san information: uci because 
> making a move should simply be 
> board[to_square]= board[from_square]; 
> board[captured_square]= empty; 
> board[from_square]=empty;

I agree, this is very comfortable, and my concept for the
binary version will take this into account. I'm not yet
sure about the textual version, because any other notation
than SAN will increase the file size significantly, and
the file size is still the big problem with textual
chess game encodings.

-------------------------------------------------------------------
Gregor wrote: -------------------------------------------------------------------
>2. Interchanging data between chess applications. Of course this is
>   possible only if this format will be widespread accepted. PGN
>   cannot be used for this purpose, no way to import an exported
>   game from ChessBase via PGN, PGN doesn't know anything about
>   multilingual comments and different comment types like
>   post-comments and pre-comments. 
Yes, but the problem is exactly this: how do you make your new format a "standard"?

> For instance just say that the file is UTF-8 & uses
> CAN and most of your problems go away.

What is CAN, I never heard this before.

> However, suppose we separate the movelist from the commentary
> ...
> 2. the commentary - which is fundamentally sequential - can be stored in
> an extended type of HTML perhaps. Provided it is possible to identify the
> actual move from text & variations, rendering of the commentary could
> almost be managed by CSS. If the moves were in divs (or spans) with unique
> Ids corresponding to move-ply then it is maybe just plain HTML

In fact this is too complicated for my purpose, you are thinking
one step ahead, but the only thing I want is to interchange/store
data without any loss in a simple way, and PGN is not providing
this. In my opinion you are proposing a format for more elaborated
purposes, online rendering for example. But the intended format
should not be expected to provide more than this store &
interchange capability. For other purposes other formats will be
invented. Due to my experience it is not a problem to have a special
format for each special purpose, in this way you can keep each format
simple and clean. If a format is simple and clean, writing a parser
in general is not more than a finger exercise (indeed it's always a
bit time consuming, especially testing). 

Gregor

>See <http://talkchess.com/forum/viewtopic.php?
topic_view=threads&p=562958&t=51666>
>about the problems with PGN. 

A very interesting thread.

My list of dreams:
1) a format that support online synchronization (git like: an online 
repository, some people have write access, other read-only access, only new or 
changed games are downloaded by the clients)

2) a format that can be stored efficiently on standard SQL/NoSql databases and 
accessed with html5 clients (pc, tablet, phone, etc.)

3) a standard hash that identify a specific game: for example the first 4 
letter of white name + 4 letter for black name + date + round. For example i 
would like to send an email with a link newchessdb.com?search?
gamehash=carlkram20130613r1 and the htm5 will connect to the SQL/NoSql database 
and show the game.

4) a format where moves include both uci and san information: uci because 
making a move should simply be 
board[to_square]= board[from_square]; 
board[captured_square]= empty; 
board[from_square]=empty;
and san because i don't want to calculate after every move if the king is in 
check or if there is another piece that can do the same move.

5) keep the amazing small size of the scid 1byte move format.... 
after all, this is a dream list :-)

Hi Gregor

On Tue, 25 Mar 2014 20:46:40 +0100
"Gregor Cramer" <re...@gm...> wrote:

> > Indeed what is your problem with PGN
> 
> See
> <http://talkchess.com/forum/viewtopic.php?topic_view=threads&p=562958&t=51666>
> about the problems with PGN. And I have forgotten one more important
> argument: support of multilingual comments, PGN doesn't know anything
> about other languages than English.

I don't think most of these are real problems though, more conventions
than anything else. For instance just say that the file is UTF-8 & uses
CAN and most of your problems go away.

> Another problem with PGN is the complexity of parsing, have a look

Yep parsing pgn IS ugly

But the main issue with any of these formats - XML or PGN or whatever is
the processing of comments in the movelist - and lots of extra stuff like
time controls (used to be a major ChessBase complaint) and the items you
list; multi-lingual comments is a really good example!

Even overloading an XML tag with lots of attributes or even
child-tags doesn't really solve the problem where comments are nested
inside one another but it does catch more or less everything else I would
suspect. 

However, suppose we separate the movelist from the commentary so:

1. the movelist has nothing but moves - in CAN lets say

2. the commentary - which is fundamentally sequential - can be stored in
an extended type of HTML perhaps. Provided it is possible to identify the
actual move from text & variations, rendering of the commentary could
almost be managed by CSS. If the moves were in divs (or spans) with unique
Ids corresponding to move-ply then it is maybe just plain HTML

2a. multi-lingual could just be the commentary repeated but with a
language attribute.

2b. Informator symbols and such could be regular entities

Hmmm, what do you think?

Apart from comments, game data is highly structured.
and Comments are not structured but a stream

Maybe that could be the key
-- 
Drew Ferguson

Please strip the replies, please do not send mails
with auto-appended content. The auto-appended
content is confusing, it is hard too see where
the reply stops, and browsing the user list
http://sourceforge.net/p/scidb/mailman/scidb-users/
is annoying with very long mails.

Thanks, Gregor

Thanks Drew, for your links.

-------------------------------------------------------------------
Drew wrote:
-------------------------------------------------------------------
> Personally I favour a simple INI-file style i.e. key-value pairs and
> sections; terser even than PGN

Yes, this is another idea, and probably I will go this way. But
using XML syntax has the advantage that I can let the XML
parser doing the syntactical stuff, except reading the game
moves of course. So my first try is to use XML, I like to have
a format as simple as possible, and XML is providing this.

> In general XML is very verbose compared to something as terse as PGN.

1. I don't think so if I don't use <move...> markups.

2. It is planned to invent an additional binary format, based on
   the text format, with a very simple and fast en/decoding. I've
   already have a concept for this.

> Indeed what is your problem with PGN

See <http://talkchess.com/forum/viewtopic.php?topic_view=threads&p=562958&t=51666>
about the problems with PGN. And I have forgotten one more important
argument: support of multilingual comments, PGN doesn't know anything
about other languages than English.

Another problem with PGN is the complexity of parsing, have a look
into Scidb's implementation of PGN file parsing:
<http://sourceforge.net/p/scidb/code/HEAD/tree/trunk/src/db/db_pgn_reader.cpp>.
This is a catastrophe, PGN has grown uncontrolled because of
deficiencies. The new format will be very simple to parse.

> especially if it is only for your own purposes.

Probably some other applications will adopt the new format as
an additional option, I'm not the only one who needs a more
powerful format than PGN.

> http://www.cybercom.net/~zbrad/Chess/pgnxml/

This project is using the syntax
<move n="nn" state="xx" annotation="xx" promote="x" from="xx" to="xx" NAG="nn" display="xx">
for game moves. Although this is strong XML, it's too much, the
file will explode. Even zipped PGN files sometimes will be very
large. But the idea how to handle the tags seems to be a good model,
for example:
<White Elo="nn" USCF="nn" NA="xx" Country="xx" Type="xx" Title="xx">

> http://www.saremba.de/chessgml/

This project is abandoned, and the syntax will cause exploding file
sizes.

> http://xml.coverpages.org/chessML.html

This project is abandoned, and all links are broken on this page.

-------------------------------------------------------------------
Pietro wrote:
-------------------------------------------------------------------
> I had some experience with xml and java package for parsing it,
> but I do not understand why you need another (apart of .pgn. ,
> .sci., cbh, .si4) format.

This new format will serve two purposes:

1. Backup of single games in a human readable format. PGN is not
   supporting the features of Scidb.

2. Interchanging data between chess applications. Of course this is
   possible only if this format will be widespread accepted. PGN
   cannot be used for this purpose, no way to import an exported
   game from ChessBase via PGN, PGN doesn't know anything about
   multilingual comments and different comment types like
   post-comments and pre-comments.

Gregor

I had some experience with xml and java package for parsing it, but I do not understand why you need another (a part of .pgn. , .sci., cbh, .si4) format.





Il Martedì 25 Marzo 2014 18:10, "sci...@li..." <sci...@li...> ha scritto:
 
Send Scidb-users mailing list submissions to
    sci...@li...

To subscribe or unsubscribe via the World Wide Web, visit
    https://lists.sourceforge.net/lists/listinfo/scidb-users
or, via email, send a message with subject or body 'help' to
    sci...@li...

You can reach the person managing the list at
    sci...@li...

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Scidb-users digest..."


Today's Topics:

   1. Re: Scidb-users Digest, Vol 20, Issue 1 (Pietro)
   2. Re: Schedule of development (Gregor Cramer)
   3. Alternative to PGN (Gregor Cramer)
   4. Re: Alternative to PGN (Drew Ferguson)


----------------------------------------------------------------------

Message: 1
Date: Fri, 21 Mar 2014 21:25:21 +0000 (GMT)
From: Pietro <ilc...@ya...>
Subject: Re: [Scidb-users] Scidb-users Digest, Vol 20, Issue 1
To: "sci...@li..."
    <sci...@li...>
Message-ID:
    <139...@we...>
Content-Type: text/plain; charset="iso-8859-1"

Nice schedule!

My opinion: 9, 10, 11, 14, 18, 19 and 21 are not of primary importance (almost 7 months in your estimate).

Also 22 is not so necessary :)

Is ticket #33 definitely up to Jessy?


Bye






Message: 1
Date: Wed, 19 Mar 2014 18:55:46 +0100
From: "Gregor Cramer" <re...@gm...>
Subject: [Scidb-users] Schedule of development
To: sci...@li...
Message-ID:
??? <trinity-e387000b-3c0a-49dd-a71a-d1702fef198f-1395251745889@3capp-gmx-bs30>
??? 
Content-Type: text/plain; charset=UTF-8

A rough overview about the development schedule, and the
estimated time how long implementation and testing will
still take. Please note that below are only the primary
functions, there are still more secondary functions.
And please consider that it is hard to estimate the
development time, the real development time may differ
significantly (in both directions).

1. New database format, allowing faster search, supports
?  opening classification, smaller index file, less memory
?  consumption, and eventually smaller game files,
?  currently in test phase (hopefully successful, this is
?  all based on a transposition-invariant classification
?  key, completely different from Scid's technology, and I
?  had to overwork the emulation of Scid databases).

2. Game links, already implemented, not yet tested,
?  about 2 weeks.

3. Opening classification tab (already partly implemented),
?  this includes the overwork of the ECO list, the current
?  list is not integrally closed, about 1 month.

4. Flexible window layout, about 3 months.

5. Multiple selection in game lists (required for
?  filter functions), about 2 weeks.

6. Search and filter functions, based on CQL, about
?  6 months (already partly implemented).

7. Support of opening books (.ctg and .bin), about 1 month
?  (already partly implemented).

8. Tablebase/endgame support, about 1 month (already partly
?  implemented).

9. HTML export, about 2 months.

10. LaTeX export, about 1 month.

11. PDF export, about 1 month.

12. Keyboard input of games, about 2 weeks.

13. Finishing move information support, about 2 weeks.

14. Internet update of player dictionary, engine dictionary,
? ? and more, about 2 weeks.

15. Finishing the help pages, this will be done concurrently.

16. Loading databases at start-up in background, about 2 weeks.

17. A new tab for database access and displaying information,
? ? about 1 month.

18. Tip-of-the-Day dialog, already implemented, but more
? ? tips have to be written, about 1 week.

19. Spell-correction functions, especially for player names,
? ? event names, and site names, about 1 month.

20. Eliminating duplicated games, about 2 weeks.

21. Finishing and testing the board/piece design dialogs,
? ? about 1 month.

22. Windows version, about 6 months.

Some comments which tasks are necessary for the first
alpha version? Some primary functions still missing?
Please note that the Windows version will not be done
until the first alpha version under Linux is released.

Furthermore after the release of the Windows version the
implementation of Jessy will start. Jessy provides:
playing against engines, automatic analysis of chess games,
playing on ICS or FICS, and some more functions. Jessy
will have a database interface to Scidb, but his own
graphical interface.

Some more tasks for Scidb, but less priority:

23. Support of Bughouse games.

24. Finishing the support of Crazyhouse, Antichess, and
? ? Three-Check Chess.

25. Optimizing the load of the player base, it's still
? ? a bit time consuming.

26. Accelerating the database and PGN import/export.

27. Support of children's chess. This will allow the input
? ? of invalid moves, like Nc1-c3.

28. Database service API (feature request #35).

29. Time usage diagram (feature request #36).

30. Integrated document viewer (feature request #30).

And more...
-------------- next part --------------
An HTML attachment was scrubbed...

------------------------------

Message: 2
Date: Sat, 22 Mar 2014 12:37:51 +0100
From: "Gregor Cramer" <re...@gm...>
Subject: Re: [Scidb-users] Schedule of development
To: sci...@li...
Message-ID:
    <trinity-637049c1-fd8b-4949-956c-be5268a42289-1395488271595@3capp-gmx-bs41>
    
Content-Type: text/plain; charset=UTF-8

> My opinion: 9, 10, 11, 14, 18, 19 and 21 are not of
> primary importance

9, 10, 11 (export to documents) may come after the first
alpha release.

14 (internet update of player data) is nowadays a must,
please consider that with the first alpha version the
distributors (Debian, Arch Linux, ...) may provide
Scidb as a package, and then internet update is the
appropriate method to be up-to-date, the player data
plays an important role for many users, and in this
way Scidb can inform about new updates.

18 (tip of the days) is important for me, the Tip-of-
the-Day dialog plays a significant role to point out
some features of Scidb.

19 (spellchecker) may come later (good task for a
volunteer!).

21 (board/piece design dialogs) may be finished later,
but testing and fixing is unavoidable, currently there
exists some nasty errors. (Somebody who likes to test?).

> Also 22 is not so necessary :)

I agree that the Windows version is not important for
Linux users, but it is unavoidable, about 80% percent
of the chess database users are friends of Windows
(I know, a Linux cannot understand this, and I cannot
understand this, too). Probably it will take less time
to do this, about 90% of the Windows functionality is
already implemented, but testing all the stuff is very
time consuming. Hopefully I will have some assistance
when doing the Windows version.

Is ticket #33 definitely up to Jessy?

It is not yet finally decided whether Ticket 33
(http://sourceforge.net/p/scidb/feature-requests/33/)
is a task for Jessy, but this is not a database
specific function, and may become part of Jessy.

PS:
I've forgotten one feature: a player report. But this
may come after the first alpha release. Any suggestions
about the content of a player report?

PPS:
It may happen that a secondary task will be realized
earlier than planned. Sometimes I have a good idea
how the implement a specific task, and then I'm doing
this. This is one my "methods" to be so fast (due to
Ohloh Scidb's code took an estimated effort of more
than 100 years <http://www.ohloh.net/p/scidb>).

Gregor



------------------------------

Message: 3
Date: Tue, 25 Mar 2014 18:04:49 +0100
From: "Gregor Cramer" <re...@gm...>
Subject: [Scidb-users] Alternative to PGN
To: sci...@li...
Message-ID:
    <trinity-e8b229de-5a85-4c67-a2d2-c4c6f357b9ff-1395767089856@3capp-gmx-bs01>
    
Content-Type: text/plain; charset=UTF-8

Due to the fact that PGN (http://en.wikipedia.org/wiki/Portable_Game_Notation)
is not sufficient for modern chess applications, I'm working on a
different format, especially for the backup of unsaved games.
I'm in fact not very familiar with XML, so I need some suggestions.
My first idea is something like this:

---------------------------------------------------------------------
<?xml version="1.0"?>
<!-- cif -->

<meta charset=utf-8"/>
<meta author="Scidb"/>

<description lang="en">Only an incomplete example</description>

<game variant="normal" startpos="518">
   <tag name="Event">At home</tag>
   <tag name="White">It's me</tag>
   <tag name="Black">It's my opponent</tag>

   <moves>
      1.e4 e5 2.Nf3 Nc6 3.Bc4 Nf6 4.Ng5 $5
         <post lang="en">Due to Tarrasch this is a <b>patzer</b>.</post>
         <post lang="de">Laut Tarrasch ist dies ein <b>Patzerzug</b>.</post>
         <arrow from="g5" to="f7"/>
      4...Bc5
         <marker type="square" color="yellow" to="f2"/>
         (d5 <post lang="en">This is the usual reply.</post>)
      5.Nxf7 Bxf2+
      <trailing lang="en">I've abandoned, that's too much for me.</trailing>
   </moves>
</game>
---------------------------------------------------------------------

This example is of course not complete.

The chess variant and the start position will be
given with the <game...> markup, because this is also
an information for the parser.
<trailing>...</trailing> is a trailing comment,
<post>-...</post> is a comment after last move,
<arrow ...> defines an arrow, and
<marker ...> defines a coloured square.
The syntax for comments is a subset of xhtml (this
makes it parseable for XML parsers, I think), and may
contain UTF-8 characters. But I'm not satisfied with
the syntax of the tags like
   <tag name="Event">At home</tag>
probably somebody knows an alternative way.

<!-- cif --> is the magic number, and is an abbreviation
for "Chess Interchange Format".

Gregor



------------------------------

Message: 4
Date: Tue, 25 Mar 2014 17:10:11 +0000
From: Drew Ferguson <dr...@af...>
Subject: Re: [Scidb-users] Alternative to PGN
To: sci...@li...
Message-ID: <201...@bl...>
Content-Type: text/plain; charset=US-ASCII

Hi Gregor

I think there already is an XML DTD 

Whether it will meet your requirements is another thing; I will try and
hunt it down

On Tue, 25 Mar 2014 18:04:49 +0100
"Gregor Cramer" <re...@gm...> wrote:

> Due to the fact that PGN
> (http://en.wikipedia.org/wiki/Portable_Game_Notation) is not sufficient
> for modern chess applications, I'm working on a different format,
> especially for the backup of unsaved games. I'm in fact not very
> familiar with XML, so I need some suggestions. My first idea is
> something like this:
> 
> ---------------------------------------------------------------------
> <?xml version="1.0"?>
> <!-- cif -->
> 
> <meta charset=utf-8"/>
> <meta author="Scidb"/>
> 
> <description lang="en">Only an incomplete example</description>
> 
> <game variant="normal" startpos="518">
>    <tag name="Event">At home</tag>
>    <tag name="White">It's me</tag>
>    <tag name="Black">It's my opponent</tag>
> 
>    <moves>
>       1.e4 e5 2.Nf3 Nc6 3.Bc4 Nf6 4.Ng5 $5
>          <post lang="en">Due to Tarrasch this is a <b>patzer</b>.</post>
>          <post lang="de">Laut Tarrasch ist dies ein
> <b>Patzerzug</b>.</post> <arrow from="g5" to="f7"/>
>       4...Bc5
>          <marker type="square" color="yellow" to="f2"/>
>          (d5 <post lang="en">This is the usual reply.</post>)
>       5.Nxf7 Bxf2+
>       <trailing lang="en">I've abandoned, that's too much for
> me.</trailing> </moves>
> </game>
> ---------------------------------------------------------------------
> 
> This example is of course not complete.
> 
> The chess variant and the start position will be
> given with the <game...> markup, because this is also
> an information for the parser.
> <trailing>...</trailing> is a trailing comment,
> <post>-...</post> is a comment after last move,
> <arrow ...> defines an arrow, and
> <marker ...> defines a coloured square.
> The syntax for comments is a subset of xhtml (this
> makes it parseable for XML parsers, I think), and may
> contain UTF-8 characters. But I'm not satisfied with
> the syntax of the tags like
>    <tag name="Event">At home</tag>
> probably somebody knows an alternative way.
> 
> <!-- cif --> is the magic number, and is an abbreviation
> for "Chess Interchange Format".
> 
> Gregor
> 
> ------------------------------------------------------------------------------
> Learn Graph Databases - Download FREE O'Reilly Book
> "Graph Databases" is the definitive new guide to graph databases and
> their applications. Written by three acclaimed leaders in the field,
> this first edition is now available. Download your free book today!
> http://p.sf.net/sfu/13534_NeoTech
> _______________________________________________
> Scidb-users mailing list
> Sci...@li...
> https://lists.sourceforge.net/lists/listinfo/scidb-users



-- 
Drew Ferguson
AFC Commercial
http://www.afccommercial.co.uk



------------------------------

------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/13534_NeoTech

------------------------------

_______________________________________________
Scidb-users mailing list
Sci...@li...
https://lists.sourceforge.net/lists/listinfo/scidb-users


End of Scidb-users Digest, Vol 20, Issue 2
******************************************

Hey Gregor

Where did your reply go...

I saw it come in on my phone and now I cannot find it anywhere, not even
the SF archive?!?

Anyway, just because the XML DTDs are really old doesn't mean they are not
in use or useful, just that folks are still stuck with PGN & ChessBase
I seem to remember one of these getting some traction at the time and even
getting registered formally somewhere; or maybe thats just what happens
with DTDs

Personally I favour a simple INI-file style i.e. key-value pairs and
sections; terser even than PGN 

On Tue, 25 Mar 2014 17:41:55 +0000
Drew Ferguson <dr...@af...> wrote:

> Here's one
> 
>   http://www.cybercom.net/~zbrad/Chess/pgnxml/
> 
> and another
> 
>   http://www.saremba.de/chessgml/
> 
> and yet another
> 
>   http://xml.coverpages.org/chessML.html
> 
> Last one came out of uni-bonn.de I believe
> 
> Oh dear, I thought there was something approaching a consensus for a
> post-PGN format but it seems not and these links appear to be almost in
> the previous millenium!
> 
> Though not sure why you would want to use XML if this is only for you. In
> general XML is very verbose compared to something as terse as PGN. Indeed
> what is your problem with PGN, you can add any tag you want especially if
> it is only for your own purposes.
> 
> On Tue, 25 Mar 2014 18:04:49 +0100
> "Gregor Cramer" <re...@gm...> wrote:
> 
> > Due to the fact that PGN
> > (http://en.wikipedia.org/wiki/Portable_Game_Notation) is not sufficient
> > for modern chess applications, I'm working on a different format,
> > especially for the backup of unsaved games. I'm in fact not very
> > familiar with XML, so I need some suggestions. My first idea is
> > something like this:
> > 
> > ---------------------------------------------------------------------
> > <?xml version="1.0"?>
> > <!-- cif -->
> > 
> > <meta charset=utf-8"/>
> > <meta author="Scidb"/>
> > 
> > <description lang="en">Only an incomplete example</description>
> > 
> > <game variant="normal" startpos="518">
> >    <tag name="Event">At home</tag>
> >    <tag name="White">It's me</tag>
> >    <tag name="Black">It's my opponent</tag>
> > 
> >    <moves>
> >       1.e4 e5 2.Nf3 Nc6 3.Bc4 Nf6 4.Ng5 $5
> >          <post lang="en">Due to Tarrasch this is a
> > <b>patzer</b>.</post> <post lang="de">Laut Tarrasch ist dies ein
> > <b>Patzerzug</b>.</post> <arrow from="g5" to="f7"/>
> >       4...Bc5
> >          <marker type="square" color="yellow" to="f2"/>
> >          (d5 <post lang="en">This is the usual reply.</post>)
> >       5.Nxf7 Bxf2+
> >       <trailing lang="en">I've abandoned, that's too much for
> > me.</trailing> </moves>
> > </game>
> > ---------------------------------------------------------------------
> > 
> > This example is of course not complete.
> > 
> > The chess variant and the start position will be
> > given with the <game...> markup, because this is also
> > an information for the parser.
> > <trailing>...</trailing> is a trailing comment,
> > <post>-...</post> is a comment after last move,
> > <arrow ...> defines an arrow, and
> > <marker ...> defines a coloured square.
> > The syntax for comments is a subset of xhtml (this
> > makes it parseable for XML parsers, I think), and may
> > contain UTF-8 characters. But I'm not satisfied with
> > the syntax of the tags like
> >    <tag name="Event">At home</tag>
> > probably somebody knows an alternative way.
> > 
> > <!-- cif --> is the magic number, and is an abbreviation
> > for "Chess Interchange Format".
> > 
> > Gregor
> > 
> > ------------------------------------------------------------------------------
> > Learn Graph Databases - Download FREE O'Reilly Book
> > "Graph Databases" is the definitive new guide to graph databases and
> > their applications. Written by three acclaimed leaders in the field,
> > this first edition is now available. Download your free book today!
> > http://p.sf.net/sfu/13534_NeoTech
> > _______________________________________________
> > Scidb-users mailing list
> > Sci...@li...
> > https://lists.sourceforge.net/lists/listinfo/scidb-users
> 
> 
> 



-- 
Drew Ferguson
AFC Commercial
http://www.afccommercial.co.uk

Here's one

  http://www.cybercom.net/~zbrad/Chess/pgnxml/

and another

  http://www.saremba.de/chessgml/

and yet another

  http://xml.coverpages.org/chessML.html

Last one came out of uni-bonn.de I believe

Oh dear, I thought there was something approaching a consensus for a
post-PGN format but it seems not and these links appear to be almost in
the previous millenium!

Though not sure why you would want to use XML if this is only for you. In
general XML is very verbose compared to something as terse as PGN. Indeed
what is your problem with PGN, you can add any tag you want especially if
it is only for your own purposes.

On Tue, 25 Mar 2014 18:04:49 +0100
"Gregor Cramer" <re...@gm...> wrote:

> Due to the fact that PGN
> (http://en.wikipedia.org/wiki/Portable_Game_Notation) is not sufficient
> for modern chess applications, I'm working on a different format,
> especially for the backup of unsaved games. I'm in fact not very
> familiar with XML, so I need some suggestions. My first idea is
> something like this:
> 
> ---------------------------------------------------------------------
> <?xml version="1.0"?>
> <!-- cif -->
> 
> <meta charset=utf-8"/>
> <meta author="Scidb"/>
> 
> <description lang="en">Only an incomplete example</description>
> 
> <game variant="normal" startpos="518">
>    <tag name="Event">At home</tag>
>    <tag name="White">It's me</tag>
>    <tag name="Black">It's my opponent</tag>
> 
>    <moves>
>       1.e4 e5 2.Nf3 Nc6 3.Bc4 Nf6 4.Ng5 $5
>          <post lang="en">Due to Tarrasch this is a <b>patzer</b>.</post>
>          <post lang="de">Laut Tarrasch ist dies ein
> <b>Patzerzug</b>.</post> <arrow from="g5" to="f7"/>
>       4...Bc5
>          <marker type="square" color="yellow" to="f2"/>
>          (d5 <post lang="en">This is the usual reply.</post>)
>       5.Nxf7 Bxf2+
>       <trailing lang="en">I've abandoned, that's too much for
> me.</trailing> </moves>
> </game>
> ---------------------------------------------------------------------
> 
> This example is of course not complete.
> 
> The chess variant and the start position will be
> given with the <game...> markup, because this is also
> an information for the parser.
> <trailing>...</trailing> is a trailing comment,
> <post>-...</post> is a comment after last move,
> <arrow ...> defines an arrow, and
> <marker ...> defines a coloured square.
> The syntax for comments is a subset of xhtml (this
> makes it parseable for XML parsers, I think), and may
> contain UTF-8 characters. But I'm not satisfied with
> the syntax of the tags like
>    <tag name="Event">At home</tag>
> probably somebody knows an alternative way.
> 
> <!-- cif --> is the magic number, and is an abbreviation
> for "Chess Interchange Format".
> 
> Gregor
> 
> ------------------------------------------------------------------------------
> Learn Graph Databases - Download FREE O'Reilly Book
> "Graph Databases" is the definitive new guide to graph databases and
> their applications. Written by three acclaimed leaders in the field,
> this first edition is now available. Download your free book today!
> http://p.sf.net/sfu/13534_NeoTech
> _______________________________________________
> Scidb-users mailing list
> Sci...@li...
> https://lists.sourceforge.net/lists/listinfo/scidb-users



-- 
Drew Ferguson
AFC Commercial
http://www.afccommercial.co.uk

Hi Gregor

I think there already is an XML DTD 

Whether it will meet your requirements is another thing; I will try and
hunt it down

On Tue, 25 Mar 2014 18:04:49 +0100
"Gregor Cramer" <re...@gm...> wrote:

> Due to the fact that PGN
> (http://en.wikipedia.org/wiki/Portable_Game_Notation) is not sufficient
> for modern chess applications, I'm working on a different format,
> especially for the backup of unsaved games. I'm in fact not very
> familiar with XML, so I need some suggestions. My first idea is
> something like this:
> 
> ---------------------------------------------------------------------
> <?xml version="1.0"?>
> <!-- cif -->
> 
> <meta charset=utf-8"/>
> <meta author="Scidb"/>
> 
> <description lang="en">Only an incomplete example</description>
> 
> <game variant="normal" startpos="518">
>    <tag name="Event">At home</tag>
>    <tag name="White">It's me</tag>
>    <tag name="Black">It's my opponent</tag>
> 
>    <moves>
>       1.e4 e5 2.Nf3 Nc6 3.Bc4 Nf6 4.Ng5 $5
>          <post lang="en">Due to Tarrasch this is a <b>patzer</b>.</post>
>          <post lang="de">Laut Tarrasch ist dies ein
> <b>Patzerzug</b>.</post> <arrow from="g5" to="f7"/>
>       4...Bc5
>          <marker type="square" color="yellow" to="f2"/>
>          (d5 <post lang="en">This is the usual reply.</post>)
>       5.Nxf7 Bxf2+
>       <trailing lang="en">I've abandoned, that's too much for
> me.</trailing> </moves>
> </game>
> ---------------------------------------------------------------------
> 
> This example is of course not complete.
> 
> The chess variant and the start position will be
> given with the <game...> markup, because this is also
> an information for the parser.
> <trailing>...</trailing> is a trailing comment,
> <post>-...</post> is a comment after last move,
> <arrow ...> defines an arrow, and
> <marker ...> defines a coloured square.
> The syntax for comments is a subset of xhtml (this
> makes it parseable for XML parsers, I think), and may
> contain UTF-8 characters. But I'm not satisfied with
> the syntax of the tags like
>    <tag name="Event">At home</tag>
> probably somebody knows an alternative way.
> 
> <!-- cif --> is the magic number, and is an abbreviation
> for "Chess Interchange Format".
> 
> Gregor
> 
> ------------------------------------------------------------------------------
> Learn Graph Databases - Download FREE O'Reilly Book
> "Graph Databases" is the definitive new guide to graph databases and
> their applications. Written by three acclaimed leaders in the field,
> this first edition is now available. Download your free book today!
> http://p.sf.net/sfu/13534_NeoTech
> _______________________________________________
> Scidb-users mailing list
> Sci...@li...
> https://lists.sourceforge.net/lists/listinfo/scidb-users



-- 
Drew Ferguson
AFC Commercial
http://www.afccommercial.co.uk

Due to the fact that PGN (http://en.wikipedia.org/wiki/Portable_Game_Notation)
is not sufficient for modern chess applications, I'm working on a
different format, especially for the backup of unsaved games.
I'm in fact not very familiar with XML, so I need some suggestions.
My first idea is something like this:

---------------------------------------------------------------------
<?xml version="1.0"?>
<!-- cif -->

<meta charset=utf-8"/>
<meta author="Scidb"/>

<description lang="en">Only an incomplete example</description>

<game variant="normal" startpos="518">
   <tag name="Event">At home</tag>
   <tag name="White">It's me</tag>
   <tag name="Black">It's my opponent</tag>

   <moves>
      1.e4 e5 2.Nf3 Nc6 3.Bc4 Nf6 4.Ng5 $5
         <post lang="en">Due to Tarrasch this is a <b>patzer</b>.</post>
         <post lang="de">Laut Tarrasch ist dies ein <b>Patzerzug</b>.</post>
         <arrow from="g5" to="f7"/>
      4...Bc5
         <marker type="square" color="yellow" to="f2"/>
         (d5 <post lang="en">This is the usual reply.</post>)
      5.Nxf7 Bxf2+
      <trailing lang="en">I've abandoned, that's too much for me.</trailing>
   </moves>
</game>
---------------------------------------------------------------------

This example is of course not complete.

The chess variant and the start position will be
given with the <game...> markup, because this is also
an information for the parser.
<trailing>...</trailing> is a trailing comment,
<post>-...</post> is a comment after last move,
<arrow ...> defines an arrow, and
<marker ...> defines a coloured square.
The syntax for comments is a subset of xhtml (this
makes it parseable for XML parsers, I think), and may
contain UTF-8 characters. But I'm not satisfied with
the syntax of the tags like
   <tag name="Event">At home</tag>
probably somebody knows an alternative way.

<!-- cif --> is the magic number, and is an abbreviation
for "Chess Interchange Format".

Gregor

> My opinion: 9, 10, 11, 14, 18, 19 and 21 are not of
> primary importance

9, 10, 11 (export to documents) may come after the first
alpha release.

14 (internet update of player data) is nowadays a must,
please consider that with the first alpha version the
distributors (Debian, Arch Linux, ...) may provide
Scidb as a package, and then internet update is the
appropriate method to be up-to-date, the player data
plays an important role for many users, and in this
way Scidb can inform about new updates.

18 (tip of the days) is important for me, the Tip-of-
the-Day dialog plays a significant role to point out
some features of Scidb.

19 (spellchecker) may come later (good task for a
volunteer!).

21 (board/piece design dialogs) may be finished later,
but testing and fixing is unavoidable, currently there
exists some nasty errors. (Somebody who likes to test?).

> Also 22 is not so necessary :)

I agree that the Windows version is not important for
Linux users, but it is unavoidable, about 80% percent
of the chess database users are friends of Windows
(I know, a Linux cannot understand this, and I cannot
understand this, too). Probably it will take less time
to do this, about 90% of the Windows functionality is
already implemented, but testing all the stuff is very
time consuming. Hopefully I will have some assistance
when doing the Windows version.

Is ticket #33 definitely up to Jessy?

It is not yet finally decided whether Ticket 33
(http://sourceforge.net/p/scidb/feature-requests/33/)
is a task for Jessy, but this is not a database
specific function, and may become part of Jessy.

PS:
I've forgotten one feature: a player report. But this
may come after the first alpha release. Any suggestions
about the content of a player report?

PPS:
It may happen that a secondary task will be realized
earlier than planned. Sometimes I have a good idea
how the implement a specific task, and then I'm doing
this. This is one my "methods" to be so fast (due to
Ohloh Scidb's code took an estimated effort of more
than 100 years <http://www.ohloh.net/p/scidb>).

Gregor

Nice schedule!

My opinion: 9, 10, 11, 14, 18, 19 and 21 are not of primary importance (almost 7 months in your estimate).

Also 22 is not so necessary :)

Is ticket #33 definitely up to Jessy?


Bye






Message: 1
Date: Wed, 19 Mar 2014 18:55:46 +0100
From: "Gregor Cramer" <re...@gm...>
Subject: [Scidb-users] Schedule of development
To: sci...@li...
Message-ID:
    <trinity-e387000b-3c0a-49dd-a71a-d1702fef198f-1395251745889@3capp-gmx-bs30>
    
Content-Type: text/plain; charset=UTF-8

A rough overview about the development schedule, and the
estimated time how long implementation and testing will
still take. Please note that below are only the primary
functions, there are still more secondary functions.
And please consider that it is hard to estimate the
development time, the real development time may differ
significantly (in both directions).

1. New database format, allowing faster search, supports
   opening classification, smaller index file, less memory
   consumption, and eventually smaller game files,
   currently in test phase (hopefully successful, this is
   all based on a transposition-invariant classification
   key, completely different from Scid's technology, and I
   had to overwork the emulation of Scid databases).

2. Game links, already implemented, not yet tested,
   about 2 weeks.

3. Opening classification tab (already partly implemented),
   this includes the overwork of the ECO list, the current
   list is not integrally closed, about 1 month.

4. Flexible window layout, about 3 months.

5. Multiple selection in game lists (required for
   filter functions), about 2 weeks.

6. Search and filter functions, based on CQL, about
   6 months (already partly implemented).

7. Support of opening books (.ctg and .bin), about 1 month
   (already partly implemented).

8. Tablebase/endgame support, about 1 month (already partly
   implemented).

9. HTML export, about 2 months.

10. LaTeX export, about 1 month.

11. PDF export, about 1 month.

12. Keyboard input of games, about 2 weeks.

13. Finishing move information support, about 2 weeks.

14. Internet update of player dictionary, engine dictionary,
    and more, about 2 weeks.

15. Finishing the help pages, this will be done concurrently.

16. Loading databases at start-up in background, about 2 weeks.

17. A new tab for database access and displaying information,
    about 1 month.

18. Tip-of-the-Day dialog, already implemented, but more
    tips have to be written, about 1 week.

19. Spell-correction functions, especially for player names,
    event names, and site names, about 1 month.

20. Eliminating duplicated games, about 2 weeks.

21. Finishing and testing the board/piece design dialogs,
    about 1 month.

22. Windows version, about 6 months.

Some comments which tasks are necessary for the first
alpha version? Some primary functions still missing?
Please note that the Windows version will not be done
until the first alpha version under Linux is released.

Furthermore after the release of the Windows version the
implementation of Jessy will start. Jessy provides:
playing against engines, automatic analysis of chess games,
playing on ICS or FICS, and some more functions. Jessy
will have a database interface to Scidb, but his own
graphical interface.

Some more tasks for Scidb, but less priority:

23. Support of Bughouse games.

24. Finishing the support of Crazyhouse, Antichess, and
    Three-Check Chess.

25. Optimizing the load of the player base, it's still
    a bit time consuming.

26. Accelerating the database and PGN import/export.

27. Support of children's chess. This will allow the input
    of invalid moves, like Nc1-c3.

28. Database service API (feature request #35).

29. Time usage diagram (feature request #36).

30. Integrated document viewer (feature request #30).

And more...