Experience version 3 : investigation

Code, algorithms, languages, construction...
Amos 4ever
Posts: 62
Joined: Sun Aug 10, 2014 9:42 pm
Real Name: Marco Zerbinati

Re: Experience version 3 : investigation

Post by Amos 4ever » Fri Jul 14, 2023 6:18 pm

deeds wrote:
Fri Jul 14, 2023 5:40 pm
Amos 4ever wrote:
Fri Jul 14, 2023 5:17 pm
Clarification please, suppose I have a v2 exp file after converting it to v3 and still using the file playing games, will I have new data stored with v2 again so can I safely reuse the tool to update it again to v3?
Huh !? When Eman 7.00 first loads a v1 experience file, it converts it to a v2 experience file and updates it with new v2 data.

Be serious, once a chess engine or chess system adopts a file format for its experience data, there's no point in jumping back and forth between versions.

Given all the investigations presented above and the possible gains, do you really think I'm going to stop at the version 3.0 which saves only 33% ? loool
I'm serious, maybe I didn't explain myself well
I try to simplify like this
if I use with the SugaR engine an esp file converted to version 3
what happens with the new data it stores?
are they stored in the new v3 format?
It seems fair to me to ask this question

User avatar
deeds
Posts: 1008
Joined: Wed Oct 20, 2021 9:24 pm
Location: France
Contact:

Re: Experience version 3 : investigation

Post by deeds » Fri Jul 14, 2023 7:01 pm

Amos 4ever wrote:
Fri Jul 14, 2023 6:18 pm
I'm serious, maybe I didn't explain myself well
I try to simplify like this
if I use with the SugaR engine an esp file converted to version 3
what happens with the new data it stores?
are they stored in the new v3 format?
It seems fair to me to ask this question
Does SugaR read v3 experience file format !?

Amos 4ever
Posts: 62
Joined: Sun Aug 10, 2014 9:42 pm
Real Name: Marco Zerbinati

Re: Experience version 3 : investigation

Post by Amos 4ever » Fri Jul 14, 2023 7:13 pm

deeds wrote:
Fri Jul 14, 2023 7:01 pm
Amos 4ever wrote:
Fri Jul 14, 2023 6:18 pm
I'm serious, maybe I didn't explain myself well
I try to simplify like this
if I use with the SugaR engine an esp file converted to version 3
what happens with the new data it stores?
are they stored in the new v3 format?
It seems fair to me to ask this question
Does SugaR read v3 experience file format !?
I suppose so.. I think it makes sense that he should read it
but i haven't tried
anyway forget it, my question was simple but it is clear that you do not want to answer, thanks all the same

User avatar
IbaiBuR
Posts: 208
Joined: Sun Jul 03, 2022 4:08 pm
Real Name: Ibai Burgos
Location: Madrid, Spain
Contact:

Re: Experience version 3 : investigation

Post by IbaiBuR » Fri Jul 14, 2023 9:05 pm

Amos 4ever wrote:
Fri Jul 14, 2023 6:18 pm
deeds wrote:
Fri Jul 14, 2023 5:40 pm
Amos 4ever wrote:
Fri Jul 14, 2023 5:17 pm
Clarification please, suppose I have a v2 exp file after converting it to v3 and still using the file playing games, will I have new data stored with v2 again so can I safely reuse the tool to update it again to v3?
Huh !? When Eman 7.00 first loads a v1 experience file, it converts it to a v2 experience file and updates it with new v2 data.

Be serious, once a chess engine or chess system adopts a file format for its experience data, there's no point in jumping back and forth between versions.

Given all the investigations presented above and the possible gains, do you really think I'm going to stop at the version 3.0 which saves only 33% ? loool
I'm serious, maybe I didn't explain myself well
I try to simplify like this
if I use with the SugaR engine an esp file converted to version 3
what happens with the new data it stores?
are they stored in the new v3 format?
It seems fair to me to ask this question
I mean... If SugaR doesnt read v3 format (because current format as you may now is 8 bytes for the zobrist hash key, 4 bytes for the rest) as it is different from v2 (SugaR's default) and if you dont change the functions that write in the experience file, it is clear that SugaR will not store the new data in the v3 format, it will conitnue storing it in its default one... You have to change the internal code (experience.h and experience.cpp) for SugaR to handle v3 format

Some changes are quite easy, for example, you could change depth to uint8_t in the experience.h (you only need 1 byte for depth as explained by DeeDs since depth is capped to 245 (with 1 byte you can reach up to 255->2⁸ -1)and used in v3 format)

Hope helps,

Ibai

User avatar
IbaiBuR
Posts: 208
Joined: Sun Jul 03, 2022 4:08 pm
Real Name: Ibai Burgos
Location: Madrid, Spain
Contact:

Re: Experience version 3 : investigation

Post by IbaiBuR » Fri Jul 14, 2023 9:21 pm

For sure, there are a lot more changes to be made, but you should start by changing the types of the variables that create the ExpEntry struct....

Just saying

:)

User avatar
deeds
Posts: 1008
Joined: Wed Oct 20, 2021 9:24 pm
Location: France
Contact:

Re: Experience version 3 : investigation

Post by deeds » Sat Jul 15, 2023 4:02 am

Amos 4ever wrote:
Fri Jul 14, 2023 7:13 pm
deeds wrote:
Fri Jul 14, 2023 7:01 pm
Does SugaR read v3 experience file format !?
I suppose so.. I think it makes sense that he should read it
but i haven't tried
I doubt !

In v3.0, the signature has changed, the 8 bytes of the position key are in normal order, there are only 2 bytes for moves in normal order, only 3 bytes for scores in normal order, scores use a new value's range (from 0 to 262 144), etc.

User avatar
deeds
Posts: 1008
Joined: Wed Oct 20, 2021 9:24 pm
Location: France
Contact:

Re: Experience version 3 : investigation

Post by deeds » Sat Jul 15, 2023 10:40 am

v3.0 specs :

Image

position key (8 bytes) : zobrish key hashed by engine, bytes stored in normal order
nb moves (1 byte) : reserved for next versions
move (2 bytes) : same as v2, bytes stored in normal order
score (3 bytes) : new range, original values stored in normal order
depth (1 byte) : same as v2
count (1 byte) : same as v2

User avatar
deeds
Posts: 1008
Joined: Wed Oct 20, 2021 9:24 pm
Location: France
Contact:

Re: Experience version 3 : investigation

Post by deeds » Sun Jul 16, 2023 10:35 am

v3.1 : let's see if we can break the 33% filesize reduction of v3.0 without losing too much information...


After conversion from v2 to v3.1, the experience files looks like this :

Image


And now, let's check if the conversion is lossless :

Image
Image


The conversion from v2 scores to v3.1 scores isn't lossless because from one format to another, we multiply/divide integers by 2.08, of course...
If the error margin stays within a range of +/- 0.01 (1 cp), it's viable !

User avatar
deeds
Posts: 1008
Joined: Wed Oct 20, 2021 9:24 pm
Location: France
Contact:

Re: Experience version 3 : investigation

Post by deeds » Mon Jul 17, 2023 1:37 pm

Some stats about experience files converted to v3.1 :

Image

Image

Image

Image

Image
...

User avatar
deeds
Posts: 1008
Joined: Wed Oct 20, 2021 9:24 pm
Location: France
Contact:

Re: Experience version 3 : investigation

Post by deeds » Mon Jul 17, 2023 1:54 pm

...
Image

Image

Ok so with v3.1 the filesize gain is quite variable but still better than with v3.0.

As the v3.0 already required a lot of modifications to the source code, i think people can be satisfied with its average filesize gain of 33%.
As i already shared a tool to convert experience file from v2 to v3.0, no need to pollute github more.

From my side, i plan to work on v3.2 maybe with an engine-independent position key or an another use of the count values... But i think newer version than 3.1 will be exclusive to D.C.S.

Post Reply