Experience version 3 : investigation

Code, algorithms, languages, construction...
User avatar
deeds
Posts: 754
Joined: Wed Oct 20, 2021 9:24 pm
Location: France
Contact:

Experience version 3 : investigation

Post by deeds » Thu Jul 06, 2023 2:14 pm

The old v2 experience file format contains :

- 8 bytes to store the position's key (ex : [B4 D3 0C D1 5A 43 43 2D] = [rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1] for the start position)
Image


- 4 bytes to store the move + promoted piece (ex : [00 00 03 1C] = [0 000 001(2) 100(e) 011(4) 100(e)] for the "e2e4" move)
Image


- 4 bytes to store the move score (ex : [00 00 00 46] = 33 centipawns, it means "+0.33/4")
Image


- 4 bytes to store the score depth (ex : [00 00 00 04] = 4, it means "+0.33/4")
Image


- 4 bytes to store the count value (ex : [00 00 00 01] = 1, it means the "e2e4" move was played 1 time)
Image

User avatar
deeds
Posts: 754
Joined: Wed Oct 20, 2021 9:24 pm
Location: France
Contact:

Re: Experience version 3 : investigation

Post by deeds » Thu Jul 06, 2023 2:33 pm

deeds wrote:
Thu Jul 06, 2023 2:14 pm
- 8 bytes to store the position's key (ex : [B4 D3 0C D1 5A 43 43 2D] = [rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1] for the start position)
Image
v2 : If a position contains several moves, the 8 bytes of its key are duplicated as many times as there are moves.
Example with the starting position which contains its 20 legal moves :
v2 : 20 * 24 bytes [8 (key) + 4 (move) + 4 (score) + 4 (depth) + 4 (count)] = 480 bytes

v3: We could only store the 8 bytes of the position key once and use a 9th byte (=0 to 255) to indicate the number of moves to load.
Example with the starting position which contains its 20 legal moves :
v3: 8 (key) + 1 (nb moves) + 20 * 16 [4 (move) + 4 (score) + 4 (depth) + 4 (count)] = 329 bytes (-31%)

User avatar
deeds
Posts: 754
Joined: Wed Oct 20, 2021 9:24 pm
Location: France
Contact:

Re: Experience version 3 : investigation

Post by deeds » Thu Jul 06, 2023 2:54 pm

deeds wrote:
Thu Jul 06, 2023 2:14 pm
- 4 bytes to store the move + promoted piece (ex : [00 00 03 1C] = [0 000 001(2) 100(e) 011(4) 100(e)] for the "e2e4" move)
Image
v2: All moves consume 32 bits or 4 bytes

v3: All possible moves can fit in 15 bits or 2 bytes (-50%):
- 12 bits are sufficient from A(000) 1(000) to H(111) 8(111)
- 3 bits are enough for special moves, no promotion (000) or promotion to queen (001), to rook (010), to bishop (011), to knight (100), short castling (101), long castling (110)

User avatar
deeds
Posts: 754
Joined: Wed Oct 20, 2021 9:24 pm
Location: France
Contact:

Re: Experience version 3 : investigation

Post by deeds » Thu Jul 06, 2023 4:53 pm

deeds wrote:
Thu Jul 06, 2023 2:14 pm
- 4 bytes to store the move score (ex : [00 00 00 46] = 33 centipawns, it means "+0.33/4")
Image
v2 : With 4 bytes, from [00 00 00 00] to [FF FF FF FF] there are 4'294'967'295 available values.

v3 : We only need 3 bytes (-25%) because generally at chess :
- the normal scores go from "-100'000" cp (-1000.00) to "+100'000" cp (+1000.00) so we need a range of 200'001 values
- the mate scores go from "mate -1000" to "mate +1000" so we need an additional range of 2'000 values
From [00 00 00] to [04 00 00] there are already 262'144 available values.
Until [FF FF FF], there are 16'777'215 available values so we even have room to handle tablebases scores.

User avatar
deeds
Posts: 754
Joined: Wed Oct 20, 2021 9:24 pm
Location: France
Contact:

Re: Experience version 3 : investigation

Post by deeds » Thu Jul 06, 2023 4:56 pm

deeds wrote:
Thu Jul 06, 2023 2:14 pm
- 4 bytes to store the score depth (ex : [00 00 00 04] = 4, it means "+0.33/4")
Image
v2 : With 4 bytes, from [00 00 00 00] to [FF FF FF FF] there are 4'294'967'295 available values.

v3 : We only need 1 byte (-75%) because so far most engines are capping at D245.

User avatar
deeds
Posts: 754
Joined: Wed Oct 20, 2021 9:24 pm
Location: France
Contact:

Re: Experience version 3 : investigation

Post by deeds » Fri Jul 07, 2023 6:28 am

deeds wrote:
Thu Jul 06, 2023 2:14 pm
- 4 bytes to store the count value (ex : [00 00 00 01] = 1, it means the "e2e4" move was played 1 time)
Image
v2 : With 4 bytes, from [00 00 00 00] to [FF FF FF FF] there are 4'294'967'295 available values.

v3 : We only need 2 bytes (-50%) because the learning needs at least 500 games/opening so the first learned moves can be played at least 500 times.

User avatar
deeds
Posts: 754
Joined: Wed Oct 20, 2021 9:24 pm
Location: France
Contact:

Re: Experience version 3 : investigation

Post by deeds » Fri Jul 07, 2023 6:49 am

deeds wrote:
Fri Jul 07, 2023 6:28 am
v3 : We only need 2 bytes (-50%) because the learning needs at least 500 games/opening so the first learned moves can be played at least 500 times.
v3 : We only need 1 byte (-75%) because if the move is played 255 times, it's even sufficient to obtain its effectiveness

User avatar
deeds
Posts: 754
Joined: Wed Oct 20, 2021 9:24 pm
Location: France
Contact:

Re: Experience version 3 : investigation

Post by deeds » Fri Jul 07, 2023 6:55 am

In sum,

v2 : we have 8 bytes/position + 16 bytes/moves
ex : 1 move/position = 24 bytes, 2 moves/position = 48 bytes, 4 moves/position = 96 bytes, etc.

v3 : we have 9 bytes/position + 7 bytes/moves
ex : 1 move/position = 16 bytes (-33%), 2 moves/position = 23 bytes (-52%), 4 moves/position = 37 bytes (-61%), etc.

Robert57
Posts: 16
Joined: Fri May 26, 2023 9:19 am
Real Name: Marco Cerri

Re: Experience version 3 : investigation

Post by Robert57 » Sat Jul 08, 2023 12:28 pm

@deeds what is meant by experience v3?
no code has been released with further learning update by the author of SugaR, am I getting confused?
If so, can you explain better, thanks

User avatar
deeds
Posts: 754
Joined: Wed Oct 20, 2021 9:24 pm
Location: France
Contact:

Re: Experience version 3 : investigation

Post by deeds » Sat Jul 08, 2023 4:28 pm

Robert57 wrote:
Sat Jul 08, 2023 12:28 pm
@deeds what is meant by experience v3?
SugaR Experience version 1 contains :
- position's key
- move
- score
- depth
- count
- next score
- next 10th score

SugaR Experience version 2 contains :
- position's key
- move
- score
- depth
- count

Etc.

Robert57 wrote:
Sat Jul 08, 2023 12:28 pm
no code has been released with further learning update by the author of SugaR, am I getting confused?
None of the authors of SugaR, Eman, BrainLearn is the author of the learning code. I bet none of them train their engines with the 500 games/opening they advise lol. At best, they fill their experience files by playing on chess servers with default settings (=training mode).

With D.C.S. i already can read/write all their experience file formats but as i'm planning to add a learning feature to any non-learning engines, i'm already working on the experience version 3.

Post Reply