Designing an analysis friendly Stockfish?

Code, algorithms, languages, construction...
Post Reply
Uly
Posts: 838
Joined: Thu Jun 10, 2010 5:33 am

Re: Designing an analysis friendly Stockfish?

Post by Uly »

Prima wrote:Just out of curiosity, what does "PA_G" mean in the updated Stockfish 2.0.1 build?
Preserve Analysis G (The G is the alphabetical version).

Thanks everyone for your efforts! Stockfish is now showing the wished behavior! Here's cross posts from RF:
Banned for Life wrote:Analyzing with the modified Stockfish is now a pleasure, whereas before it was maddening to watch the evaluations jump up and down, and as you mentioned, if you moved along the PV, the engine would need to fill in the holes in the tt, which seemed like a waste of time. It would be interesting to test this configuration and see how it compares in game play against the default, as Dr. Hyatt's claim that using exact entries adds a few Elo points certainly seems reasonable.

The second step that was discussed was putting in what is referred to in Rybka as "Preserve Analysis" where hash replacement is no longer weighted (or more lightly weighted) by generation. This will allow moving to different root locations without deemphasizing current hash entries. Of course this is only suitable for analysis and will require the operator to manually clear the hash when moving to a different position.
I see what he means, for instance, the root (opening position) shows the wished behavior:

21/26 0:06 +0.40 1.e4 e5 2.Nf3 Nf6 3.d4 Nxe4 4.Bd3 d5 5.dxe5 Be7
22/26 0:08 +0.40 1.e4 e5 2.Nf3 Nf6 3.d4 Nxe4 4.Bd3 d5 5.dxe5 Be7

(e4 forced)

21/25 0:00 +0.40 1...e5 2.Nf3 Nf6 3.d4 Nxe4 4.Bd3 d5 5.dxe5 Be7
22/05 0:03 +0.32++ 1...e5 2.Nf3 Nf6 3.d4 Nxe4 (7.694.059) 2499
22/19 0:07 +0.28 1...e5 2.Nf3 Nf6 3.d4 Nxe4 4.Nxe5 d6 5.Nf3 d5 6.Bd3

Back to root

22/20 0:00 +0.28 1.e4 e5 2.Nf3 Nf6 3.d4 Nxe4 4.Nxe5 d6 5.Nf3 d5
23/08 0:01 +0.28 1.e4 e5 2.Nf3 Nf6 3.d4 Nxe4 4.Nxe5 d6 (3.746.798)

However, it's enough for one to analyze some other position

(1. e4 Nc6 2. d4 d5 3. e5 Bf5 4. Ne2 e6 5. Ng3)

18/21 0:02 +0.16 5...Bg6 6.Bb5 Qd7 7.O-O f6 8.Nd2 a6 9.Bxc6 Qxc6
19/21 0:02 +0.16 5...Bg6 6.Bb5 Qd7 7.O-O f6 8.Nd2 a6 9.Bxc6 Qxc6
20/22 0:03 +0.24-- 5...Bg6 6.Bb5 Qd7 7.O-O f6 8.Nd2 a6 9.Bxc6 Qxc6
20/23 0:05 +0.12 5...Bg6 6.Bb5 Qd7 7.O-O a6 8.Bxc6 Qxc6 9.c3 O-O-O
21/20 0:09 +0.16 5...Bg6 6.Bb5 Qd7 7.O-O a6 8.Bd3 f6 9.f4 O-O-O
22/22 0:13 +0.12 5...Bg6 6.Bb5 Qd7 7.O-O a6 8.Bd3 f6 9.f4 O-O-O
23/21 0:17 +0.12 5...Bg6 6.Bb5 Qd7 7.O-O a6 8.Bd3 f6 9.f4 O-O-O

For Stockfish to forget all the other analysis

(opening position)

20/18 0:04 +0.16-- 1.Nf3 Nf6 2.d4 e6 3.e3 d5 4.Bd3 Be7 5.O-O O-O 6.c4
20/23 0:05 +0.24 1.e4 e5 2.Nf3 Nc6 3.Bb5 Nf6 4.Nc3 Bd6 5.O-O O-O
21/23 0:07 +0.24 1.e4 e5 2.Nf3 Nc6 3.Bb5 Nf6 4.O-O Bd6 5.Nc3 O-O
22/19 0:09 +0.32++ 1.e4 e5 2.Nf3 Nc6 3.Bb5 Nf6 4.O-O Bd6 5.d4 Qe7
22/23 0:10 +0.40++ 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Qe2 Qe7
22/24 0:13 +0.48 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.d4 Be7
23/22 0:26 +0.24-- 1.e4 e6 2.Nf3 d5 3.exd5 exd5 4.d4 Nf6 5.Bb5+ c6
23/30 0:36 +0.20 1.e4 e6 2.Nf3 d5 3.exd5 exd5 4.d4 Nf6 5.Bb5+ c6

So it seems the changes made improve behavior overall and could even be worth some elo, but going beyond that, one would like for Stockfish to protect stored entries (in the example, one would like Stockfish to remember that e4 is 0.28 at depth 23, regardless of the positions analyzed afterwards, until the hash is manually cleared).

Of course, one possible solution of this could be implementing learning (saving the results to disk, not the entire hash, just the PV, score and depth of the position analized).
Top
Jeremy Bernstein
Site Admin
Posts: 1226
Joined: Wed Jun 09, 2010 7:49 am
Real Name: Jeremy Bernstein
Location: Berlin, Germany

Re: Designing an analysis friendly Stockfish?

Post by Jeremy Bernstein »

I just wanted to chime in and congratulate everyone on this effort. Great work!

Jeremy
Top
Jeremy Bernstein
Site Admin
Posts: 1226
Joined: Wed Jun 09, 2010 7:49 am
Real Name: Jeremy Bernstein
Location: Berlin, Germany

Re: Designing an analysis friendly Stockfish?

Post by Jeremy Bernstein »

For OSX:

http://www.expr-i0.net/stockfish_pag_osx.zip

Intel 32- and 64-bit gcc profiled builds in the archive.
Top
Prima
Posts: 328
Joined: Tue Dec 14, 2010 6:12 am

Re: Designing an analysis friendly Stockfish?

Post by Prima »

Uly wrote:
Prima wrote:Just out of curiosity, what does "PA_G" mean in the updated Stockfish 2.0.1 build?
Preserve Analysis G (The G is the alphabetical version).
Thanks Uly for the clarification.
Top
gaard
Posts: 127
Joined: Thu Jun 10, 2010 1:39 am
Real Name: Martin Wyngaarden
Location: Holland, Michigan

Re: Designing an analysis friendly Stockfish?

Post by gaard »

Uly wrote:
Prima wrote:Just out of curiosity, what does "PA_G" mean in the updated Stockfish 2.0.1 build?
Preserve Analysis G (The G is the alphabetical version).

Thanks everyone for your efforts! Stockfish is now showing the wished behavior! Here's cross posts from RF:
Banned for Life wrote:Analyzing with the modified Stockfish is now a pleasure, whereas before it was maddening to watch the evaluations jump up and down, and as you mentioned, if you moved along the PV, the engine would need to fill in the holes in the tt, which seemed like a waste of time. It would be interesting to test this configuration and see how it compares in game play against the default, as Dr. Hyatt's claim that using exact entries adds a few Elo points certainly seems reasonable.

The second step that was discussed was putting in what is referred to in Rybka as "Preserve Analysis" where hash replacement is no longer weighted (or more lightly weighted) by generation. This will allow moving to different root locations without deemphasizing current hash entries. Of course this is only suitable for analysis and will require the operator to manually clear the hash when moving to a different position.
I see what he means, for instance, the root (opening position) shows the wished behavior:

21/26 0:06 +0.40 1.e4 e5 2.Nf3 Nf6 3.d4 Nxe4 4.Bd3 d5 5.dxe5 Be7
22/26 0:08 +0.40 1.e4 e5 2.Nf3 Nf6 3.d4 Nxe4 4.Bd3 d5 5.dxe5 Be7

(e4 forced)

21/25 0:00 +0.40 1...e5 2.Nf3 Nf6 3.d4 Nxe4 4.Bd3 d5 5.dxe5 Be7
22/05 0:03 +0.32++ 1...e5 2.Nf3 Nf6 3.d4 Nxe4 (7.694.059) 2499
22/19 0:07 +0.28 1...e5 2.Nf3 Nf6 3.d4 Nxe4 4.Nxe5 d6 5.Nf3 d5 6.Bd3

Back to root

22/20 0:00 +0.28 1.e4 e5 2.Nf3 Nf6 3.d4 Nxe4 4.Nxe5 d6 5.Nf3 d5
23/08 0:01 +0.28 1.e4 e5 2.Nf3 Nf6 3.d4 Nxe4 4.Nxe5 d6 (3.746.798)

However, it's enough for one to analyze some other position

(1. e4 Nc6 2. d4 d5 3. e5 Bf5 4. Ne2 e6 5. Ng3)

18/21 0:02 +0.16 5...Bg6 6.Bb5 Qd7 7.O-O f6 8.Nd2 a6 9.Bxc6 Qxc6
19/21 0:02 +0.16 5...Bg6 6.Bb5 Qd7 7.O-O f6 8.Nd2 a6 9.Bxc6 Qxc6
20/22 0:03 +0.24-- 5...Bg6 6.Bb5 Qd7 7.O-O f6 8.Nd2 a6 9.Bxc6 Qxc6
20/23 0:05 +0.12 5...Bg6 6.Bb5 Qd7 7.O-O a6 8.Bxc6 Qxc6 9.c3 O-O-O
21/20 0:09 +0.16 5...Bg6 6.Bb5 Qd7 7.O-O a6 8.Bd3 f6 9.f4 O-O-O
22/22 0:13 +0.12 5...Bg6 6.Bb5 Qd7 7.O-O a6 8.Bd3 f6 9.f4 O-O-O
23/21 0:17 +0.12 5...Bg6 6.Bb5 Qd7 7.O-O a6 8.Bd3 f6 9.f4 O-O-O

For Stockfish to forget all the other analysis

(opening position)

20/18 0:04 +0.16-- 1.Nf3 Nf6 2.d4 e6 3.e3 d5 4.Bd3 Be7 5.O-O O-O 6.c4
20/23 0:05 +0.24 1.e4 e5 2.Nf3 Nc6 3.Bb5 Nf6 4.Nc3 Bd6 5.O-O O-O
21/23 0:07 +0.24 1.e4 e5 2.Nf3 Nc6 3.Bb5 Nf6 4.O-O Bd6 5.Nc3 O-O
22/19 0:09 +0.32++ 1.e4 e5 2.Nf3 Nc6 3.Bb5 Nf6 4.O-O Bd6 5.d4 Qe7
22/23 0:10 +0.40++ 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Qe2 Qe7
22/24 0:13 +0.48 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.d4 Be7
23/22 0:26 +0.24-- 1.e4 e6 2.Nf3 d5 3.exd5 exd5 4.d4 Nf6 5.Bb5+ c6
23/30 0:36 +0.20 1.e4 e6 2.Nf3 d5 3.exd5 exd5 4.d4 Nf6 5.Bb5+ c6

So it seems the changes made improve behavior overall and could even be worth some elo, but going beyond that, one would like for Stockfish to protect stored entries (in the example, one would like Stockfish to remember that e4 is 0.28 at depth 23, regardless of the positions analyzed afterwards, until the hash is manually cleared).

Of course, one possible solution of this could be implementing learning (saving the results to disk, not the entire hash, just the PV, score and depth of the position analized).
My pleasure. If we could just get it tested now :) I've PM'ed Ingo asking him so I'll just have to wait and see.
Top
Uly
Posts: 838
Joined: Thu Jun 10, 2010 5:33 am

Re: Designing an analysis friendly Stockfish?

Post by Uly »

I still haven't seen a single game of Stockfish PAG, so I played a 4 games micro-match at 1+3 time controls. I know about the statistics and that the changes weren't made for games, but Stockfish PAG didn't seem to have any problems against Default and won a game (4CPU 64bit 2 Bookless 2 1.e4).








I think it's possibles that these changes become default in the next official release, which would be a triumph of common sense! :)
Top
BB+
Posts: 1484
Joined: Thu Jun 10, 2010 4:26 am

Re: Designing an analysis friendly Stockfish?

Post by BB+ »

There's some way to give a bunch of pgns on one board. I forget what it is. KLO always does it. EDIT: Looks like you just put "[/pgn]" at the end of the last game, and not after the others.

Probably too much to ask, but can you get average depth (and NPS for those who care) stats from these games (or more ideally, from a larger set of games)?
Top
Jeremy Bernstein
Site Admin
Posts: 1226
Joined: Wed Jun 09, 2010 7:49 am
Real Name: Jeremy Bernstein
Location: Berlin, Germany

Re: Designing an analysis friendly Stockfish?

Post by Jeremy Bernstein »

BB+ wrote:There's some way to give a bunch of pgns on one board. I forget what it is. KLO always does it. EDIT: Looks like you just put "[/pgn]" at the end of the last game, and not after the others.

Probably too much to ask, but can you get average depth (and NPS for those who care) stats from these games (or more ideally, from a larger set of games)?
Exactly. Then you get a games menu above the board. In this case:


Top
gaard
Posts: 127
Joined: Thu Jun 10, 2010 1:39 am
Real Name: Martin Wyngaarden
Location: Holland, Michigan

Re: Designing an analysis friendly Stockfish?

Post by gaard »

mcostalba wrote:
gaard wrote:
mcostalba wrote:It would look like this:

Code: Select all

if (tte && (PvNode ? ok_to_use_TT_PV(tte, depth, alpha, beta, ply) : ok_to_use_TT(tte, depth, beta, ply))
 {
    TT.refresh(tte);
    ss->bestMove = ttMove; // Can be MOVE_NONE
    return value_from_tt(tte->value(), ply);
}
I made the changes both in the normal and qsearch parts of search.cpp. I am seeing a slow down of ~15%. Is this to be expected?
The above code should have zero speed impact.

Code: Select all

Stockfish 2.01 JA :600 (+188,=277,-135), 54,4%

Naum 4.2 : 100 (+ 36,= 52,- 12), 62.0 %
Deep Rybka 4 : 100 (+ 19,= 55,- 26), 46.5 %
Deep Shredder 12 : 100 (+ 44,= 41,- 15), 64.5 %
Critter 0.90 SSE42 : 100 (+ 31,= 47,- 22), 54.5 %
Houdini 1.5a : 100 (+ 20,= 39,- 41), 39.5 %
Komodo64 1.3 JA : 100 (+ 38,= 43,- 19), 59.5 %

Stockfish 2.0.1 PA :600 (+184,=286,-130), 54.5 %

Naum 4.2 : 100 (+ 42,= 38,- 20), 61.0 %
Deep Rybka 4 : 100 (+ 12,= 59,- 29), 41.5 %
Deep Shredder 12 : 100 (+ 41,= 52,- 7), 67.0 %
Critter 0.90 SSE42 : 100 (+ 24,= 52,- 24), 50.0 %
Houdini 1.5a : 100 (+ 12,= 47,- 41), 35.5 %
Komodo64 1.3 JA : 100 (+ 53,= 38,- 9), 72.0 %
Ingo was kind enough to run some tests for me. The two scored almost identically against the same 6 opponents and this is from a compile that is ~7% slower than JA's.
Top
Uly
Posts: 838
Joined: Thu Jun 10, 2010 5:33 am

Re: Designing an analysis friendly Stockfish?

Post by Uly »

Thanks gaard and Ingo, that looks very promising.
Top
Post Reply

Return to “Programming and Technical Discussions”