New builds, based on PA_J + GTB, attached. github updated.
Testing so far seems good. I chucked the tournament I was running in favor of a very fast tournament when I discovered an issue with the opening books (I was using Stockfish's own book for all 3 engines. Apparently Stockfish only knows how to play the Najdorf.). I guess that this is why I don't generally run tests -- too dumb. Stockfish_64 and Stockfish_PA_I_GTB_64 were neck in neck, with PA_J_64 about 6 games behind. With a round-robin tourney of 75 games, that could mean anything.
In general, very drawish play between these 3 engines, which is to be expected. I don't think the modifications have much, if any effect on strength, time management, etc. whatsoever, at least not at these time controls.
BTW, I increased the number of cores on my VM to 2, so we're seeing multicore runs here.
New tourney. 5 seconds + 100ms, 300 games total which makes for a better sample size. I suspect that the PA_J is a tiny bit weaker than PA_I based on what I was seeing in the 1/1 games, but let's see how it fares at these time controls.
Designing an analysis friendly Stockfish?
-
Jeremy Bernstein
- Site Admin
- Posts: 1226
- Joined: Wed Jun 09, 2010 7:49 am
- Real Name: Jeremy Bernstein
- Location: Berlin, Germany
Re: Designing an analysis friendly Stockfish?
- Attachments
-
- Stockfish_PA_J_GTB.7z
- (543.37 KiB) Downloaded 215 times
-
Jeremy Bernstein
- Site Admin
- Posts: 1226
- Joined: Wed Jun 09, 2010 7:49 am
- Real Name: Jeremy Bernstein
- Location: Berlin, Germany
Re: Designing an analysis friendly Stockfish?
In case anyone else wants to run some tests, here are the 3 engines I'm working with, all compiled with exactly the same settings.
- Attachments
-
- Brueder.7z
- (430.91 KiB) Downloaded 202 times
-
Jeremy Bernstein
- Site Admin
- Posts: 1226
- Joined: Wed Jun 09, 2010 7:49 am
- Real Name: Jeremy Bernstein
- Location: Berlin, Germany
Re: Designing an analysis friendly Stockfish?
Here's a game:
Note how the PA_J+GTB finds a forced mate around move 80 or so and then loses its way by move 99. What happened?
Note how the PA_J+GTB finds a forced mate around move 80 or so and then loses its way by move 99. What happened?
-
gaard
- Posts: 127
- Joined: Thu Jun 10, 2010 1:39 am
- Real Name: Martin Wyngaarden
- Location: Holland, Michigan
Re: Designing an analysis friendly Stockfish?
Any results with the longer time control game for this fix? These updates to history/killers are in version I. Against the default with the new ok_to_use_tt() SF I have a score of +235 -140 =156 from the perspective of the default version at g/24"fruity wrote:The change regarding history and killer moves seems to work well only at lower depths. I'm actually running a 1000 games 20 sec. match, 6 Threads. SF default is clearly in front after 800 games. I'll test SF PA_G against SF default at longer tc tonight.Uly wrote:Thanks fruity and gaard, here's a quick comparison of Stockfish PA_G and PA_H at 1CPU in the opening position. It seemed mostly luck, but PA_H found 1...e6 much earlier and reached depth 24 much faster, PA_G caught up at depth 25 and finished one minute faster in the end, though.
Engine: Stockfish 2.0.1 PA_G (128 MB) by Tord Romstad, Marco Costalba, Joona Kii 15/14 0:00 +0.16-- 1.Nf3 Nf6 2.e3 d5 3.d4 e6 4.Bd3 c5 5.O-O c4 6.Be2 Nc6 7.Nc3 Bd6 (509.807) 679 15/09 0:01 +0.32++ 1.e4 Nf6 2.e5 Nd5 3.Nf3 Nc6 4.d4 e6 5.Bd3 (928.123) 715 15/09 0:01 +0.40++ 1.e4 Nf6 2.e5 Nd5 3.Nf3 Nc6 4.d4 e6 5.Bd3 (1.152.385) 730 15/09 0:01 +0.56++ 1.e4 Nf6 2.e5 Nd5 3.Nf3 Nc6 4.d4 e6 5.Bd3 (1.424.940) 741 15/16 0:03 +0.44 1.e4 e5 2.Nf3 Bd6 3.Nc3 Nf6 4.Be2 Nc6 5.O-O O-O 6.d3 b6 7.d4 exd4 8.Nxd4 Bb7 (2.247.548) 749 16/18 0:04 +0.24-- 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Bd3 Nf6 6.O-O Be7 7.Nc3 O-O 8.Re1 Nc6 9.Ng5 d5 (3.151.485) 746 16/26 0:05 +0.32 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Qe2 Qe7 6.Nc3 Nxc3 7.dxc3 Qxe2+ 8.Bxe2 Nc6 9.Be3 Be7 10.O-O-O O-O 11.Kb1 Be6 12.Ng5 a6 13.Nxe6 fxe6 (4.052.349) 747 17/26 0:05 +0.32 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Qe2 Qe7 6.Nc3 Nxc3 7.dxc3 Qxe2+ 8.Bxe2 Nc6 9.Be3 Be7 10.O-O-O O-O 11.Kb1 Be6 12.Ng5 a6 13.Nxe6 fxe6 (4.252.062) 745 18/26 0:06 +0.32 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Qe2 Qe7 6.Nc3 Nxc3 7.dxc3 Qxe2+ 8.Bxe2 Nc6 9.Be3 Be7 10.O-O-O O-O 11.Kb1 Be6 12.Ng5 a6 13.Nxe6 fxe6 (4.601.681) 747 19/25 0:07 +0.40++ 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Qe2 Qe7 6.Nc3 Nxc3 7.dxc3 Qxe2+ 8.Bxe2 Nc6 9.Be3 Be7 10.O-O-O O-O 11.Kb1 Be6 12.Ng5 a6 13.h4 (5.619.232) 750 19/19 0:09 +0.48++ 1.e4 e5 2.Nf3 Nf6 3.d4 exd4 4.e5 Qe7 5.Be2 Ng4 6.Qxd4 d6 7.exd6 cxd6 8.Nc3 Nc6 9.Qa4 Qe6 10.O-O (6.811.131) 754 19/26 0:11 +0.32 1.e4 e5 2.Nf3 Nf6 3.d4 exd4 4.e5 Qe7 5.Be2 Ng4 6.Qxd4 h5 7.Nc3 Nc6 8.Qf4 Ncxe5 9.O-O c6 10.Re1 d6 11.h3 Nxf3+ 12.Qxf3 Ne5 13.Qf4 Be6 (8.935.601) 753 20/26 0:13 +0.32 1.e4 e5 2.Nf3 Nf6 3.d4 exd4 4.e5 Qe7 5.Be2 Ng4 6.Qxd4 h5 7.Nc3 Nc6 8.Qf4 Ncxe5 9.O-O c6 10.Re1 d6 11.h3 Nxf3+ 12.Qxf3 Ne5 13.Qf4 Be6 (10.320.940) 752 21/26 0:17 +0.32 1.e4 e5 2.Nf3 Nf6 3.d4 exd4 4.e5 Qe7 5.Be2 Ng4 6.Qxd4 h5 7.Nc3 Nc6 8.Qf4 Ncxe5 9.O-O c6 10.Re1 d6 11.h3 Nxf3+ 12.Qxf3 Ne5 13.Qf4 Be6 (13.161.101) 750 22/17 0:28 +0.40++ 1.e4 e5 2.Nf3 Nf6 3.d4 exd4 4.e5 Qe7 5.Be2 Ng4 6.Qxd4 h5 7.Nc3 Nc6 8.Qd1 Nxf2 9.Kxf2 (20.919.656) 744 22/26 0:35 +0.24-- 1.e4 e5 2.Nf3 Nf6 3.d4 exd4 4.e5 Ne4 5.Qxd4 f5 6.exf6 Nxf6 7.Nc3 Nc6 8.Qe3+ Be7 9.Bd3 O-O 10.O-O d5 11.Qg5 d4 12.Bc4+ Kh8 13.Rd1 Nd7 (26.047.732) 742 22/25 0:48 +0.44 1.e4 e5 2.Nf3 Nc6 3.Bc4 Be7 4.Nc3 Nf6 5.O-O O-O 6.d4 exd4 7.Nxd4 Bb4 8.Bg5 Bxc3 9.bxc3 Qe7 10.Bd3 d6 11.Rb1 h6 12.Bh4 Re8 13.f3 (35.672.600) 742 23/26 2:12 +0.36 1.e4 e5 2.Nf3 d6 3.d4 exd4 4.Nxd4 Nf6 5.Nc3 Be7 6.Be2 O-O 7.Bf4 Bd7 8.O-O Nc6 9.Qd2 Ne5 10.Rad1 Re8 11.Bg3 a6 12.f4 Nc6 13.Bf2 Nxd4 (96.852.976) 733 24/16 3:09 +0.20-- 1.e4 e6 2.Nf3 d5 3.e5 Ne7 4.Bb5+ Bd7 5.Bxd7+ Nxd7 6.O-O Nc6 7.d4 Be7 8.c3 O-O (138.809.221) 731 24/28 3:51 +0.24 1.e4 e6 2.Nf3 d5 3.exd5 exd5 4.d4 Nf6 5.Bb5+ c6 6.Bd3 Be7 7.O-O O-O 8.Nc3 Bd6 9.a3 Re8 10.Bg5 Bg4 11.h3 Bh5 12.g4 Bg6 13.Bxg6 hxg6 14.Re1 (168.932.447) 730 25/31 4:50 +0.20 1.e4 e6 2.Nf3 d5 3.exd5 exd5 4.d4 Nf6 5.Bb5+ c6 6.Bd3 Be7 7.O-O O-O 8.Nc3 Bd6 9.a3 Re8 10.Re1 Bg4 11.Rxe8+ Qxe8 12.Bg5 Nbd7 13.h3 Bh5 14.g4 (212.238.135) 730 best move: e2-e4 time: 5:06.000 min
Engine: Stockfish 2.0.1 PA_H (128 MB) by Tord Romstad, Marco Costalba, Joona Kii 16/16 0:00 +0.28-- 1.Nf3 Nf6 2.e3 e6 3.d4 d5 4.Bd3 Be7 5.O-O Nc6 6.c4 O-O 7.cxd5 exd5 8.Nc3 Be6 (497.513) 740 16/13 0:01 +0.44++ 1.e4 Nc6 2.d4 Nf6 3.e5 Nd5 4.c4 Nb6 5.Nf3 d6 6.exd6 cxd6 7.Nc3 (1.211.455) 760 16/13 0:02 +0.52++ 1.e4 Nc6 2.d4 Nf6 3.e5 Nd5 4.c4 Nb6 5.Nf3 d6 6.exd6 cxd6 7.Nc3 (1.549.036) 768 16/18 0:03 +0.44 1.e4 e6 2.Nf3 d5 3.e5 Nd7 4.Nc3 f6 5.d4 Bb4 6.Bd3 c5 7.O-O cxd4 8.Nxd4 Bxc3 9.bxc3 Nxe5 (2.323.010) 770 17/19 0:04 +0.52 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Nc3 Nxc3 6.dxc3 Be7 7.Bc4 c6 8.O-O d5 9.Bb3 O-O 10.Bf4 (3.581.756) 764 18/20 0:05 +0.40-- 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Nc3 Nxc3 6.dxc3 Be7 7.Bc4 c6 8.O-O d5 9.Bb3 O-O 10.Bf4 Be6 (4.032.770) 763 18/18 0:05 +0.28-- 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Nc3 Nxc3 6.dxc3 Be7 7.Bc4 Nc6 8.O-O O-O 9.Qd5 Be6 (4.541.806) 764 18/24 0:08 +0.32 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Nc3 Nxc3 6.dxc3 Be7 7.Bc4 Nc6 8.O-O O-O 9.Re1 Bf5 10.Bf4 Be6 11.Bd5 Bh4 12.Qd3 Qf6 (6.262.735) 761 19/24 0:08 +0.32 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Nc3 Nxc3 6.dxc3 Be7 7.Bc4 Nc6 8.O-O O-O 9.Re1 Bf5 10.Bf4 Be6 11.Bd5 Bh4 12.Qd3 Qf6 (6.726.012) 764 20/24 0:10 +0.32 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Nc3 Nxc3 6.dxc3 Be7 7.Bc4 Nc6 8.O-O O-O 9.Re1 Bf5 10.Bf4 Be6 11.Bd5 Bh4 12.Qd3 Qf6 (8.026.327) 764 21/24 0:14 +0.32 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Nc3 Nxc3 6.dxc3 Be7 7.Bc4 Nc6 8.O-O O-O 9.Re1 Bf5 10.Bf4 Be6 11.Bd5 Bh4 12.Qd3 Qf6 (10.885.253) 761 22/24 0:21 +0.24-- 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Nc3 Nxc3 6.dxc3 Be7 7.Bc4 Nc6 8.O-O O-O 9.Re1 Bf5 10.Bf4 Bf6 11.Nd4 Bxd4 12.cxd4 Nb4 (16.389.829) 755 22/19 0:31 +0.40++ 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.d4 d5 6.Bd3 Be7 7.O-O O-O 8.Re1 Nc6 9.c4 Bb4 10.cxd5 (23.765.595) 751 22/24 1:03 +0.20 1.e4 e6 2.Nf3 d5 3.exd5 exd5 4.d4 Nf6 5.Bd3 Nc6 6.O-O Be7 7.Nc3 Bg4 8.Be3 O-O 9.h3 Bh5 10.Re1 Re8 11.a3 Bd6 12.Bf5 Bg6 (47.007.795) 744 23/25 1:17 +0.32++ 1.e4 e6 2.Nf3 d5 3.exd5 exd5 4.d4 Nf6 5.Bd3 Nc6 6.O-O Be7 7.c3 O-O 8.Bf4 Bd6 9.Bxd6 Qxd6 10.Nbd2 Re8 11.Qc2 Qf4 12.Rfe1 Bg4 13.Qb3 (57.654.660) 741 23/22 1:28 +0.20 1.e4 e6 2.Nf3 d5 3.exd5 exd5 4.d4 Nf6 5.Bd3 Bd6 6.Nc3 Nc6 7.O-O O-O 8.Bg5 Nb4 9.Nb5 Bg4 10.Re1 h6 11.Nxd6 Qxd6 (65.272.603) 739 24/10 1:45 +0.28++ 1.e4 e6 2.Nf3 d5 3.exd5 exd5 4.d4 Nf6 5.Bd3 Bd6 (77.754.590) 736 24/26 1:57 +0.28 1.e4 e6 2.Nf3 d5 3.exd5 exd5 4.d4 Nf6 5.Bd3 Nc6 6.O-O Be7 7.Bf4 Bg4 8.c3 O-O 9.Nbd2 Bd6 10.Bxd6 Qxd6 11.Qb3 b6 12.Rae1 Rae8 13.h3 Bh5 (86.755.475) 735 25/20 2:44 +0.20-- 1.e4 e6 2.Nf3 d5 3.exd5 exd5 4.d4 Nf6 5.Bd3 Nc6 6.O-O Be7 7.Bf4 O-O 8.Nc3 Bg4 9.Nb5 a6 10.Nxc7 Rc8 (120.464.810) 733 25/18 3:49 +0.12-- 1.e4 e6 2.Nf3 d5 3.exd5 exd5 4.Be2 Nf6 5.O-O Bd6 6.d4 O-O 7.Nc3 Re8 8.Bg5 c6 9.Be3 Qe7 (168.546.917) 733 25/24 6:10 +0.20 1.e4 e6 2.Nf3 d5 3.exd5 exd5 4.Nc3 Nf6 5.d4 Bb4 6.Bd3 Qe7+ 7.Be3 Ng4 8.Qe2 Nxe3 9.Qxe3 Nc6 10.O-O Bxc3 11.bxc3 Qxe3 12.fxe3 O-O (272.208.066) 734 best move: e2-e4 time: 6:12.578 min
-
Jeremy Bernstein
- Site Admin
- Posts: 1226
- Joined: Wed Jun 09, 2010 7:49 am
- Real Name: Jeremy Bernstein
- Location: Berlin, Germany
Re: Designing an analysis friendly Stockfish?
I would imagine that fast time controls would amplify any slowdown due to additional tablebase or hashing activity. In which case, all seems well. I'm running a 450 game round-robin at 5 minutes per game. I'll post about that tomorrow.
Engine Score St St St S-B Stockfish_64 57,5/108 ······················································ 11=111101=1111==0001100=1110000000011100=100100010000= 010=1101001000=011101101100111111111101=1001100=101011 3002,7 Stockfish_PA_J_GTB_64 52,5/108 00=000010=0000==1110011=0001111111100011=011011101111= ······················································ 111011==100011=01000110100000=11==10110010000010110000 2886,7 Stockfish_PA_I_GTB_64 52,0/108 101=0010110111=100010010011000000000010=0110011=010100 000100==011100=10111001011111=00==01001101111101001111 ······················································ 2840,0 162 of 300 games played Level: Blitz 0:05/0,1 Hardware: Intel(R) Xeon(R) CPU W3520 @ 2.67GHz 2659 MHz with 1,0 GB Memory Operating system: Windows 7 Professional Professional (Build 7600) 64 bit
-
gaard
- Posts: 127
- Joined: Thu Jun 10, 2010 1:39 am
- Real Name: Martin Wyngaarden
- Location: Holland, Michigan
Re: Designing an analysis friendly Stockfish?
http://dl.dropbox.com/u/11904592/stockfish_def_G_J.7zUly wrote:What time control are you using? This may be exclusive to incremental time controls (or nothing at all, but Rybka's time management specially sucks at incremental TC, so one is used to seeing patterns).
Bozo the Clown at RF suggested to make a new compile of Stockfish Default and use it for the testing, instead of JA's compile, to compare apples to apples (speed of the compile would be a non factor, clear improvements would stand as better code.)
Good idea. These are all compiled under the same conditions and are the most promising versions, IMO.
-
Jeremy Bernstein
- Site Admin
- Posts: 1226
- Joined: Wed Jun 09, 2010 7:49 am
- Real Name: Jeremy Bernstein
- Location: Berlin, Germany
Re: Designing an analysis friendly Stockfish?
Engine Score St St St S-B Stockfish_PA_I_GTB_64 103,5/200 ···································································································· 000100==011100=10111001011111=00==0100110111110100111101110101010111=11101011==11111111=0=1111101111 101=0010110111=100010010011000000000010=0110011=0101000==000=0=110000011000010=1==00100101=100101010 10175, Stockfish_PA_J_GTB_64 98,5/200 111011==100011=01000110100000=11==1011001000001011000010001010101000=00010100==00000000=1=0000010000 ···································································································· 00=000010=0000==1110011=0001111111100011=011011101111=110=111=1110111111001=00111111=1110001=1111111 9848,2 Stockfish_64 98,0/200 010=1101001000=011101101100111111111101=1001100=1010111==111=1=001111100111101=0==11011010=011010101 11=111101=1111==0001100=1110000000011100=100100010000=001=000=0001000000110=11000000=0001110=0000000 ···································································································· 9958,0 300 games played / Tournament is finished
- Attachments
-
- SF SBRR.7z
- PGNs
- (416.76 KiB) Downloaded 167 times
Re: Designing an analysis friendly Stockfish?
I'm still fiddling around. According to my tests at 20'' all the versions so far are 10-15 Elo below SF default. Best result I've found so far isgaard wrote: Any results with the longer time control game for this fix? These updates to history/killers are in version I. Against the default with the new ok_to_use_tt() SF I have a score of +235 -140 =156 from the perspective of the default version at g/24"
(1) remove the history/killer tweak. It seems to work well only for low depths. Could be reintroduced later with tweaking for low depths only, but at the moment it doesn't help to find the best PV TT code, so leave it away for now.
(2) use template and inline for ok_to_use_TT() and the following code
Code: Select all
template <NodeType PvNode>
inline bool ok_to_use_TT(const TTEntry* tte, Depth depth, Value alpha, Value beta, int ply) {
Value v = value_from_tt(tte->value(), ply);
return
PvNode ? tte->depth() >= depth && tte->type() == VALUE_TYPE_EXACT && v < beta && v > alpha
: ( tte->depth() >= depth
|| v >= Max(value_mate_in(PLY_MAX), beta)
|| v <= Min(value_mated_in(PLY_MAX), alpha)
)
&&
(
((tte->type() & VALUE_TYPE_LOWER) && v >= beta)
|| ((tte->type() & VALUE_TYPE_UPPER) && v <= alpha)
);
}
It seems there is still work left to find a real neat solution regarding the playing strength.
-
Jeremy Bernstein
- Site Admin
- Posts: 1226
- Joined: Wed Jun 09, 2010 7:49 am
- Real Name: Jeremy Bernstein
- Location: Berlin, Germany
Re: Designing an analysis friendly Stockfish?
I'd guess it's probably a bug in the GTB probing. Is the Decemberists' patch applied?Jeremy Bernstein wrote:Here's a game:
...
Note how the PA_J+GTB finds a forced mate around move 80 or so and then loses its way by move 99. What happened?
I added a UCI option to control the depth in ply it's used up to. With my default setting of 20 (which could/should be tweaked a bit), it's tad better than SF default at 40 moves/1 second.fruity wrote:(1) remove the history/killer tweak. It seems to work well only for low depths. Could be reintroduced later with tweaking for low depths only, but at the moment it doesn't help to find the best PV TT code, so leave it away for now.
I've noticed that our SF tends to lose on time more often than the default at hyperbullet (40 moves/1 second), especially in the first 4-5 games. Has anyone else noticed this behavior?
Peter