Run engine tests with no opening books.

[Hagen]

I'm curious to see the results.

[Uly]

You'll basically see the same result many times, and it won't show anything but how the engines perform on the opening that they'd choose to play (it's as if you forced that opening and decreased the clock time, so it's pointless.)

A more interesting test is minimal books, where you force the engines to play as close to no books as possible. One way to do it is forcing every single move on the board (1.a4 1.b4 1.c4...) but that leads to alien looking games, I run my tests by running the interesting variations, it'll work as long as you don't mind forcing two half moves for the engines (1.e4 e5, 1.e4 c5, 1.e4 c6...) after you're done forcing interesting variations of one halfmove (1.e4, 1.d4, 1.Nf3...)

[Hagen]

Interesting idea...minimal books. Like forcing the engine to only use maybe up to 3 or 4 moves from the opening book and making it play "blind". For this reason I pay alot of attention to the FRC tournaments done with the engines. FRC as a subject is not theory heavy as it's relatively new ground. Tactically Rybka is the champ in FRC. Although Stockfish is closing fast on Rybka's dominance. If I'm not mistaken...but Stockfish 1.8 conquers Rybka 3. Is this right?

[Swaminathan]

You'll basically see the same result many times, and it won't show anything but how the engines perform on the opening that they'd choose to play (it's as if you forced that opening and decreased the clock time, so it's pointless.)

A more interesting test is minimal books, where you force the engines to play as close to no books as possible. One way to do it is forcing every single move on the board (1.a4 1.b4 1.c4...) but that leads to alien looking games, I run my tests by running the interesting variations, it'll work as long as you don't mind forcing two half moves for the engines (1.e4 e5, 1.e4 c5, 1.e4 c6...) after you're done forcing interesting variations of one halfmove (1.e4, 1.d4, 1.Nf3...)

I like tests with test positions rather than opening books. Test sets such as Nunn, Sedat, Noomen.

Some engines always repeat the same openings out of the their own opening book so it's not much different than games without opening books.

[Hagen]

You mean engine shootouts with FEN postion setups? The only problem here is this can favor one side over the other depending on the position used. For example, I've heard it said that FRC can't be adopted for serious chess playing because the opening setups can favor White overwhelmingly. A simple move and Black is already defending a weak position. Either that or Black ignores the threat and decides on a counter attack instead.

Once again this is pointing up to Chess960 as a viable alternative not only for chess playing but maybe even engine strength tests. Forcing engines to play FRC can allow testers to see which engine really knows tactics. In fact it would be fascinating to see the draw/win/loss percentages in FRC in computer tournaments.

[Uly]

Some engines always repeat the same openings out of the their own opening book so it's not much different than games without opening books.

The point of testing without opening books is to see how the engine behaves on the opening, what is its strength on it, common testing with books and suite positions hide the engines' weaknesses on the opening, and I think it's a reason for why Rybka is really bad on the opening (I think the reason that Rybka is so good at FRC is because of the time management logic being perfect for a game without books, this is still being tested).

turbojuice at Rybka forum ran at some point a test to know the strength of several engines in the opening, I recall he came to the conclusion at the time that Ktulu 8 and Loop 10.32f were better on the opening stage than all the other top engines at the time, but played the rest of the game considerably weaker than the other engines, and all the opening knowledge is wasted and hidden by opening books.

The opening stage has been neglected just because opening books exist, but testing with minimal books is as valid as testing without tablebases, I wonder why it's not more popular (I forgot to say, the idea of minimal books came from Larry Kaufman.)

[kingliveson]

Code: Select all

MS, Blitz:1'+1  2010                   

1   IvanHoe 9.55b w32     +188/=165/-47 67.63%  270.5/400
2   Rybka 3 1-cpu 32-bit  +47/=165/-188 32.38%  129.5/400

- No book/test suite
- Ponder off
- No Egtb
- Hash 64 MB
- AMD Turion(tm) 64 Mobile Technology ML-37
- 1 CPU
- Engines use default settings.
- 1+1 Time control
- No competition

Code: Select all

Games        :    400 (finished)

White Wins   :    146 (36.5 %)
Black Wins   :     89 (22.2 %)
Draws        :    165 (41.2 %)
Unfinished   :      0

White Perf.  : 57.1 %
Black Perf.  : 42.9 %

ECO A =     28 Games ( 7.0 %)
ECO B =    146 Games (36.5 %)
ECO C =      2 Games ( 0.5 %)
ECO D =    210 Games (52.5 %)
ECO E =     14 Games ( 3.5 %)

Futher Breakdown:

Code: Select all

ECO D00 	= 28 %
ECO D02 	= 24 %
ECO B07 	= 30 %
ECO B08 	= 04 %

Game data available: http://chess.cygnitec.com/pgn/

Data above is good enough reason not to test without books or test suites. You will end up playing less than a handful ECO variation. It is best in my opinion to use book/test suite (test suite preferable) with 4 <= moves <= 8 since most variation can be achieved after the 4th move.

P.S. Looking at the result, it is clear one is a clone.

[hyatt]

You mean engine shootouts with FEN postion setups? The only problem here is this can favor one side over the other depending on the position used. For example, I've heard it said that FRC can't be adopted for serious chess playing because the opening setups can favor White overwhelmingly. A simple move and Black is already defending a weak position. Either that or Black ignores the threat and decides on a counter attack instead.

Once again this is pointing up to Chess960 as a viable alternative not only for chess playing but maybe even engine strength tests. Forcing engines to play FRC can allow testers to see which engine really knows tactics. In fact it would be fascinating to see the draw/win/loss percentages in FRC in computer tournaments.

All you have to do is to take each position and play it twice, giving each opponent the chance to play both sides. If you get a biased position, it won't influence the results at all, since each side will win when they have the bias.

I have 4,000 such positions that are roughly 12 moves into the game, on my ftp box. These are what I use to play games during our cluster testing, since I only care about engine improvements and don't give a hoot about the books and such. I just give the programs starting positions, and they are allowed no opening books at all beyond that starting position.

[Cubeman]

I think the suggestion that FRC has positions that are too favourable to the White side is only a theory.Some one did a test on all the 960 starting positions with high powered hardware and high quality chess software and came to the conclusion that all positions were playable for Black.The evaluations never got above 0.4 and the same could be said of normal chess.

[Edmund]

I think the suggestion that FRC has positions that are too favourable to the White side is only a theory.Some one did a test on all the 960 starting positions with high powered hardware and high quality chess software and came to the conclusion that all positions were playable for Black.The evaluations never got above 0.4 and the same could be said of normal chess.

According to CCRLthere is indeed a position that has a white score of 68% (RKNBNQBR).