OpenChess

For the period of 2006-10 or so, most of the bookmakers used Rybka as their main engine. The books they created were then used by various ratings groups. Does anyone have any sense of whether this could lead to bias, one way or the other?

BB+ wrote:For the period of 2006-10 or so, most of the bookmakers used Rybka as their main engine. The books they created were then used by various ratings groups. Does anyone have any sense of whether this could lead to bias, one way or the other?

Since presumably Rybka judged each position when coming out of book as approximately equal, one would assume there would not be any bias for Rybka. I guess that some positions were such that only Rybka could see the line of play that would ensure equality coming out of book, but I also have to assume that there were positions that Rybka misjudges. In other words, I do not think that a significant portion of the resulting positions were such that Rybka held an advantage over every other engine. So, I do not see any engine-specific bias.

I will think about this more.

I have a different take on this. Tuning a book to a specific program is a really big advantage. Some (Vincent Diepeveen, for example) claim > 100 Elo for really good tournament books. And he has given lots of examples to support that. The issue is that not all programs play all types of positions (and therefore openings) equally well. Just because you guide a program into a line that is "relatively equal" (more on this in a minute) doesn't mean you have equal chances. The opponent might not understand this at all.

"relatively equal" is an extremely interesting topic. You really do NOT want to play toward positions where the score is equal, and things are balanced. That's known as a "draw". What you REALLY want to do is play toward positions where things are unbalanced, but with a reasonable score. For example, you have the opportunity for a king-side attack, but your opponent is up a pawn or has more space on the queenside. If your program is a good attacker, this position (with a fairly equal score) would be a perfect position to play toward. If you don't handle king safety very well, it would be a terrible position to play toward.

Bottom line is that guiding a program into positions that it is particularly adept at handling pays dividends. Significant dividends...

It takes a LOT of time to produce the books however. That's why there are full-time book makers busy at work...

1. I think 'tuning a book to a specific program' (in the sense of trying to steer it into book exit positions which it handles particularly well) was not the issue here, at least if I understood BB+'s post correctly. Rather, the book makers created books which were based on a) analyses with Rybka and b) (I guess mainly) a huge number of Rybka games played on the Chessbase server. (I would suppose that these book makers did not intend to create opening books particularly suitable for Rybka.)
I don't really see how this process would lead to a bias in favor of Rybka.
2. The question of to what extent book tuning in favor of a specific engine is possible is nevertheless interesting. The best way to do this would probably be based on success statistics with respect to specific ECO codes. Do the rating lists provide any information of this kind?

Kappatoo wrote:1. I think 'tuning a book to a specific program' (in the sense of trying to steer it into book exit positions which it handles particularly well) was not the issue here, at least if I understood BB+'s post correctly. Rather, the book makers created books which were based on a) analyses with Rybka and b) (I guess mainly) a huge number of Rybka games played on the Chessbase server. (I would suppose that these book makers did not intend to create opening books particularly suitable for Rybka.)
I don't really see how this process would lead to a bias in favor of Rybka.

I think, in most cases, the source games have been IM and GM level human games that have been filtered by various criteria and then analyzed by Rybka.

Kappatoo wrote: 2. The question of to what extent book tuning in favor of a specific engine is possible is nevertheless interesting. The best way to do this would probably be based on success statistics with respect to specific ECO codes. Do the rating lists provide any information of this kind?

The CCRL 40/40 list has games sorted by ECO and by engine:
http://computerchess.org.uk/ccrl/4040/e ... y_eco.html

Kappatoo wrote:1. I think 'tuning a book to a specific program' (in the sense of trying to steer it into book exit positions which it handles particularly well) was not the issue here, at least if I understood BB+'s post correctly. Rather, the book makers created books which were based on a) analyses with Rybka and b) (I guess mainly) a huge number of Rybka games played on the Chessbase server. (I would suppose that these book makers did not intend to create opening books particularly suitable for Rybka.)
I don't really see how this process would lead to a bias in favor of Rybka.
2. The question of to what extent book tuning in favor of a specific engine is possible is nevertheless interesting. The best way to do this would probably be based on success statistics with respect to specific ECO codes. Do the rating lists provide any information of this kind?

If you tune the book to choose lines that Rybka likes, how is that NOT tuning the book to rybka? That is what most of us do to create tournament-style books. Play selected openings manually, looking for any "non-known-book move" that is playable and which the particular program seems to like and handle well based on a batch of quick test games... It is slow, but it produces big results. I have seen WCCC games where one side was mated, and the other side was STILL in book, due to a deeply prepared opening trap...

hyatt wrote:

Kappatoo wrote:1. I think 'tuning a book to a specific program' (in the sense of trying to steer it into book exit positions which it handles particularly well) was not the issue here, at least if I understood BB+'s post correctly. Rather, the book makers created books which were based on a) analyses with Rybka and b) (I guess mainly) a huge number of Rybka games played on the Chessbase server. (I would suppose that these book makers did not intend to create opening books particularly suitable for Rybka.)
I don't really see how this process would lead to a bias in favor of Rybka.
2. The question of to what extent book tuning in favor of a specific engine is possible is nevertheless interesting. The best way to do this would probably be based on success statistics with respect to specific ECO codes. Do the rating lists provide any information of this kind?

If you tune the book to choose lines that Rybka likes, how is that NOT tuning the book to rybka? That is what most of us do to create tournament-style books. Play selected openings manually, looking for any "non-known-book move" that is playable and which the particular program seems to like and handle well based on a batch of quick test games... It is slow, but it produces big results. I have seen WCCC games where one side was mated, and the other side was STILL in book, due to a deeply prepared opening trap...

We are discussing the books used by the ratings agencies. I guess in this context we are really talking about the CEGT and the CCRL. Those books are not tournament-style books. They are made in a way to try to ensure the position, when coming out of book, has equal chances for White and for Black (some are more successful than others). They are not tuned to suit Rybka. Rather, when Rybka was the strongest engine it was used to judge if the positions were fairly neutral (in terms of centipawns). The question, I believe, was concerning whether using Rybka in this manner gave Rybka some advantage in engine-engine matches conducted by the rating agencies. Hence the title of this thread," "Neutral Book" bias?".

Let's think about books that were constructed by only using Rybka for evaluation. Any position that Rybka scored higher than a predetermined number of centipawns, say 50 to 75, would probably be removed. Or very lightly weighted (I don't know the exact method used by book makers to handle such positions; I deal with PGNs and EPDs myself). Therefore, the predominant positions that occur when coming out of book would be judged by Rybka to be neutral. In any case, the CCRL uses a filter to remove games when an engine's score is -75 centipawns (IIRC) coming out of book and it does not improve during the game. So, if an engine really does not like an opening, it gets to refuse it (in a limited sense).

The main point is that, even though Rybka was used to help create books for engine testing, it was done in a way that seems to minimize bias for Rybka. Perhaps some subtle bias did creep in, but I am not certain on how it could nor am I certain of the amount of effect it would have had on subsequent ratings.

I think there is a misunderstanding here.
If by 'likes' you mean 'handles better than other engines', then yes, this would be tuning a book to Rybka. One can even define it this way. Let's say that 'tuning a book to x' means 'buliding a book which tends to have exit positions which x handles better than other engines'.
I was saying that the book makers which BB+ had in mind did not try to tune their books to Rybka intentionally. (I take it that he made the same assumption, since he said

Does anyone have any sense of whether this could lead to bias, one way or the other?

(emphasis mine).)
Rather, what these book makers did was use Rybka games and analyses to create opening books, and the question was whether this process could lead to books tuned to (or against) Rybka. My (not very deep) answer to this question was 'I don't see how'.

I have seen WCCC games where one side was mated, and the other side was STILL in book, due to a deeply prepared opening trap...

That's a different story, which should be irrelevant for rating lists.

Adam Hair wrote:The CCRL 40/40 list has games sorted by ECO and by engine:
http://computerchess.org.uk/ccrl/4040/e ... y_eco.html

Thanks. The relevant information could easily be extracted from the games. Bot the sample size would be much too small.

Probably the easiest to understand example centers around gambits. We used to play a lot of them OTB when we had very fast hardware compared to everyone else, inviting them into "a house of horizon effects." Our eval was tuned so that for some of these 1 pawn gambits, our score was still positive. While competitors would have a score of +1.0 for themselves.

What do you think would happen if you used my program to evaluate book lines and flagged these gambit lines as "appears to be pretty equal from Cray Blitz's perspective"?? You have included openings that individual programs might do "OK" with because they are all pretty equal in tactics, but then Cray Blitz drops by with an extra 3-4-5 plies of search, and rips them all using those openings.

So the potential is there, whether it was actually realized or not is really hard to judge.