Questions for BB about Rybka PST = Fruit PST

[mballicora]

Miguel, I respect what you're trying to do, to show that the PST similarity may not be so significant. If it were the only similarity, all this debate about whether or not the PSTs could end up being structurally similar if they were independently developed from first principles, would be more interesting. I think Mark W. and Zach both acknowledged in their reports that the PSTs by themselves were not strong evidence of copying (but I dont have them handy right now to go back and check).

There is nothing wrong with looking at one evidence at the time. What prompted me to look at this was the exaggerated claims I saw in R forum. I started with:

"Zach's speech is different than Bob. Zach's evidence, which involved a lot of careful work, is compatible with Zachs' theory that Vas started with Fruit, and kept modifying it until obtaining R1. Note that I said it is compatible, if it is a demonstration, it is another issue (let's not go into that yet). In other words, Zach's evidence and his interpretation is a valid explanation. The key question here should be: Is it the only explanation? That is where the effort should be placed. Bob's speech imply a blatant cut and paste."

The more I looked, the more unconvinced I became. At this point, I believe that Zach PST evidence is very important for 1) historical reasons. It shows Fruit's influence on R1 and that he at least learned from the PSTs. 2) It does not allow Vas to say that PST values are proof that this part of the engine is completely detached from Fruit. But, I do not believe they are evidence for incrimination. In this respect, I believe they have zero value. Zach destroyed an alibi, but that does not mean the accused was in the crime scene, to give a silly analogy. If you really believe this evidence has any weight, you should really consider that Stockfish has copied Fruit (still legal, though, but legality is a different point).

Miguel

But the PST similarity is only one element of the eval that ended up being extremely similar between Rybka 1.0 Beta and Fruit 2.1. And if you consider the totality of that evidence, and apply the principle of Occam's Razor, the obvious conclusion is that Vas simply copied them from Fruit along with everything else, scaled the weights to finer values, and tuned them or adjusted them.

I considered the evidence of the evaluation features much more damning than the PSTs, which are only one piece of the puzzle here (though it does fit with the rest).

[mballicora]

I guess I am missing something here. You are suggesting that the PST contents were independently arrived at, despite the fact that they can be generated using the PST initialization code from Fruit with only tweaking of the constants, and that the PSTs of other engines from that period can not, and that Vas reproduced every feature from Fruit's evaluation in his own program, in the same order as they appeared in Fruit?

Give me a break. The most probable explanation is that he copied it.

By the way, the ICGA made a ruling on all of this already. You're fighting the last war. All of the evidence presented to the panel was available to Vas. He was invited, multiple times, to come and view it, and make whatever statements he wanted about it. He was sent a copy of the reports by Zach and Mark W. He chose to make no effort to defend himself. He didn't raise any argument about how he had independently created his own PST from first principles, and how even though he had been through Fruit forwards and backwards and "took many things", the PST initialization code from Fruit was not one of them.

There's no point in second-guessing now, the investigation is over and Vas is banned and I don't think that's going to change. But since you guys obviously care about these issues, the next time the panel is given a case to investigate, you should participate. I'm sure your input would be welcome, and if you are on the panel then your arguments will get considered while they still have a chance to affect the outcome.

I guess that this paragraph addresses me too (I do not know how to see threads but it looks like it is an answer to Ed).

1) you do not need any reason to give an opinion about evidence that is presented to the public.
2) The fact that Vas was found guilty is irrelevant. Even if he is guilty, there are degrees of guilt and I saw that the balance was tilted towards not even recognizing his contributions to CC and his demonization in the press.
3) I said beforehand that at least two people from the secretariat should have recused themselves. I cannot participate in that situation. I am very glad I did not even sign up to the panel forum, otherwise, my name would have been attached to the document as if I signed it and approved it.

Miguel

[wgarvin]

I've looked back through the reports and I did not find any disclaimer about the PSTs. So I remembered that incorrectly. My apologies.

To Ed and Miguel, I think its unfortunate that you don't want to participate. But I don't think the panel on Rybka was the witch-hunt that you guys seem to think it was. Also, the panel did not punish Vas. All it did was examine the collected evidence, discuss and debate and then vote on whether we were convinced that Rybka had violated tournament rules. Several members, while voting, made specific statements which ended up being quoted in the Secretariat's report. My own reaction was quoted in there, which was a bit unexpected but I don't think there was anything unfair about that. If Vas had chosen to make any statements, I'm sure they would have been included too.

This report was then sent to the ICGA board, who sent it to Vas and gave him a month to respond before it was made public. He didn't bother to reply, and there's no evidence that he even bothered to read the report. If he was harshly condemmed, that may be part of the reason. What more should the ICGA have done? He didn't cooperate. He didn't respond meaningfully to any of the attempts to engage him and find out his side of the story. A small mountain of circumstantial evidence was presented, and he opted not to defend himself (not even to sneer at the process, as you and others have been doing ever since). Vas has nobody to blame for the outcome but himself, and those who complain about the process now are kind of missing the point. Its fine to throw stones, but if you refuse to participate and try to make the process better, then your criticisms are not going to affect much. If you had joined the panel, your arguments about the PSTs would have been heard and debated while it still mattered.

[BB+]

I won't dispute the mathematics of the issue, as I think Miguel and I largely agree there (though maybe don't state it the same way). However, I think the following is an incorrect opinion about copyright/originality.

The point is that the fact that there are real alternative ways to do this alerts us about something else. There is an underlying common knowledge that can be hardly "copyrightable".
[...]
There are many ways to end up with the allegedly magic [-3,-1,0,+1], which can ultimately be decomposed in rudimentary, non-unique, non-copyrightable chess knowledge. It is not a surprise that we find this in Crafty and Stockfish!
[...]
I cannot see how this could be an issue of copying and that is the point I am trying to make, not only that there are some alternatives.

It is unsurprising that one can find one PST in Crafty [when changed by an additive constant, which was not a Fruit parameter], another (when approximated, and perhaps overparametrised) in Stockfish, a third PST in another engine, etc. -- but I'm sure you realise that from the statistical point of view.

However, what I which to stress is that any creative collection is indeed copyrightable (how many times have I said this by now?), which is exactly what the claim is in the first place -- that Rybka has many more "Fruit matches" than any other engine, or that could be derived by chance. In the case here, I can certainly admire Richard Vida's sentiment, that PST is too minute to matter from the standpoint of computer chess [a similar argument was made (and accepted) when the R4/Buzz magic multiplication code was evidenced in the Panel], but I would insist that there is at least some creative content in choosing a specific list of 10 or so "ramping arrays" and/or PST methodologies.

So I'll say it again (as it's a general principle) until I'm blue in the face: the fact that there is a "general source of knowledge" out there from which almost all engines borrow has little (if any) relevance to the usage of specific examples or encodings therein. This is the accusation against Rybka -- not that Fruit had a bunch of general chess knowledge and so does Rybka, but that they share way too much of the specific aspects of rendering this. The principal "creativity" (and/or originality) is thus in making choices concerning what aspects of the general knowledge to implement, how to do so, whether or not these choices jive well with each other, etc.

Some additional arguments in this genre are developed in the introduction to EVAL_COMP [for instance, the first footnote about an essay on Argentina -- forming a composite work out of a general store of knowledge is legitimate, but when one follows a single source the issue becomes more hairy]. My impression is that most claims of "nonliteral copying" tend to end up in this sort of a debate.

[Adam Hair]

The more I looked, the more unconvinced I became. At this point, I believe that Zach PST evidence is very important for 1) historical reasons. It shows Fruit's influence on R1 and that he at least learned from the PSTs. 2) It does not allow Vas to say that PST values are proof that this part of the engine is completely detached from Fruit. But, I do not believe they are evidence for incrimination. In this respect, I believe they have zero value. Zach destroyed an alibi, but that does not mean the accused was in the crime scene, to give a silly analogy. If you really believe this evidence has any weight, you should really consider that Stockfish has copied Fruit (still legal, though, but legality is a different point).

Miguel

I am guessing that you mean that the PST evidence has no weight, on its own. It does gain a little weight when considered en masse
with the other pieces of evidence. Much like the other individual pieces of evidence.

1) you do not need any reason to give an opinion about evidence that is presented to the public.
2) The fact that Vas was found guilty is irrelevant. Even if he is guilty, there are degrees of guilt and I saw that the balance was tilted towards not even recognizing his contributions to CC and his demonization in the press.
3) I said beforehand that at least two people from the secretariat should have recused themselves. I cannot participate in that situation. I am very glad I did not even sign up to the panel forum, otherwise, my name would have been attached to the document as if I signed it and approved it.

+3

[BB+]

I guess I am missing something here. You are suggesting that the PST contents were independently arrived at, despite the fact that they can be generated using the PST initialization code from Fruit with only tweaking of the constants, and that the PSTs of other engines from that period can not, and that Vas reproduced every feature from Fruit's evaluation in his own program, in the same order as they appeared in Fruit?

Give me a break. The most probable explanation is that he copied it.

I think the first paragraph exaggerates the evidence somewhat. However, I do agree that with this "most probable explanation", and indeed find the evidence to be overwhelming.

Sorry, but the documents label the PST as a major issue.

I consider the PST to be a major issue certainly from the copying standpoint. Whether or not it has chess playing significance is more disputable, but my own opinion is that PST has a demonstrable effect on the style/strength of an engine, presumably much more than the 0.3% [has anyone ever checked this number?] of code that Hsu/Campbell borrowed from the HITECH project.

I will address the 5.5 month timeline issue in another thread, though I still it is incumbent on those who find it this to be "too short" of a time to give some sense of their rationale therein.

[Adam Hair]

I won't dispute the mathematics of the issue, as I think Miguel and I largely agree there (though maybe don't state it the same way). However, I think the following is an incorrect opinion about copyright/originality.

The point is that the fact that there are real alternative ways to do this alerts us about something else. There is an underlying common knowledge that can be hardly "copyrightable".
[...]
There are many ways to end up with the allegedly magic [-3,-1,0,+1], which can ultimately be decomposed in rudimentary, non-unique, non-copyrightable chess knowledge. It is not a surprise that we find this in Crafty and Stockfish!
[...]
I cannot see how this could be an issue of copying and that is the point I am trying to make, not only that there are some alternatives.

It is unsurprising that one can find one PST in Crafty [when changed by an additive constant, which was not a Fruit parameter], another (when approximated, and perhaps overparametrised) in Stockfish, a third PST in another engine, etc. -- but I'm sure you realise that from the statistical point of view.

However, what I which to stress is that any creative collection is indeed copyrightable (how many times have I said this by now?), which is exactly what the claim is in the first place -- that Rybka has many more "Fruit matches" than any other engine, or that could be derived by chance. In the case here, I can certainly admire Richard Vida's sentiment, that PST is too minute to matter from the standpoint of computer chess [a similar argument was made (and accepted) when the R4/Buzz magic multiplication code was evidenced in the Panel], but I would insist that there is at least some creative content in choosing a specific list of 10 or so "ramping arrays" and/or PST methodologies.

So I'll say it again (as it's a general principle) until I'm blue in the face: the fact that there is a "general source of knowledge" out there from which almost all engines borrow has little (if any) relevance to the usage of specific examples or encodings therein. This is the accusation against Rybka -- not that Fruit had a bunch of general chess knowledge and so does Rybka, but that they share way too much of the specific aspects of rendering this. The principal "creativity" (and/or originality) is thus in making choices concerning what aspects of the general knowledge to implement, how to do so, whether or not these choices jive well with each other, etc.

Some additional arguments in this genre are developed in the introduction to EVAL_COMP [for instance, the first footnote about an essay on Argentina -- forming a composite work out of a general store of knowledge is legitimate, but when one follows a single source the issue becomes more hairy]. My impression is that most claims of "nonliteral copying" tend to end up in this sort of a debate.

And this is the inescapable conclusion is reached upon reviewing the evidence. Possibly none of the evidence that has been presented can be considered the "smoking gun", the thing that convincts Vas for plagarism. Yet, the preponderance of these similarities points toward that verdict. Each piece may be given a plausible alternative explanation, yet those explanations have to coincide in order to explain how the totality of the similarities is an innocent occurrance.

The only plausible explanation presented so far is that Fruit 2.1 was the blueprint for Rybka 1.0, and, though spurious alternative positions have been espoused by Ed and Chris at times, it seems that this explanation is generally recognized to be true. So, it has to be determined if the "blueprint" was followed too faithfully. In the end, the debate falls along the lines of whether or not Vas took too much. And regardless of the ICGA's decision, it is still important that this is debated in the community, by people who are qualified to conduct the debate (or at least know enough to be quiet when the discussion goes into more technical areas). Court decisions involving non-literal copying are informative. Yet, the criteria for determining right and wrong for the general community maybe could use more liberal standards, as long they are not in conflict with licensing. What I think would be important is that the general community develop somewhat more of a consensus on the issue, whatever the standards might be.

[Rebel]

I guess that this paragraph addresses me too (I do not know how to see threads but it looks like it is an answer to Ed).

It was

I think what perhaps is underestimated by the math guys is that the numbers in chess programs are far from random.

[BB+]

Court decisions involving non-literal copying are informative. Yet, the criteria for determining right and wrong for the general community maybe could use more liberal standards, as long they are not in conflict with licensing.

I agree that there should be more debate about what can be "expected" from computer chess programs, in whatever milieu they choose to compete ("basement testing", online tourneys, WCCC).

In this regard, I still think a major problem was the "clandestine" nature of the Rybka/Fruit overlap. It was recently suggested in the Rybka Forum that the "main thing" Vas did wrong was perhaps to minimise Fruit's influence on Rybka (whether deliberately or not). I can agree with some of this -- but I might point out this was indeed the crux of a lot of the future events, so I wouldn't call it "just" a mis-speak. For instance, if he had guessed "200 Elo", I think there would have been a much better sense of its overall Fruit-iness, tournaments would be more cautious in questioning this "newcomer" as to his engine origins, etc.

For instance, Ed (quite reasonably, at least in retrospect) found the ICGA to be rather languid in diligence when accepting Rybka in mid-2006 -- yet Rybka was (to the best of my knowledge) generally "accepted by the community" at that time, in no small part due to the "20 Elo" wild guess from VR's interview, and his statements in mid-Dec 2005: The Rybka source code is original and pre-dates all of the Fruit releases, and As far as I know, Rybka has a very original search and evaluation framework. The CC community has at least some level of trust in its dramatis personae, so unless there was a specific complaint against Rajlich in 2006, I can't see why the ICGA should "definitely" have done more than just have suspicious whisperings.

[hyatt]

I guess I am missing something here. You are suggesting that the PST contents were independently arrived at, despite the fact that they can be generated using the PST initialization code from Fruit with only tweaking of the constants, and that the PSTs of other engines from that period can not, and that Vas reproduced every feature from Fruit's evaluation in his own program, in the same order as they appeared in Fruit?

Give me a break. The most probable explanation is that he copied it.

I think the first paragraph exaggerates the evidence somewhat. However, I do agree that with this "most probable explanation", and indeed find the evidence to be overwhelming.

Sorry, but the documents label the PST as a major issue.

I consider the PST to be a major issue certainly from the copying standpoint. Whether or not it has chess playing significance is more disputable, but my own opinion is that PST has a demonstrable effect on the style/strength of an engine, presumably much more than the 0.3% [has anyone ever checked this number?] of code that Hsu/Campbell borrowed from the HITECH project.

I will address the 5.5 month timeline issue in another thread, though I still it is incumbent on those who find it this to be "too short" of a time to give some sense of their rationale therein.

That was actually investigated. Berliner complained that parts of Hitech were taken and used in Deep Thought (they were competing projects at CMU at the time, Berliner not involved in the DT project at all). CMU did some sort of code review since this involved potential plagiarism between multiple Ph.D. students (Ebeling did the hardware move generator and then the hardware evaluation as Berliner directed, Hsu and Campbell were both working on their Ph.D. degrees under a different advisor). The result was "no code was common between the two engines." Whether that means that there was some and it was rewritten, or there was nothing from the get-go, I don't know. Hsu and I corresponded about this quite a bit as he was convinced that "someone" wanted to sabotage his Ph.D. study... There was significant "bad blood" between the two camps. (Shoot, Berliner also accused me of cheating as well, which the ICGA investigated just as carefully as they did in the Fruit / Rybka case and exonerated us completely - this was about the 1986 WCCC tournament where we _soundly_ outplayed and defeated HiTech to win the title).

I'm not sure what the 5.5 is about. I started writing Crafty after the 1994 ACM tournament in early November. By the end of the Christmas break, Crafty was playing on ICS (now ICC). Yes, the changes came fast and furious for a while. But 5.5 months is a long time to a competent programmer for doing anything chess related. Unless you dive in over your head into something like parallel search, which has a _lot_ of potential pitfalls waiting...