Positional gain in LazyEval

[BB+]

I would agree that the seemingly dominant current paradigm with "lightweight" eval means that lazy eval is not so needed. However if eval takes up 80% of your compute time, the balance would change. Similarly with the idea of off-loading eval to an FPGA --- the main advantage is not that your current eval would be sped up (so as to gain maybe 20-30 ELO), but that you could vastly improve the quality of the eval without it sinking your time usage.

[mcostalba]

I would agree that the seemingly dominant current paradigm with "lightweight" eval means that lazy eval is not so needed. However if eval takes up 80% of your compute time, the balance would change. Similarly with the idea of off-loading eval to an FPGA --- the main advantage is not that your current eval would be sped up (so as to gain maybe 20-30 ELO), but that you could vastly improve the quality of the eval without it sinking your time usage.

I think that the design choice that kills lazy-eval is when you decide to evaluate each node and use that static position value to do things like pruning and/or move ordering.

In that case the extra precision you get from the eval fully compensate the speed up of lazy eval: with a more precise node evaluation you could push on the futility margins pedal and this additional pruning overcompensates the slowdown.

At least this is what we have found in SF. Of course each engine should be tested indipendentely because you cannot copy & paste tests results across engines.

[Pawel Koziol]

A couple of answers and things that cross my mind:

- Glass loses between 10 and 20 % when run without lazy eval, depending upon the position. The difference might be further exaggerated by the fact that many of its gauntlet opponents are outsearching it already with LE. Well, it has been good to do that test even if the only result was to notice that Glass is starving for speed (and there was one little bugfix on the way, since I haven't switch LE off ever since adding it).

- I'm no expert on software engineering, code quality etc. Nevertheless I think that LE might be not that ugly codewise, if we use a wrapper function that (1) inits whatever LE needs (2) does LE (3) performs a cutoff or else (3) completes the eval. Glass is a lot uglier (at least I learned to notice it over the last year and I slowly work on this aspect too).

- if data gathered during eval() are heavily re-used, then LE might be indeed rendered useless

- as for returning consistent values: I bow before Your ability to do that, even though currently available Stockfish uses aproximate eval (for deciding if null move is acceptable), normal eval and refined eval (which is either eval or TT score). It must have been a great effort to get that stuff right.

regards,

pawel

[Pawel Koziol]

OK, statistics for Glass after 408 games (gauntlet, the same set of opponents) - and I think I'll not continue this test.

with LE: 61.0%
without LE: 58,3%

[mcostalba]

OK, statistics for Glass after 408 games (gauntlet, the same set of opponents) - and I think I'll not continue this test.

with LE: 61.0%
without LE: 58,3%

There is not a lot that can be commented on given the number of games, but as you can see there is not a big difference even in a simple vanilla test like this, and if you use that extra position info that you get from the full evaluation possibly you close the gap and could even reach something more without LE.

The key point is that with full eval you open the doors to potential improvments and ideas that were precluded before when only LE score was at your disposal.

[Sentinel]

OK, statistics for Glass after 408 games (gauntlet, the same set of opponents) - and I think I'll not continue this test.

with LE: 61.0%
without LE: 58,3%

There is not a lot that can be commented on given the number of games, but as you can see there is not a big difference even in a simple vanilla test like this, and if you use that extra position info that you get from the full evaluation possibly you close the gap and could even reach something more without LE.

The key point is that with full eval you open the doors to potential improvments and ideas that were precluded before when only LE score was at your disposal.

On the other hand it seams there is no positional loss thanks to lazy eval, which is important.

There is a very accurate way to measure positional loss thanks to LE.
First disable LE and measure the exact speed loss. Then, give the version w/o LE that exact time odd. And then test it with 20k+ games. What comes as a result is an exact positional loss due to imprecise LE.

I wouldn't be surprised if in Glass's case the loss is still unmeasurable after 20k+ games.

[Rebel]

I would agree that the seemingly dominant current paradigm with "lightweight" eval means that lazy eval is not so needed. However if eval takes up 80% of your compute time, the balance would change.

You are right Sir. My experiences with LE are on: http://www.top-5000.nl/authors/rebel/chess840.htm (click lazy eval)

It's highly questionable if LE is of any use if you have a cheap evaluation with only few chess knowledge, I wouldn't know for sure because REBEL never had a cheap and fast evaluation. The power of LE lies in a big fat evaluation routine, for the latest REBEL the speed-up factor of LE is 3.2.

It's a strange mechanism, the bigger the evaluation routine, the more the speed-up factor of LE will increase. In this way it is not so bad to have an expensive evaluation routine, adding new chess knowledge is relative cheap because of LE.

The drawbacks of LE are only few, at least if you practice it in the way REBEL does, this includes that you must have the evaluation of depth-1, if you don't have it and have a big fat evaluation routine yourself, consider it as useful to do full evaluations starting at horizon-1 depth and onwards, it might give your program a real push, I sincerely hope so.

Ed

[hyatt]

Our results are certainly positive. Over a period of months, Tracy fiddled with lazy eval and eventually _improved_ on what I was previously doing by +30 Elo overall, roughly. Probably the most accurate one I ever saw was what Bruce did in Ferret. For each piece, he had calculated a +/- window of scores it could produce. He incrementally updated a "lazy eval window" using these values, so that he had a pretty accurate window, rather than a wild approximation. I fooled with this for a while, but it is really messy when going beyond simple first-order eval terms and start letting pieces interact. I decided it was getting way too complex to trust (and to keep up to date as the eval changed) and went to a more simple model. Tracy does some computations now to compute the lazy eval window on the fly, rather than just taking +/- X as the right value, and it seems to work pretty well.