personal experience regarding "self learning"

[deeds]

How many of the most effective moves are obtained with 2000 games ?

As it needed at least 500 games/opening just for the engine to explore few combinations among the "close" moves of the "key" positions, I decided that with 2000 games/opening the quantity of the most effective moves would be quite substantial so as not to have to train the engines again on these openings.
In my opening book, when I add the moves learned at training by the engines, I often add lines until my "out-of-book" 60 plies limit (30 plies for white, 30 plies for black). I try to retain only lines that have at least 100 games (out of 2000 games).

[deeds]

Just one simple question, how much time you spent so far for the learning (i'll not ask about the burned kWh)?

With a TC 1m+1s, the game duration is about 3min/game so for 2000 games/opening it's about 4-5 days/opening.
With a TC 2m+2s, the game duration is about 6min/game so for 2000 games/opening it's about 8-9 days/opening.
As the average depth with a TC 2m+2s is greater, the combinaisons (tried by the engines at learning) are less numerous.
I expect to have a lot less experience data to reinforce at D40 since i'm use a TC 2m+2s.
For power consumption, before training engines, I trained neural networks and even before them I made opening books.
So my cluster has always been more or less used since 2015.

[deeds]

Then on the original source, as we arrive on the posts revealing the lies followed by the denials of their authors and the appearance of censorship, we are getting out of the subject so :
- if you like shame, blood, executions => here is the link
- if not, let's continue to share our experiences...

Sooner or later the success will come, but before
I suggest work harder...!!
Meanwhile, maybe the time is already came..if so
Do you have any strong experience file, where may defeat the books ?
Perhaps it is time to for another new duel matches ?)
But this time I hope in friendly way...otherwise, please...!

To my knowledge, It's been almost a year since an experience file has equaled or even beaten several opening books :
https://www.open-chess.org/viewtopic.ph ... ience file

The method was explained several months ago too :
experience file vs opening book : the right way !

[deeds]

I don't understand why does some people test experience by matching Eman against SF or any other clones. Eman's only difference is not the experience. There are more differences and these tests act like the wins and losses are caused by the experience but the results can be affected by lots of things. If you truly want to test the effect of experience feature, you have to test the engine against itself without learning.

For having fallen into this trap (experience vs standard), it is as much a bad idea to match "Eman experience against Eman only" as "Eman experience against Stockfish" :

Code: Select all

   # PLAYER             :  RATING  ERROR   POINTS  PLAYED   (%)      W      D      L  D(%)  OppAvg  OppN
   1 Eman experience    :      59      2  27788.0   48000  57.9  11690  32196   4114  67.1       0     1
   2 Eman only          :       0   ----  20212.0   48000  42.1   4114  32196  11690  67.1      59     1

White advantage = 87.60 +/- 1.17
Draw rate (equal opponents) = 50.00 % +/- 0.00

link

Code: Select all

   # PLAYER             :  RATING  ERROR   POINTS  PLAYED   (%)      W      D      L  D(%)  OppAvg  OppN
   1 Eman experience    :       0   ----  23651.5   43500  54.4  11651  24001   7848  55.2     -34     1
   2 stockfish 15.0     :     -34      3  19848.5   43500  45.6   7848  24001  11651  55.2       0     1

White advantage = -112.92 +/- 1.22
Draw rate (equal opponents) = 50.00 % +/- 0.00

link 1
link 2
link 3

As soon as the learning feature finds a slightest imprecise move in its opponent's defense, it will test different attacks until it finds the right ones. If its opponent does not have a learning feature, there is a good chance that it will replay this imprecise move. In a match of several hundred/thousands of games, it's going to be carnage because the learning feature will find more and more effective moves against it.

[deeds]

I've been observing the experience feature from its start, (when andrea implemented kelly's code into Shashchess 6) and I truly believe that it's plusses are much more than it's cons. And from what I saw, there are only few people truly understand how experience works and how great it is. And DeeDs is one of them.

Thanks moonstonelight !

But I'm saying that believe what you want to believe, just put your beliefs on valid tests. I really respect you Sedat, you're the best tester in our country, but I don't find your tests about the learning feature valid.

08.07.2022:SCCT Duel: Book Vs Experience I
07.07.2022:SCCT Duel: Book Vs Experience II
Everything is wrong in these tests :
- experience file from another engine therefore from another evaluation function therefore the scores contained in the experience data mislead the engine
- experience file not trained with the opening books so when there is no experience it amounts to testing an "engine + book against an engine"
- cumulate 12 results vs 1 to mislead the public
- etc.

[deeds]

By the way,
If Eman engine gained so many poişnts (about 50+Elo) over Stockfish 15.1
Why not vs deep opening books ?

Links about these experiments :
round 2 completed
round 2b completed

Code: Select all

   # PLAYER             :  RATING  ERROR  POINTS  PLAYED   (%)     W     D     L  D(%)  OppAvg  OppN
   1 Eman experience    :       0   ----  2835.5    4682  60.6  1689  2293   700  49.0    -105     1
   2 stockfish 15.1     :    -105      9  1846.5    4682  39.4   700  2293  1689  49.0       0     1

White advantage = 211.66 +/- 4.48
Draw rate (equal opponents) = 50.00 % +/- 0.00

It was already explained :
"This does not mean that Eman has a +100 elo evaluation function superior to Stockfish..." (link)

Additionally, like all engines without learning feature, SF15.1 does not have an infinite variety, and its variety will shrink by increasing threads and/or TC. In the "key" positions, it will very often play its same favorite moves (bestmoves). If the moves configured in the opening books are different from those of SF15.1, the "key" positions will also be different, logically the experience file will not have the same results especially if the opening books are stronger than SF15.1.

[deeds]

How to force the engine to use the most effective moves it learned ?

Which TC for learning / tournaments ?

What does it mean for an engine to "learn an opening" ?

[sarona]

Good to see you here, Chris.

I am glad you are continuing to demonstrate the power of engine learning. This was the first forum where I discovered your posts (back in 2021) regarding engine learning/training and it is only fitting that you continue here at OpenChess.

I remember discussing those dubious "Book vs Experience Duels" with you last year at another place and then, later, reading your thread showing Eman Experience destroying one of the Solista books (no offense, Eduard - I enjoy using your books and appreciate your generosity) after a few training runs.

Regards from Canada,
Ron

[deeds]

Thanks Sarona.

Yes at OCF they call themselves "amateur" so as not to recognize their match-fixing, not to explain their bogus settings, not to accept proof, etc.

They don't have any rules against cheating but they do have some against proving them. And now OCF falls back into prehistory without learning or experience.

[deeds]

How to force the engine to use the most effective moves it learned ?

As explained here previously :

- the engine does not always follow the recommendations of its experience file because of its optimistic behavior leads it to explore new moves.
He still believes he is at learning.

- it is better to avoid experimental features such as "experience book" (=use experience file as opening book) because the experience file does not always contain only the most effective moves for all positions.
Very often, some less effective moves get better quality score because they have been played more often, analyzed more deeply, etc.


So there is only one solution : an opening book !

It should contain the most effective moves learned through practice.
At the start of the game, these moves will very often be identical to those found in the common opening books.
But they will very quickly be different in the "key" positions in the middle game.
...