OpenChess

ezpz420 wrote: ↑

Wed Jun 21, 2023 9:48 pm

Hi Everyone, I'm new here and joined after becoming interested in the relationship between LLMs and chess programming. I think a lot of us have seen that LLMs hallucinate quite a bit with chess and have difficulty reasoning and understanding the game. I thought it would be interesting the leverage existing chess engines with their insights as context for LLMs to create more human intelligible understanding of what is going on in a game. However, when I feed chat gpt a pgn of a game, an evaluation from stockfish, and a suggested move from stockfish, it still does a poor job of gleaning insights and demonstrating understanding of what is happening during a given board state.

To improve upon this I have hypothesized a few ways to work around this
Finetuning: This would require a manicured dataset with annotations of games
Reinforcement Learning: This would be a very difficult endeavor, but I was thinking that theoretically it would be possible to have an LLM give a move suggestion along with its 'reasoning' and then check the move suggestion with stockfish. If the evaluation is strong then this could act as a reward function. The hope would be that the LLM would update its weights in a way that would provide more human intelligible commentary, move suggestions and reasoning that is in line with the game.

I am new to chess programming so I would love everyones' thoughts on this idea. I'd also be interested in working on a project with anyone who is interested in taking this on. It is likely outside of my technical ability so far.