Evaluasi Performa Pengembangan Solusi pada Permainan Catur dengan Menggunakan Stockfish Engine dan Large Language Model (LLM)

Abstract

Although Large Language Models (LLMs) possess strategic potential, their application in chess is often hindered by an inability to adhere to formal rules. This study evaluates an Analyst-Executive hybrid system, where Stockfish 17.1 provides valid move options and the LLM acts as the decision-maker. Four models were comparatively tested: three local models (Llama 3.2, Mistral, DeepSeek-Coder) and one cloud-based model (Gemini 2.5 Flash) in simulated matches against pure Stockfish. The results show a one hundred percent loss rate for all hybrid systems. However, process analysis revealed two vital findings. First, a significant reliability gap exists; local models showed response failure (fallback) rates between 5.7 to 13.6 percent, whereas the cloud model achieved near-perfect reliability with only 0.4 percent failure. Second, distinct decision patterns emerged; DeepSeek-Coder was identified as the most logical (80.7 percent selecting the primary recommendation), while Mistral and Gemini tended to be exploratory (predominantly selecting the third option). In summary, cloud infrastructure successfully solves the issue of rule integrity, but the strategic reasoning of current LLMs is not yet mature enough to rival the tactical calculation of a pure chess engine.