Implementasi Large Language Model Llama-3 dan Retrieval-Augmented Generation untuk Chatbot Customer Service LolosASN

Abstract

Manual customer service for the LolosASN application faces challenges such as slow response times and limited operational hours, which can lead to user dissatisfaction. To address these issues, this research proposes the implementation of a customer service chatbot using the Large Language Model (LLM) Llama-3 and a Retrieval-Augmented Generation (RAG) framework. This chatbot is designed to automatically answer user questions based on a dataset of LolosASN customer service WhatsApp chat history and official CPNS documents. The development process involves data preprocessing (cleaning, case folding, text normalization, repetition handling, and text deduplication), document chunking, vector indexing, and integration with the language model. This study evaluates five chunking techniques: Fixed-size, Sentence-based, Recursive, Semantic, and Document-based. Testing results indicate that the effectiveness of chunking techniques is highly dependent on the characteristics of the dataset used. Overall, Document-based chunking proved to be the most optimal chunking technique with BERTScore Precision of 0.803, Recall of 0.778, and F1-Score of 0.782, as well as a Mean Reciprocal Rank (MRR) of 0.720, consistently demonstrating relevance, completeness, and the ability to place relevant answers at the top positions. Thus, the proposed chatbot shows potential in improving the efficiency and quality of LolosASN's customer service.