Integrasi Snort dan Fail2ban Untuk Deteksi dan Pencegahan Serangan Brute Force Pada Protokol MQTT

Abstract

Message Queuing Telemetry Transport (MQTT) is a popular protocol within the Internet of Things (IoT) ecosystem. However, this protocol lacks built-in security mechanisms, leaving it vulnerable to attacks such as brute force, which can exploit weak authentication mechanisms, as well as functional attacks like topic brute force and Quality of Service (QoS) Abuse. These vulnerabilities can lead to server resource exhaustion, malicious data injection, and service disruption. This research aims to develop a security system capable of effectively detecting and preventing attacks on the Mosquitto MQTT broker. The methodology employed integrates Snort as an Intrusion Detection System (IDS) with Fail2ban as an Intrusion Prevention System (IPS). Snort is tasked with analyzing network traffic in real-time using a ruleset designed to identify attack patterns. When an attack is detected, Snort generates an alert log. Subsequently, Fail2ban automatically executes a temporary blocking action against the attacker's IP address by updating the iptables rules at the firewall level, based on the detection logs from Snort. The test results demonstrate that this integrated system is highly effective: Snort detected brute force attacks with an average time of approximately 6 seconds, topic attacks in an average of 5 seconds, and QoS Abuse attacks in less than 0.5 seconds. Meanwhile, Fail2ban successfully blocked the attacker's IP in approximately 1 second for all attack types post-detection. This system was proven to successfully prevent all simulated attacks while maintaining broker performance stability by suppressing CPU usage spikes and without disrupting legitimate MQTT communication.