sysCrow: Log-Based Anomaly Detection and Fault Localization
Detecting anomalous behavior in system logs and locating the component most likely responsible for a failure.
Research Software reliability #distributed-systems#system-architecture#ai
- Date
- Dec 2024
- Role
- Minor project, BE in Software Engineering, NCIT. Built with Prasiddha Acharya.
- Stack
- Python, TypeScript
- Source
- github.com/SatyaRajAwasth1/sysCrow
Overview
When a production system fails, the evidence is spread across thousands of log lines from many components. sysCrow is an autonomous system that detects anomalous behavior in logs and points to the components and events most likely related to the fault. It adapts ideas from peer-reviewed research into a practical, end-to-end tool.
Approach
sysCrow combines two complementary methods.
Supervised anomaly detection. Inspired by DeepLog, a sequence model learns the normal patterns of log events and flags deviations from them as anomalies.
Fault localization with unsupervised segmentation. An unsupervised method segments logs into meaningful contexts, which highlights the suspicious components and events around a fault.
Background
The project was developed with Prasiddha Acharya as our minor project for the BE in Software Engineering at NCIT, Pokhara University, and defended in December 2024.
References
- [1] Du, M., Li, F., Zheng, G., and Srikumar, V. DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning. ACM CCS, 2017. doi.org
- [2] Dobrowolski, W., Iwach-Kowalski, K., Nikodem, M., and Unold, O. Log-Based Fault Localization with Unsupervised Log Segmentation. Applied Sciences, 14, 8421, 2024. doi.org