This project is an enterprise-grade log monitoring and anomaly detection system developed to support Data Oceans and their clients in identifying unusual patterns across large-scale transactional and system log data. The solution leverages a combination of unsupervised machine learning models (such as Isolation Forest and K-means) and statistical methods to detect anomalies without the need for labeled data. Built with flexibility in mind, the system supports multiple data sources—including CSV files and direct database connections—and processes data using customizable, client-specific preprocessing and aggregation logic. The framework includes a prototype that integrates seamlessly with alerting systems, enabling automatic notifications when anomalies are detected. Results can be exported to various formats (CSV, JSON, SQL), and real-time insights are delivered through a comprehensive visualization layer, featuring interactive dashboards and graphs. A modular architecture allows dynamic parameter tuning and easy integration of new models and thresholds, ensuring adaptability to different error types and business needs. This system provides Data Oceans and their clients with a powerful foundation for proactive monitoring, operational visibility, and data-driven decision-making.
Mentor: Lauren Nichols
