An Efficient Web Log Filter Scheme for Web Abnormal Behaviors

隨著科技的快速發展，網路相關服務愈發複雜與趨向多變化。為了更好地保護這些網路服務的安全與隱私，主動進行異常檢查與分析的動作就變得相當重要。故，因應日常安全檢查為事後主動追蹤的需求，我們提出一種針對Web異常行為的Web Log篩選機制，結合挖掘已知攻擊行為與未知異常行為兩種層面的資料，並藉這些資料之互補性，達成較快且不失過多準確度的目的。此機制首先利用PHPIDS的特徵規則進行正則表示式比對，再進行資料預處理挑出可疑的Log及轉化HTTP請求欄位形成特徵矩陣，使用隨機投影進行降階，並利用馬氏距離找出異常者且為其計算異常分數。若找出的該筆Log 太過相異將被標記為異常者，直至全數Log檢測完為止。為了有更切合的結果，此機制利用了現實公司的真實資料測試，並實驗出適切之參數。除此之外，此機制尚有簡單容易實現、快速且不失過多準確度、無需清理訓練資料的優點。

With the rapid development of technology, network services are becoming more complex and changeful. To protect the security and privacy of these network services, check and analyze abnormal behaviors actively becomes very important. In order to meet the need of check and analyze abnormal behaviors actively for routine security check, we try to find the data of known attacks and anomaly behaviors, propose a web log filter scheme for web abnormal behaviors which aims to quickly anomaly detection and also provides accuracy. The scheme uses the signature rules of PHPIDS to match, preprocesses network logs to find suspicious logs and form a feature matrix, reduces the dimensionality of matrix using random projection, uses Mahalanobis distance to identify outliers and calculate an anomaly score of the outliers. If the log line is too different, we flag it anomaly, until all of the logs are checked. In order to get the better outcome, we use the data of real-world company to test the scheme and find the suitable parameter. In addition, the advantage of the scheme is simple to implement easily, fast and without losing too much accuracy and does not need to clean training data.