Readme.md
2.64 KB
Mlicious Detection
Outline
Mlicious request detection based on http request access log.
http://git.yoho.cn/ops/malicious-detection/blob/master/arch.jpg
Access Log
Access Log is generated by inteceptor at YOHO Gateway project. It logs every http request info, including client ip, user-agent, request params, http reponse status, etc.
File name: /Data/logs/gateway/gateway_access.log @gateway
File sample:
172.16.6.206|127.0.0.1|2017-03-28 14:29:15|GET|Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36|/gateway/operations/api/v5/resource/get|app_version=3.7.1.1510230001&client_secret=a89f86ce75e828a276e286bb3e343eb9&client_type=iphone&content_code=201504091403002&gender=2%2C3&limit=20&os_version=9.1&page=1&screen_size=375x667&uid=10166061&v=7|200|128
172.16.6.206|127.0.0.1|2017-03-28 14:29:16|GET|Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36|/gateway/operations/api/v5/resource/get|app_version=3.7.1.1510230001&client_secret=a89f86ce75e828a276e286bb3e343eb9&client_type=iphone&content_code=201504091403002&gender=2%2C3&limit=20&os_version=9.1&page=1&screen_size=375x667&uid=10166061&v=7|200|128
172.16.6.206|127.0.0.1|2017-03-28 14:29:16|GET|Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36|/gateway/operations/api/v5/resource/get|app_version=3.7.1.1510230001&client_secret=a89f86ce75e828a276e286bb3e343eb9&client_type=iphone&content_code=201504091403002&gender=2%2C3&limit=20&os_version=9.1&page=1&screen_size=375x667&uid=10166061&v=7|200|123
172.16.6.206|127.0.0.1|2017-03-28 14:29:17|GET|Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36|/gateway/operations/api/v5/resource/get|app_version=3.7.1.1510230001&client_secret=a89f86ce75e828a276e286bb3e343eb9&client_type=iphone&content_code=201504091403002&gender=2%2C3&limit=20&os_version=9.1&page=1&screen_size=375x667&uid=10166061&v=7|200|119File Pattern: 本地IP|用户IP|当前时间|HTTP请求方式|User-Agent|请求标示|请求参数|HTTP响应码|请求处理时间
Spark streaming proccessing
Top 100 IP QPS in 1 seconds
Given the table for Top QPS in 1 seconds. For example:
| IP | QPS |
|---|---|
| 1.2.3.4 | 100 |
| 1.2.3.5 | 80 |
Top QPS IP Access Summary
Given the IPs ( in above table) Access summary, For example:
| IP | method | QPS |
|---|---|---|
| 1.2.3.4 | app.passport.signinAES | 100 |
| 1.2.3.4 | app.passport.profile | 80 |
| 1.2.3.5 | app.passport.profile | 30 |