Readme.md 2.79 KB

Raw Blame History Permalink



Mlicious Detection


Outline

Mlicious request detection based on http request access log.


Access Log

Access Log is generated by inteceptor at YOHO Gateway project. It logs every http request info, including client ip, user-agent, request params, http reponse status, etc.
File name: /Data/logs/gateway/gateway_access.log  @gateway
File sample:

172.16.6.206|127.0.0.1|2017-03-28 14:29:15|GET|Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36|/gateway/operations/api/v5/resource/get|app_version=3.7.1.1510230001&client_secret=a89f86ce75e828a276e286bb3e343eb9&client_type=iphone&content_code=201504091403002&gender=2%2C3&limit=20&os_version=9.1&page=1&screen_size=375x667&uid=10166061&v=7|200|128
172.16.6.206|127.0.0.1|2017-03-28 14:29:16|GET|Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36|/gateway/operations/api/v5/resource/get|app_version=3.7.1.1510230001&client_secret=a89f86ce75e828a276e286bb3e343eb9&client_type=iphone&content_code=201504091403002&gender=2%2C3&limit=20&os_version=9.1&page=1&screen_size=375x667&uid=10166061&v=7|200|128
172.16.6.206|127.0.0.1|2017-03-28 14:29:16|GET|Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36|/gateway/operations/api/v5/resource/get|app_version=3.7.1.1510230001&client_secret=a89f86ce75e828a276e286bb3e343eb9&client_type=iphone&content_code=201504091403002&gender=2%2C3&limit=20&os_version=9.1&page=1&screen_size=375x667&uid=10166061&v=7|200|123
172.16.6.206|127.0.0.1|2017-03-28 14:29:17|GET|Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36|/gateway/operations/api/v5/resource/get|app_version=3.7.1.1510230001&client_secret=a89f86ce75e828a276e286bb3e343eb9&client_type=iphone&content_code=201504091403002&gender=2%2C3&limit=20&os_version=9.1&page=1&screen_size=375x667&uid=10166061&v=7|200|119


File Pattern: 本地IP|用户IP|当前时间|HTTP请求方式|User-Agent|请求标示|请求参数|HTTP响应码|请求处理时间

server path:


Spark : http://ops-hdp.yohops.com/cluster/scheduler


kafka 0.8:  172.31.80.214:9092


logstash 2.4.1: @gateway


Spark streaming proccessing


Top 100 IP QPS in 1 min

Given the table for Top QPS in 1 min. For example:


IP
QPS


1.2.3.4
100


1.2.3.5
80


Top QPS IP Access Summary

Given the IPs ( in above table) Access summary, For example:


IP
method
QPS


1.2.3.4
app.passport.signinAES
100


1.2.3.4
app.passport.profile
80


1.2.3.5
app.passport.profile
30