Readme.md



Mlicious Detection


Outline

Mlicious request detection based on http request access log.


Deploy


Spark : http://ops-hdp.yohops.com/cluster/scheduler  (172.31.80.215, 172.31.80.213)

kafka 0.8:  172.31.80.214:9092, topic: gateway_access_log


logstash 2.4.1: @gateway


Access Log

Access Log is generated by inteceptor at YOHO Gateway project. It logs every http request info, including client ip, user-agent, request params, http reponse status, etc.
File name: /Data/logs/gateway/gateway_access.log  @gateway
File sample:

172.16.6.206|127.0.0.1|2017-03-28 14:29:15|GET|Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36|/gateway/operations/api/v5/resource/get|app_version=3.7.1.1510230001&client_secret=a89f86ce75e828a276e286bb3e343eb9&client_type=iphone&content_code=201504091403002&gender=2%2C3&limit=20&os_version=9.1&page=1&screen_size=375x667&uid=10166061&v=7|200|128
172.16.6.206|127.0.0.1|2017-03-28 14:29:16|GET|Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36|/gateway/operations/api/v5/resource/get|app_version=3.7.1.1510230001&client_secret=a89f86ce75e828a276e286bb3e343eb9&client_type=iphone&content_code=201504091403002&gender=2%2C3&limit=20&os_version=9.1&page=1&screen_size=375x667&uid=10166061&v=7|200|128
172.16.6.206|127.0.0.1|2017-03-28 14:29:16|GET|Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36|/gateway/operations/api/v5/resource/get|app_version=3.7.1.1510230001&client_secret=a89f86ce75e828a276e286bb3e343eb9&client_type=iphone&content_code=201504091403002&gender=2%2C3&limit=20&os_version=9.1&page=1&screen_size=375x667&uid=10166061&v=7|200|123
172.16.6.206|127.0.0.1|2017-03-28 14:29:17|GET|Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36|/gateway/operations/api/v5/resource/get|app_version=3.7.1.1510230001&client_secret=a89f86ce75e828a276e286bb3e343eb9&client_type=iphone&content_code=201504091403002&gender=2%2C3&limit=20&os_version=9.1&page=1&screen_size=375x667&uid=10166061&v=7|200|119


File Pattern: 本地IP|用户IP|当前时间|HTTP请求方式|User-Agent|请求标示|请求参数|HTTP响应码|请求处理时间


Spark streaming proccessing


Top 100 IP QPS in 1 min

Given the table for Top QPS in 1 min. For example:


IP
QPS


1.2.3.4
100


1.2.3.5
80


Top QPS IP Access Summary

Given the IPs ( in above table) Access summary, For example:


IP
method
QPS


1.2.3.4
app.passport.signinAES
100


1.2.3.4
app.passport.profile
80


1.2.3.5
app.passport.profile
30