Mlicious Detection

Outline

Mlicious request detection based on http request access log.

Deploy

Spark : http://ops-hdp.yohops.com/cluster/scheduler (172.31.80.215(slave), 172.31.80.213(master))
kafka 0.8: 172.31.80.214:9092, topic: gateway_access_log
logstash 2.4.1: @gateway

How to restart hadoop cluster?

 ssh 172.31.80.213
 su spark
 /Data/hadoop/sbin/stop-all.sh
 /Data/hadoop/sbin/start-all.sh

Access Log

Access Log is generated by inteceptor at YOHO Gateway project. It logs every http request info, including client ip, user-agent, request params, http reponse status, etc. File name: /Data/logs/gateway/gateway_access.log @gateway File sample:

172.16.6.206|127.0.0.1|2017-03-28 14:29:15|GET|Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36|/gateway/operations/api/v5/resource/get|app_version=3.7.1.1510230001&client_secret=a89f86ce75e828a276e286bb3e343eb9&client_type=iphone&content_code=201504091403002&gender=2%2C3&limit=20&os_version=9.1&page=1&screen_size=375x667&uid=10166061&v=7|200|128
172.16.6.206|127.0.0.1|2017-03-28 14:29:16|GET|Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36|/gateway/operations/api/v5/resource/get|app_version=3.7.1.1510230001&client_secret=a89f86ce75e828a276e286bb3e343eb9&client_type=iphone&content_code=201504091403002&gender=2%2C3&limit=20&os_version=9.1&page=1&screen_size=375x667&uid=10166061&v=7|200|128
172.16.6.206|127.0.0.1|2017-03-28 14:29:16|GET|Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36|/gateway/operations/api/v5/resource/get|app_version=3.7.1.1510230001&client_secret=a89f86ce75e828a276e286bb3e343eb9&client_type=iphone&content_code=201504091403002&gender=2%2C3&limit=20&os_version=9.1&page=1&screen_size=375x667&uid=10166061&v=7|200|123
172.16.6.206|127.0.0.1|2017-03-28 14:29:17|GET|Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36|/gateway/operations/api/v5/resource/get|app_version=3.7.1.1510230001&client_secret=a89f86ce75e828a276e286bb3e343eb9&client_type=iphone&content_code=201504091403002&gender=2%2C3&limit=20&os_version=9.1&page=1&screen_size=375x667&uid=10166061&v=7|200|119

Spark streaming proccessing

Spark分析出如下结果: （统计周期支持5s、60s）

单IP敏感接口QPS、敏感接口占比；
单IP不同udid个数
单IP中不存在udid个数，占比；

YOHOOPS分析:

IP是否在点击流上报的IP池中；

Top 100 IP QPS in 1 min

Given the table for Top QPS in 1 min. For example:

IP	QPS
1.2.3.4	100
1.2.3.5	80

Top QPS IP Access Summary

Given the IPs ( in above table) Access summary, For example:

IP	method	QPS
1.2.3.4	app.passport.signinAES	100
1.2.3.4	app.passport.profile	80
1.2.3.5	app.passport.profile	30

ops / malicious-detection

GitLab

malicious-detection