Mlicious Detection

Outline

Mlicious request detection based on http request access log.

Access Log

Access Log is generated by inteceptor at YOHO Gateway project. It logs every http request info, including client ip, user-agent, request params, http reponse status, etc. File name: /Data/logs/gateway/gateway_access.log @gateway File sample:

172.16.6.206|127.0.0.1|2017-03-28 14:29:15|GET|Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36|/gateway/operations/api/v5/resource/get|app_version=3.7.1.1510230001&client_secret=a89f86ce75e828a276e286bb3e343eb9&client_type=iphone&content_code=201504091403002&gender=2%2C3&limit=20&os_version=9.1&page=1&screen_size=375x667&uid=10166061&v=7|200|128
172.16.6.206|127.0.0.1|2017-03-28 14:29:16|GET|Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36|/gateway/operations/api/v5/resource/get|app_version=3.7.1.1510230001&client_secret=a89f86ce75e828a276e286bb3e343eb9&client_type=iphone&content_code=201504091403002&gender=2%2C3&limit=20&os_version=9.1&page=1&screen_size=375x667&uid=10166061&v=7|200|128
172.16.6.206|127.0.0.1|2017-03-28 14:29:16|GET|Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36|/gateway/operations/api/v5/resource/get|app_version=3.7.1.1510230001&client_secret=a89f86ce75e828a276e286bb3e343eb9&client_type=iphone&content_code=201504091403002&gender=2%2C3&limit=20&os_version=9.1&page=1&screen_size=375x667&uid=10166061&v=7|200|123
172.16.6.206|127.0.0.1|2017-03-28 14:29:17|GET|Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36|/gateway/operations/api/v5/resource/get|app_version=3.7.1.1510230001&client_secret=a89f86ce75e828a276e286bb3e343eb9&client_type=iphone&content_code=201504091403002&gender=2%2C3&limit=20&os_version=9.1&page=1&screen_size=375x667&uid=10166061&v=7|200|119

File Pattern: 本地IP|用户IP|当前时间|HTTP请求方式|User-Agent|请求标示|请求参数|HTTP响应码|请求处理时间

Spark streaming proccessing

Top 100 IP QPS in 1 seconds

Given the table for Top QPS in 1 seconds. For example:

IP	QPS
1.2.3.4	100
1.2.3.5	80

Top QPS IP Access Summary

Given the IPs ( in above table) Access summary, For example:

IP	method	QPS
1.2.3.4	app.passport.signinAES	100
1.2.3.4	app.passport.profile	80
1.2.3.5	app.passport.profile	30