| 208 | 18 | 105 |
| 下载次数 | 被引频次 | 阅读次数 |
针对IT运维人员对基础设施各类日志进行统一管理和查询的需求,分析、设计了基于Hadoop的日志系统功能架构、技术架构以及应用场景。系统通过Rsyslog统一收集各类日志数据,利用Hive对日志进行进一步清洗和转换,将海量日志数据分类存储于HDFS,按业务规则统计分析日志信息,将计算结果定时导入MySQL数据库用于前端展示。系统实现了日志统一收集、分析处理、查询告警等功能,为IT运维人员提前发现系统故障、排查安全隐患提供有效工具。
Abstract：For managing and querying various types of logs for IT infrastructure,the functional architecture,technical architecture and application scenarios of the log analysis system based on Hadoop are analyzed and designed.Various logs are collected by Rsyslog,and then further cleaned and transformed by using Hive.The massive log data are stored into HDFS according to their classification,and are clustered,sequenced and analyzed on the basis of business rules.The calculation results are regularly imported to the MySQL database used for front-end display.The system realizes log collection,analysis,processing,query and other functions,and provides an effective tool for system administrators to identify system failures or security risks.
[6]庞松建.Boss系统日志分析研究与应用[J].计算机工程应用技术,2013,9(28):6433-6436.
[7]罗新.防火墙日志分析系统的设计与分析[J].Computer Era,2012,2:64-68.
[8]Chen HauWang,Ching TsorngTsai,Chia Chen Fan,Shyan Ming Yuan.A Hadoop Based Weblog Analysis System[C].2014 7th International Conference onUbi-Media Computing and Workshops(UMEDIA),12-14 July 2014:72-77.
[9]Yong Qing Wei,Guang Gang Zhou,Di Xu,Yu Chen.Design of the Web Log Analysis System Based on Hadoop[J].Advanced Materials Research,2014(926-930):2474-2477.
[10]景波,刘莹,陈耿.基于petri网的数据库日志分析方法研究[J].计算机科学,2014,41(6):250-253.
[11]Fengying Yang,Huichao Liu,Zhanping Zhao.Research on Cloud-Based Mass Log DataManagement Mechanism[J].JOURNAL OF COMPUTERS,2014,9(6):1371-1377.
[12]MilindBhandare,Prof.KuntalBarua,VikasNagare,DynaneshwarEkhande,Rahul Pawar.Generic Log Analyzer Using HadoopMapreduceFramework[J].2013,3(9):603-607.
[13]陈文波.基于Hadoop的日志分析系统[J].广西大学学报:自然科学学报,2011,36(1):339-342.
基本信息:
中图分类号:TP311.13;TP311.52
引用信息:
[1]张春生,郭长杰,尹兆涛.基于大数据技术的IT基础设施日志分析系统设计与实现[J].微型电脑应用,2016,32(06):49-52.
2016-06-20
2016-06-20