Fluentd通过HTTP方式获取应用POST日志数据

Fluentd作为一个日志收集工具，不仅可以收集日志文件，还可以接收HTTP的Post请求数据，以HTTP流的方式接受应用输出的日志。

1、配置fluentd

• INPUT

 # Allow application send json log with form of HTTP POST requests
<source>
  @type http
  port 24225
  bind 0.0.0.0
  body_size_limit 32m
  keepalive_timeout 10s
  <parse>
    @type none              # 默认不解析，即日志信息会被message默认字段包成json格式
  </parse>
  tag test-demo       # 日志tag，方便区分索引
</source>

• Filte

# test-----------------------------------------------------
<filter test-demo**>
    @type parser
    key_name message        # 解析被message字段包在里面的json日志
    replace_invalid_sequence true
    <parse>
            @type json
            time_key time
            time_type string
            time_format "%Y-%m-%d %H:%M:%S.%L"
            keep_time_key true
            timezone "+08:00"
            estimate_current_event true
    </parse>
</filter >

• OUPUT to ES

# To ES Server
<match test-demo**>
        @type elasticsearch
        hosts http://user:passwd@192.168.1.108:9200
        reload_connections false
        reconnect_on_error true
        type_name test
        logstash_format true
        logstash_prefix logstash-test-demo
        logstash_dateformat %Y.%W
        flush_interval  1s
</match>

2、服务传输日志方式

此处用curl方式模拟POST请求，提交内容为json格式的日志表单

curl -X POST -d '{"time":"2018-04-16 15:27:18.73","log":{"pageIndex":"0","pageSize":"20","sortField":"","sortOrder":"","_URL_":["user","list"]},"url":"demo.test.com\/test.php?r=\/user\/list","userinfo":{"aid":"5","auid":"0","aname":"test","role_id":"1","realname":"","mobile":"12345678910","email":"","status":"11","login_count":"1929","login_time":"1523844398","login_ip":"61.183.198.62","error_count":"3","error_ip":"59.172.152.2","error_time":"1523846444","uptime":"0","upip":"59.172.207.218","addtime":"1434530815","addip":"59.172.5.94"},"log_level":"info"}' http://localhost:24225/test-demo

注意请求后的名称"test-demo"即索引tag标签，以匹配配置文件中的tag。