格式化nginx日志

首先，为了方便收集nginx日志，先格式化nginx日志为json，打开nginx.conf配置文件，在http{}中添加如下日志格式化参数

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
log_format logstash_json '{ "@fields": { '
                         '"@timestamp": "$time_iso8601", '
                         '"remote_addr": "$remote_addr", '
                         '"remote_user": "$remote_user", '
                         '"body_bytes_sent": "$body_bytes_sent", '
                         '"request_time": "$request_time", '
                         '"http_device_id": "$http_device_id", '
                         '"http_client_type": "$http_client_type",'
                         '"http_device_name":"$http_device_name",'
                         '"status": "$status", '
                         '"request": "$request", '
                         '"request_method": "$request_method", '
                         '"host": "$host", '
                         '"server_port": "$server_port", '
                         '"http_referrer": "$http_referer", '
                         '"body_bytes_sent":"$body_bytes_sent", '
                         '"http_x_forwarded_for": "$http_x_forwarded_for", '
                         '"http_user_agent": "$http_user_agent" } }';

其次，在站点配置中引用，如下示例

1
access_log  logs/access.log logstash_json;

Fluentd添加地图库插件fluent-plugin-geoip

插件地址，以下为CentOS7的安装

1
2
3
yum groupinstall "Development Tools"
yum install geoip-devel --enablerepo=epel
td-agent-gem install fluent-plugin-geoip

在GeoIP官网下载免费地图数据库包(GeoLite City即可)，解压并放在路径/etc/td-agent/下，假设文件名为GeoLiteCity.dat

Fluentd添加geoip配置信息

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
# Parse IP to Geo
<filter nginx**>
  @type geoip

  # Specify one or more geoip lookup field which has ip address (default: host)
  # in the case of accessing nested value, delimit keys by dot like 'host.ip'.
  geoip_lookup_keys @fields.remote_addr

  # Specify optional geoip database (using bundled GeoLiteCity databse by default)
  geoip_database    "/etc/td-agent/GeoLiteCity.dat"
  # Specify optional geoip2 database
  # geoip2_database   "/path/to/your/GeoLite2-City.mmdb" (using bundled GeoLite2-City.mmdb by default)
  # Specify backend library (geoip2_c, geoip, geoip2_compat)
  backend_library geoip2_c

  # Set adding field with placeholder (more than one settings are required.)
  <record>
    city            ${city.names.en["@fields.remote_addr"]}
    #latitude        ${location.latitude["@fields.remote_addr"]}
    #longitude       ${location.longitude["@fields.remote_addr"]}
    location       '[${location.longitude["@fields.remote_addr"]},${location.latitude["@fields.remote_addr"]}]'
    country         ${country.iso_code["@fields.remote_addr"]}
    country_name    ${country.names.en["@fields.remote_addr"]}
    #postal_code     ${postal.code["@fields.remote_addr"]}
  </record>

  # To avoid get stacktrace error with `[null, null]` array for elasticsearch.
  skip_adding_null_record  true

  # Set @log_level (default: warn)
  @log_level         info
</filter>

修改location字段默认类型为geo_point

先不要启动fluentd，需要在elasticsearch未生成nginx的索引日志时，将location字段的默认类型个性为geo_point(若不修改，默认会是float，无法生成地图数据)；直接在kibana的Dev Tools中执行如下命令(索引根据实际情况)

1
2
3
4
5
6
7
8
9
10
11
PUT _template/logstash-nginx-*
{
  "template": "logstash-nginx-*",
  "mappings": {
    "_default_": {
      "properties" : {
        "location": { "type": "geo_point"}
      }
    }
  }
}

然后开启nginx的日志收集

1
service td-agent start/restart

然后在kibana中直接创建Tile Map即可

默认地图全是英文名称，要添加高德中文名称地图支持，在kibana.yaml文件最后添加如下一行配置，并重启kibana即可

1
tilemap.url: 'http://webrd02.is.autonavi.com/appmaptile?lang=zh_cn&size=1&scale=1&style=7&x={x}&y={y}&z={z}'