i send json-formatted logs logstash server. log looks (note: whole message on 1 line, show in multi-line ease reading)
2016-09-01t21:07:30.152z 153.65.199.92 { "type":"trm-system", "host":"susralcent09", "timestamp":"2016-09-01t17:17:35.018470-04:00", "@version":"1", "customer":"cf_cim", "role":"app_server", "sourcefile":"/usr/share/tomcat/dist/logs/trm-system.log", "message":"some message" } what need put in logstash configuration "sourcefile" value, , filename, e.g., trm-system.log?
if pump hash field (w/o timestamp) es should recognize it.
if want inside logstash pipeline use json filter , point source => second part of line (possibly adding timestamp prefix in).
this results in fields added current message, , can access them directly or combined:
config:
input { stdin { } } filter { # split line in timestamp , json grok { match => [ message , "%{notspace:ts} %{notspace:ip} %{greedydata:js}"] } # parse json part (called "js") , add new field above json { source => "js" } } output { # stdout { codec => rubydebug } # access fields directly %{fieldname}: stdout { codec => line { format => "sourcefile: %{sourcefile}"} } } sample run
2016-09-01t21:07:30.152z 153.65.199.92 { "sourcefile":"/usr" } sourcefile: /usr and rubydebug (host , @timestamp removed):
{ "message" => "2016-09-01t21:07:30.152z 153.65.199.92 { \"sourcefile\":\"/usr\" }", "@version" => "1", "ts" => "2016-09-01t21:07:30.152z", "ip" => "153.65.199.92", "js" => "{ \"sourcefile\":\"/usr\" }", "sourcefile" => "/usr" } as can see, field sourcefile directly known value in rubydebug output.
depending on source of log records might need use multiline codec well. might want delete js field, rename @timestamp _parsedate , parse ts records timestamp (for kibana happy). not shown in sample. remove message save space.
Comments
Post a Comment