Filebeat Pipeline

Filebeat Pipeline

Filebeat Pipeline 將會介紹 Filebeat Pipeline yml上的設定。

Yml Important reminder:

1. Case sensitive
2. Use indentation to represent hierarchical relationships
3. Tabs are not allowed when indenting, only spaces are allowed.
4. The number of indented spaces is not important,
   as long as elements of the same level are aligned to the left

Sample pipeline yml:

• type: log
  paths:
• “${PWD}/sampleLogs/FullRecord.log”
  fields_under_root: true
  scan_frequency: 3s
  idle_timeout: 3s
  pipeline: fullrecordapi
  ignore_older: 2h
  close_inactive: 1h
  clean_inactive: 3h
  close_removed: true
  clean_removed: true
  processors:
• dissect:
  tokenizer: “[%{logdate}][%{class}][%{severity}][%{thread}]
  [Status]:%{Status},[Agent_ID]:%{Agent_ID},[Request]:%{Request},[Response]:%{Response}”
  
  field: “message”
  target_prefix: “”
• drop_event:
  when:
  has_fields: [“log.flags”]

Paths in pipeline yml:

If there are multiple paths in Paths, it is recommended to separate different yml for future
debugging and maintenance. The following is an example.

Multiple paths
$ vi fullrecordapi.yml

• type: log
  paths:
• /var/data/log/fullrecordapi/*.log
• /var/data/log1/fullrecordapi/*.log
  fields_under_root: true
  scan_frequency: 3s
  idle_timeout: 3s
  pipeline: fullrecordapi

Separate yml:

$ vi fullrecordapi.yml

• type: log
  paths:
• /var/data/log/fullrecordapi/*.log
  fields_under_root: true
  scan_frequency: 3s
  idle_timeout: 3s
  pipeline: fullrecordapi
  $ vi fullrecordapi1.yml
• type: log
  paths:
• /var/data/log1/fullrecordapi/*.log
  fields_under_root: true
  scan_frequency: 3s
  idle_timeout: 3s
  pipeline: fullrecordapi