Dynamic index - ElasticSearch

[ashish-gpt]

data.json

{"transactionId":"4321","transactionType":"Cards","request":{"senderName":"Name1"}}

{"transactionId":"1234","transactionType":"Accounts","request":{"senderName":"Name2"}}

fluent-bit.yaml

service:
flush: 5
daemon: off
log_level: debug
Parsers_File: parsers.conf
pipeline:
inputs:
- name: tail
tag: my_cpu
path: /fluent-bit/logs/data.json
Read_from_Head: true
Parser: json

outputs:
    - name: es
      match: 'my*cpu'
      host: <elasticsearch url>
      port: 9200
      tls.verify: off
      tls: on
      Index: fluent-test-${transactionType}
      Http_User: <user>
      Http_Passwd: <pwd>
      Suppress_Type_Name: On 

In the above example how to create opensearch index dynamically based on transactionType coming in the request.

For eg – If transactionType is Cards then index name should be fluent-test-Cards

[patrick-stephens]

The Opensearch output supports using record-accessor syntax in the index: https://docs.fluentbit.io/manual/pipeline/outputs/opensearch

[ashish-gpt]

Thanks for the response Patrick.
I tried to use record accessor but no luck.

service:
flush: 5
daemon: off
log_level: debug
pipeline:
inputs:
- name: dummy
Dummy: '{"transactionId":"4321","transactionType":"Cards","request":{"senderName":"XYZ"}}'
outputs:
- name: es
match: ''
host:
port: 9243
tls.verify: off
tls: on
Index: fluent-test-$transactionType
Http_User:
Http_Passwd:
Suppress_Type_Name: On
outputs:
- name: stdout
match: ''

Can you please let me know what I am missing here?

[patrick-stephens]

Please read my actual response - you're using ES not Opensearch output.