forked from elastic/logstash
-
Notifications
You must be signed in to change notification settings - Fork 0
/
CHANGELOG
1534 lines (1412 loc) · 80.1 KB
/
CHANGELOG
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1.5.0.rc2
# general
- Fixed an issue with packaging our release artifacts. Release artifacts were missing jar files
which caused Logstash to not start with Elasticsearch output while using the node or transport
client. (#2780)
# input
- Lumberjack: Fixed Logstash crashing because it was using old jls-lumberjack version (#7)
1.5.0.rc1
# general
- You can now use LS_HOME/patterns directory to add generic patterns for those that may not be
associated with a particular plugin. Patterns in this dir will be loaded by default (#2225)
- We now check if the config file is correctly encoded. Otherwise we show a verbose error message
to convert the failing config file(s) to UTF-8 (#LOGSTASH-1103)
- Fixed bug in pipeline to gracefully teardown output workers when num workers > 1 (#2180)
- Fixed nologin path in release debian packages (#2283)
- Resolved issue where Logstash was crashing for users still using exclude_tags in their output
configuration (#2323)
- Updated and releases a new version of the logstash-event gem with latest changes in 1.5.0 RC1
- Windows: Significantly improved the initial user experience with Windows platform (#2504, #1426).
Fixed many issues related to File input. Added support for using the plugin
framework (installing, upgrading, removing)
- With the release of Kibana 4, we have removed the `bin/logstash web` command and any reference to
Kibana from Logstash (#2661)
# input
- Added new CouchDB input plugin to fetch data from CouchDB. Using the _changes API, data can be kept
in sync with any output like Elasticsearch by using this input
- File: Fixed a number of issues on Windows platform. These include:
- Resolving file locking issues which was causing log files to not rotate (#1557, #1389)
- Added support for using SinceDB to record multiple files' last read information (#1902)
- Fixed encoding issues which applies to many inputs (#2507)
- Resolved Logstash skipping lines when moving between files which are being followed (#1902)
- Kafka: Added support for whitelisting and blacklisting topics in the input.
- EventLog: For Windows, this input gracefully shutsdown if there is a timeout while receiving events
This also prevents Logstash from being stuck (#1672)
- Heartbeat: We created a new input plugin for generating heartbeat messages at periodic intervals.
Use this to monitor Logstash -- you can measure the latency of the pipeline using these heartbeat events,
and also check for availability
- S3:
- Added IAM roles support so you can securely read and write events from S3 without providing your
AWS credentials (#1575).
- Added support for using temporary credentials obtained from AWS STS (#1946)
- Lumberjack: Resolved issue where unrelated events were getting merged into a single event while using
this input with with the multiline codec (#2016)
# filter
- Mutate:
- Resolved issue where convert option was creating an extra field in the event (#2268)
- Fixed issue where mutate with non-existent field was throwing an error (#2379)
- Multiline: Handled cases where we unintentionally deduplicated lines, such
as repeated lines in xml messages (#3)
# output
- Elasticsearch:
- Added support to be more resilient to transient errors in Elasticsearch. Previously, partial
failures from the bulk indexing functionality were not handled properly. With this fix, we added the ability
to capture failed requests from Elasticsearch and retry them. Error codes like 429 (too many requests) will
now be retried by default for 3 times. The number of retries and the interval between consecutive retries
can be configured (#1631)
- Logstash does not create a "message.raw" by default whic is usually not_analyzed; this
helps save disk space (#11)
- Added sniffing config to be able to list machines in the cluster while using the transport client (#22)
- S3:
- Fixed a critical problem in the S3 Output plugin when using the size_file option. This could cause
data loss and data corruption of old logs ()
- Added IAM roles support so you can securely read and write events from S3 without providing your AWS
credentials (#1575)
- Added support for using temporary credentials obtained from AWS STS (#1946)
- Fixed a bug when the tags were not set in the plain text format (#1626)
# codec
- Added new Elasticsearch bulk codec which can be used to read data formatted in the Elasticsearch
Bulk API (multiline json) format. For example, this codec can be used in combination with RabbitMQ input
to mirror the functionality of the RabbitMQ Elasticsearch river
- Cloudfront: Added support for handling Amazon CloudFront events
- Avro: We added a new codec for data serialization (#1566)
1.5.0.beta1
# general
- Performance improvements: Logstash 1.5.0 is much faster -- we have improved the throughput
of grok filter in some cases by 100%. In our benchmark testing, using only grok filter and
ingesting apache logs, throughput increased from 34K eps to 50K eps.
JSON serialization/deserialization are now implemented using JrJackson library which
improved performance significantly. Ingesting JSON events 1.3KB in size measured a throughput
increase from 16Keps to 30K eps. With events 45KB in size, throughput increased from
850 eps to 3.5K eps
- Allow spaces in field references like [hello world] (#1513)
- Add Plugin manager functionality to Logstash which allows to install, delete and
update Logstash plugins
- Remove ability to run multiple subcommands from bin/logstash like
bin/logstash agent -f something.conf -- web (#1747)
- Fixed Logstash crashing on converting from ASCII to UTF-8. This was caused by charset
conversion issues in input codec (LOGSTASH-1789)
- Allow storing 'metadata' to an event which is not sent/encoded on output. This eliminates
the need for intermediate fields for example, while using date filter. (#1834)
- Accept file and http uri in -f command line option for specifying config files (#1873)
- Filters that generated events (multiline, clone, split, metrics) now propagate those events
correctly to future conditionals (#1431)
- Bump Kibana version to 3.1.2
- Fixed file descriptor leaks when using HTTP. The fix prevents Logstash from stalling, and
in some cases crashing from out-of-memory errors (#1604, LOGSTASH-892)
# input
- Lumberjack: fixed Logstash crashes with Java Out Of Memory because of TCP
thread leaks (#LOGSTASH-2168)
- TCP: fixed connection threads leak (#1509)
- Stdin: prevent overwrite of host field if already present in Event (#1668)
- S3: AWS credentials can be specfied through environment variables (#1619)
- Kafka: merge @joekiller's plugin to Logstash to get events from Kafka (#1472)
- RabbitMQ: fixed march_hare client uses incorrect connection url (LOGSTASH-2276)
- RabbitMQ: use Bunny 1.5.0+ (#1894)
- Twitter: added improvements, robustness, fixes. full_tweet option now works, we handle
Twitter rate limiting errors (#1471)
- TCP: fixed input host field also contains source port (LOGSTASH-1849)
- Syslog: if input does not match syslog format, add tag _grokparsefailure_sysloginputplugin
which can be used to debug (#1593)
# filter
- Mutate: gsub evaluates variables like %{format} in the replacement text (#1529)
- Grok: "break_on_match => false" option now works correctly (#1547)
- Grok: allow user@hostname in commonapache log pattern (#1500 #1736)
- Grok: use optimized ruby-grok library which improves throughput in some cases by 50% (#1657)
- Date: fixed match defaults to 1970-01-01 when none of the formats matches and
UNIX format is present in the list (#1236, LOGSTASH-1597)
- Mutate: fixed confusing error message for invalid type conversion (#1656, LOGSTASH-2003)
- Date: support parsing almost-ISO8601 patterns like 2001-11-06 20:45:45.123-0000 (without a T)
which does not match %{TIMESTAMP_ISO8601}
- KV: allows dynamic include/exclude keys. For example, if an event has a key field and the user
wants to parse out a value using the kv filter, the user should be able to
include_keys: [ "%{key}" ]
- DNS: fixed add_tag adds tags even if filter was unsuccessful (#1785)
- XML: fixed UndefinedConversionError with UTF-8 encoding (LOGSTASH-2246)
- Mutate: fixed nested field notation for convert option like
'convert => [ "[a][0]", "float" ]' (#1401)
# output
- RabbitMQ: fixed crash while running Logstash for longer periods, typically when there's no
traffic on the logstash<->rabbitmq socket (LOGSTASH-1886)
- Statsd: fixed issue of converting very small float numbers to scientific notation
like 9.3e-05 (#1670)
- Fixed undefined method error when conditional on an output (#LOGSTASH-2288)
- Elasticsearch output: Added support for multiple hosts in configuration and enhanced stability
- Elasticsearch output: Logstash will not create a message.raw field by default now. Message field
is not_analyzed by Elasticsearch and adding a multi-field was essentially doubling the disk space
required, with no benefit
- Elasticsearch output: We have improved the security of the Elasticsearch output, input, and filter by
adding authentication and transport encryption support. In http protocol you can configure SSL/TLS to
enable encryption and HTTP basic authentication to provide a username and password while making
requests (#1453)
- Kafka: merge @joekiller's plugin into Logstash to produce events to Kafka (#1472)
- File: Added enhancements and validations for destination path. Absolute path cannot start with a dynamic
string like /%{myfield}/, /test-%{myfield}/
1.4.2 (June 24, 2014)
# general
- fixed path issues when invoking bin/logstash outside its home directory
# input
- bugfix: generator: fixed stdin option support
- bugfix: file: fixed debian 7 path issue
# codecs
- improvement: stdin/tcp: automatically select json_line and line codecs with the tcp and stdin streaming imputs
- improvement: collectd: add support for NaN values
# outputs
- improvement: nagios_nsca: fix external command invocation to avoid shell escaping
1.4.1 (May 6, 2014)
# General
- bumped Elasticsearch to 1.1.1 and Kibana to 3.0.1
- improved specs & testing (Colin Surprenant), packaging (Richard Pijnenburg) & doc (James Turnbull)
- better $JAVA_HOME handling (Marc Chadwick)
- fixed bin/plugin target dir for when installing out from form logstash home (lr1980)
- fixed Accessors reset bug in Event#overwrite that was causing the infamous
"undefined method `tv_sec'" bug with the multiline filter (Colin Surprenant)
- fixed agent stalling when also using web option (Colin Surprenant)
- fixed accessing array-indexed event fields (Jonathan Van Eenwyk)
- new sysv init style scripts based on pleaserun (Richard Pijnenburg)
- better handling of invalid command line parameters (LOGSTASH-2024, Colin Surprenant)
- fixed running from a path containing spaces (LOGSTASH-1983, Colin Surprenant)
# inputs
- improvement: rabbitmq: upgraded Bunny gem to 1.1.8, fixes a threading leak and improves
latency (Michael Klishin)
- improvement: twitter: added "full_tweet" option (Jordan Sissel)
- improvement: generator: fixed the example doc (LOGSTASH-2093, Jason Kendall)
- improvement: imap: option to disable certificate validation (Sverre Bakke)
# codecs
- new: collectd: better performance & error handling than collectd input (Aaron Mildenstein)
- improvement: graphite: removed unused charset option (Colin Surprenant)
- improvement: json_spooler: is now deprecated (Colin Surprenant)
- improvement: proper charset support in all codecs (Colin Surprenant)
# filters
- bugfix: date: on_success actions only when date parsing actually succeed (Philippe Weber)
- bugfix: multiline: "undefined method `tv_sec'" fix (Colin Surprenant)
- bugfix: multiline: fix for "undefined method `[]' for nil:NilClass" (#1258, Colin Surprenant)
- improvement: date: fix specs for non "en" locale (Olivier Le Moal)
- improvement: grok: better pattern for RFC-5424 syslog format (Guillaume Espanel)
- improvement: grok: refactored the LOGLEVEL pattern (Lorenzo González)
- improvement: grok: fix example doc (LOGSTASH-2093, Jason Kendall)
- improvement: metrics: document .pXX metric (Juarez Bochi)
# outputs
- improvement: rabbitmq: upgraded Bunny gem to 1.1.8, fixes a threading leak and improves
latency (Michael Klishin)
- improvement: elasticsearch: start embedded server before creating a client to fix discovery
problems "waited for 30s ..." (Jordan Sissel)
- improvement: elasticsearch: have embedded ES use "bind_host" option for "network.host"
ES config (Jordan Sissel)
1.4.0 (March 20, 2014)
# General
- We've included some upgrade-specific release notes with more details about
the tarball changes and contrib packaging here:
http://logstash.net/docs/1.4.0/release-notes
- Ships with Kibana 3.0.0
- Much faster field reference implementation (Colin Surprenant)
- Fix a bug in character encoding which would cause inputs using non-UTF-8
codecs to accidentally skip re-encoding the text to UTF-8. This should
solve a great number of UTF-8-related bugs. (Colin Surprenant)
- Fixes missing gem for logstash web which was broken in 1.4.0 beta1
(LOGSTASH-1918, Jordan Sissel)
- Fix 'help' output being emitted twice when --help is invoked.
(LOGSTASH-1952, #1168)
- Logstash now supports deletes! See outputs section below.
- Update template to fit ES 1.0 API changes (untergeek)
- Lots of Makefile, gem and build improvements courtesy of untergeek, Faye
Salwin, mrsolo, ronnocol, electrical, et al
- Add `env` command so you can run arbitrary commands with the logstash
environment setup (jordansissel)
- Bug fixes (lots). Did I mention bug fixes? (Thanks, community!)
- Elasticsearch 1.0 libraries are now included. See the Elasticsearch
release notes for details: http://www.elasticsearch.org/downloads/1-0-0/
- Kibana 3 milestone 5 is included as the 'web' process.
- An empty --pluginpath directory is now accepted (#917, Richard Pijnenburg)
- Piles of documentation improvements! A brand new introductory tutorial is
included, and many of the popular plugins have had their docs greatly
improved. This effort was lead by Kurt Hurtado with assists by James
Turnbull, Aaron Mildenstein, Brad Fritz, and others.
- Testing was another focus of this release. We added many more tests
to help us prevent regressions and verify expected behavior. Helping with
this effort was Richard Pijnenburg, Jordan Sissel, and others.
- The 'debug' setting was removed from most plugins. Prior to this,
most plugins advertised the availability of this setting but actually
did not use it (#996, Jordan Sissel).
- bugfix: --pluginpath now lets you load codecs. (#1077, Sergey Zhemzhitsky)
# inputs
- bugfix: collectd: Improve handling of 'NaN' values (#1015, Pieter Lexis)
- bugfix: snmptrap: Fixes exception when not specifying yamlmibdir (#950, Andres Koetsier)
- improvement: Add Multi-threaded workers and queues to UDP input (johnarnold + untergeek)
- improvement: log4j: port now defaults to 4560, the default log4j
SocketAppender port. (#757, davux)
- bugfix: rabbitmq: auto_delete and exclusive now default to 'false'.
The previous version's defaults caused data loss on logstash restarts.
Further, these settings are recommended by the RabbitMQ folks. (#864,
Michael Klishin)
This change breaks past default behavior, so just be aware. (Michael
Klishin)
- bugfix: collectd: fix some type calculation bugs (#905, Pieter Lexis)
- improvement: collectd: Now supports decryption and signature verification
(#905, Pieter Lexis)
- improvement: wmi: now supports remote hosts (#918, Richard Pijnenburg)
- bugfix: elasticsearch: Long scrollids now work correctly (#935, Jonathan
Van Eenwyk)
- bugfix: tcp: the 'host' field is correctly set now if you are using the
json codec and include a 'host' field in your events (#937, Jordan Sissel)
- bugfix: file: the 'host' field is correctly set now if you are using the
json codec and include a 'host' field in your events (#949, Piotr
Popieluch)
- bugfix: udp: the 'host' field is correctly set now if you are using the
json codec and include a 'host' field in your events (#965, Devin
Christensen)
- bugfix: syslog: fix regression (#986, Joshua Bussdieker)
# codecs
- improvement: netflow: You can now specify your own netflow field
definitions using the 'definitions' setting. See the netflow codec
docs for examples on how to do this. (#808, Matt Dainty)
# filters
- bugfix: clone: Correctly clone events with numeric field values.
(LOGSTASH-1225, #1158, Darren Holloway)
- bugfix: zeromq: Add `timeout` and `retries` settings for retrying on
request failures. Also adds `add_tag_on_timeout` so you can act on retry
failures. (logstash-contrib#23, Michael Hart)
- new: fingerprint: Checksum, anonymize, generate UUIDs, etc! A generalized
solution to replace the following filters: uuid, checksum, and anonymize.
(#907, Richard Pijnenburg)
- new: throttle: Allows you to tag or add fields to events that occur with a
given frequency. One use case is to have logstash email you only once if an
event occurs at least 3 times in 60 seconds. (#940, Mike Pilone) -
- improvement: translate: A new 'refresh_interval' setting lets you tell
logstash to periodically try reloading the 'dictionary_path' file
without requiring a restart. (#975, Kurt Hurtado)
- improvement: geoip: Now safe to use with multiple filter workers and
(#990, #997, LOGSTASH-1842; Avleen Vig, Jordan Sissel)
- improvement: metrics: Now safe to use with multiple filter workers (#993,
Bernd Ahlers)
- bugfix: date: Fix regression that caused times to be local time instead of
the intended timezone of UTC. (#1010, Jordan Sissel)
- bugfix: geoip: Fix encoding of fields created by geoip lookups
(LOGSTASH-1354, LOGSTASH-1372, LOGSTASH-1853, #1054, #1058; Jordan Sissel,
Nick Ethier)
# outputs
- bugfix: elasticsearch: flush any buffered events on logstash shutdown
(#1175)
- feature: riemann: Automatically map event fields to rieman event fields
(logstash-contrib#15, Byron Pezan)
- bugfix: lumberjack: fix off-by-one errors causing writes to another
logstash agent to block indefinitely
- bugfix: elasticsearch: Fix NameError Socket crash on startup
(LOGSTASH-1974, #1167)
- improvement: Added `action` awesomeness to elasticsearch output (#1105, jordansissel)
- improvement: Implement `protocol => http` in elasticsearch output (#1105, jordansissel)
- bugfix: fix broken pipe output to allow EBADF instead of EPIPE,
allowing pipe command to be restarted (#974, Paweł Puterla)
- improvement: Adding dns resolution to lumberjack output (#1048, Nathan Burns )
- improvement: added pre- and post-messages to the IRC output (#1111, Lance O'Connor)
- bugfix: pipe: fix handling of command failures (#1023, #1034, LOGSTASH-1860; ronnocol, Jordan Sissel)
- improvement: lumberjack: now supports codecs (#1048, LOGSTASH-1680; Nathan Burns)
1.3.3 (January 17, 2014)
# general
- bugfix: Fix SSL cert load problem on plugins using aws-sdk: S3, SNS, etc.
(LOGSTASH-1778, LOGSTASH-1787, LOGSTASH-1784, #924; Adam Peck)
- bugfix: Fix library load problems for aws-sdk (LOGSTASH-1718, #923; Jordan
Sissel)
- bugfix: Fix regression introduced in 1.3.2 while trying to improve time
parsing performance. (LOGSTASH-1732, LOGSTASH-1738, #913; Jordan Sissel)
- bugfix: rabbitmq: honour the passive option when creating queues.
(LOGSTASH-1461, Tim Potter)
# codecs
- bugfix: json_lines, json: Fix bug causing invalid json to be incorrectly
handled with respect to encoding (#920, LOGSTASH-1595; Jordan Sissel)
1.3.2 (December 23, 2013)
# upgrade notes
- Users of logstash 1.3.0 or 1.3.1 should set 'template_overwrite => true' in
your elasticsearch (or elasticsearch_http) outputs before upgrading to this
version to ensure you receive the fixed index template.
# general
- web: don't crash if an invalid http request was sent
(#878, LOGSTASH-704; Jordan Sissel)
- Ships with Elasticsearch 0.90.9
- logstash will now try to make sure the @timestamp field is of the
correct format.
- Fix a bug in 1.3.1/1.3.0's elasticsearch index template causing phrase
searching to not work. Added tests to ensure search behavior works as
expected with this template. (Aaron Mildenstein, Jordan Sissel)
- Update README.md to be consistent with Makefile use of JRuby 1.7.8
- Time parsing in things like the json codec (and other similar parts of
logstash) are *much* faster now. This fixes a speed regression that was
introduced in logstash 1.2.0.
# filters
- improvement: date: roughly 20% faster (Jordan Sissel)
# outputs
- new: csv: write csv format to files output. (Matt Gray)
(This output will become a codec usable with file output in the next
major version!)
1.3.1 (December 11, 2013)
# general
- Fix path to the built-in elasticsearch index template
1.3.0 (December 11, 2013)
# general
- oops: The --help flag now reports help again, instead of barfing an "I need
help" exception (LOGSTASH-1436, LOGSTASH-1392; Jordan Sissel)
- Resolved encoding errors caused by environmental configurations, such as
'InvalidByteSequenceError ... on US-ASCII' (LOGSTASH-1595, #842;
Jordan Sissel)
- Fix bug causing "no such file to load -- base64" (LOGSTASH-1310,
LOGSTASH-1519, LOGSTASH-1325, LOGSTASH-1522, #834; Jordan Sissel)
- Elasticsearch version 0.90.7
- Bug fixes galore!
## inputs
- new: collectd: receive metrics from collectd's network protocol
(#785, Aaron Mildenstein)
- bugfix: gelf: handle chunked gelf message properly (#718, Thomas De Smedt)
- bugfix: s3: fix bug in region endpoint setting (#740, Andrea Ascari)
- bugfix: pipe: restart the command when it finishes (#754, Jonathan Van
Eenwyk)
- bugfix: redis: if redis fails, reconnect. (#767, LOGSTASH-1475; Jordan Sissel)
- feature: imap: add 'content_type' setting for multipart messages and
choosing the part that becomes the event message. (#784, Brad Fritz)
- bugfix: zeromq: don't override the 'host' field if the event already
has one. (Jordan Sissel)
- bugfix: ganglia: fix regressions; plugin should work again (LOGSTASH-1655,
#818; Jordan Sissel)
- bugfix: Fix missing library in sqs input (#775, LOGSTASH-1294; Toby
Collier)
## filters
- new: unique: removes duplicate values from a given field in an event.
(#676, Adam Tucker)
- new: elapsed: time duration between two tagged events. (#713, Andrea Forni)
- new: i18n: currently supports 'transliterate' which does best-effort
conversion of text to "plain" letters. Like 'ó' to 'o'. (#671,
Juarez Bochi)
- bugfix: restore filter flushing thread (LOGSTASH-1284, #689; Bernd Ahlers)
- new: elasticsearch: query elasticsearch and update your event based on the
results. (#707, Jonathan Van Eenwyk)
- new: sumnumbers: finds all numbers in a message and sums them (#752, Avleen
Vig)
- feature: geoip: new field 'location' is GeoJSON derived from the lon/lat
coordinates for use with elasticsearch, kibana, and anything else that
understands GeoJSON (#763, Aaron Mildenstein)
- new: punct: Removes all text except punctuation and stores it in another
field. Useful for as a means for fingerprinting events. (#813, Guixing Bai)
- feature: metrics: Make percentiles configurable. Also make rates (1, 5,
15-minute) optional. (#817, Juarez Bochi)
## codecs
- new: compressed_spooler: batches events and sends/receives them in
compressed form. Useful over high latency links or with transports
with higher-than-desired transmission costs. (Avleen Vig)
- new: fluent: receive data serialized using the Fluent::Logger for easier
migration away from fluentd or for folks who simply like the logger
library (#759, Jordan Sissel)
- new: edn: encode and decode the EDN serialization format. Commonly used
in Clojure. For more details, see: https://github.com/edn-format/edn
(#778, Lee Hinman)
- bugfix: oldlogstashjson: Fix encoding to work correctly. (#788, #795;
Brad Fritz)
- bugfix: oldlogstashjson: Fallback to plain text on invalid JSON
(LOGSTASH-1534, #850; Jordan Sissel)
## outputs
- feature: elasticsearch and elasticsearch_http now will apply a default
index mapping template (included) which has the settings recommended by
Elasticsearch for Logstash specifically.
Configuration options allow disabling this feature and providing a path
to your own template. (#826, #839; Aaron Mildenstein)
- feature: elasticsearch_http: optional 'user' and 'password' settings to
make use of http authentication (LOGSTASH-902, #684; Ian Neubert)
- new: google_bigquery: upload logs to bigquery for analysis later (Rodrigo
De Castro)
- bugfix: datadog_metrics: fix validation bug (#789, Ian Paredes)
- feature: elasticsearch: new 'transport' setting letting you tell logstash
to act as a cluster node (default, prior behavior) or as a 'transport
client'. With the new 'transport' mode, your firewall rules may be simpler
(unicast, one direction) and transport clients do not show up in your
cluster node list. (LOGSTASH-102, #841; Jordan Sissel)
- feature: elasticsearch: new 'bind_port setting for 'node' protocol which
lets you chose the local port to bind on (#841, Jordan Sissel)
- bugfix: Fix missing library in sqs input (#775, LOGSTASH-1294; Toby
Collier)
1.2.2 (October 22, 2013)
# general
- new 'worker' setting for outputs. This helps improve throughput on
request-oriented outputs such as redis, rabbitmq, elasticsearch,
elasticsearch_http, etc. Workers run in separate threads each handling
events as they come in. This allows you to linearly scale up outputs across
cores or as blocking-io permits.
- grok performance is up 600%
- lots of bug fixes
- bugfixes to conditionals (#682, Matt Dainty)
- rabbitmq now replaces the old deprecated amqp plugins. amqp plugins are
removed.
- inputs will now do their best to handle text which is encoded differently
than the charset you have specified (LOGSTASH-1443, Jordan Sissel)
## inputs
- bugfix: udp: respects teardown requests via SIGINT, etc (LOGSTASH-1290,
Jordan Sissel)
- bugfix: rabbitmq: disable automatic connection recovery (LOGSTASH-1350,
#641, #642; Michael Klishin)
- bugfix: twitter: works again (#640, Bernd Ahlers)
- compatibility: Restored the old 'format' setting behavior. It is still
deprecated, but was accidentally removed in 1.2.0. It will be removed
later, but is restored as part of our backwards-compat promise (Jordan
Sissel)
- bugfix: s3: fix LOGSTASH-1321 and LOGSTASH-1319 (Richard Pijnenburg)
- bugfix: log4j: fix typo (Jordan Sissel)
- bugfix: rabbitmq: disable automatic connection recover because logstash
will handle it (LOGSTASH-1350, Michael Klishin)
- bugfix: heroku: works again (LOGSTASH-1347, #643; Bernd Ahlers)
- bugfix: tcp: improve detection of closed connections to reduce lost events
(Jordan Sissel)
- bugfix: elasticsearch: now works correctly (#670, Richard Pijnenburg)
- improvement: elasticsearch: make size and scroll time configurable (#670,
Richard Pijnenburg)
- improvement: elasticsearch: tunable search type (#670, Richard Pijnenburg)
- compatibility: restore 'format' setting which was accidentally removed in
1.2.0. This feature is still deprecated, but it has been restored
temporarily as part of our backwards compatibility promise. (#706, Jordan
Sissel)
- bugfix: syslog: fix socket leakage (#704, Bernd Ahlers)
- improvement: all aws-related plugins: Add proxy_uri setting (#714, Malthe
Borch)
- bugfix: unix: fix variable name crash (#720, Nikolay Bryskin)
## codecs
- new: graphite: parse graphite formated events (Nick Ethier)
- new: json_lines: parse streams that are lines of json objects (#731, Nick
Ethier)
- bugfix: multiline: time is now correctly in UTC. (Jordan Sissel)
- bugfix: oldlogstashjson: improved conversion of old logstash json to the
new schema (#654, Jordan Sissel)
- bugfix: oldlogstashjson: fix typo breaking encoding (#665, Tom Howe)
- bugfix: json: now assumes json delimited by newline character
(LOGSTASH-1332, #710; Nick Ethier)
- improvements: netflow: new target and versions settings (#686, Matt Dainty)
## filters
- performance: grok: 6.3x performance improvement (#681, Jordan Sissel)
- bugfix: geoip: empty values (nil, empty string) are not put into the event
anymore. (Jordan Sissel)
- bugfix: geoip: allow using Maxmind's ASN database (LOGSTASH-1394, #694;
Bernd Ahlers)
- improvement: kv: target will now overwrite any existing fields, including
the source (Jordan Sissel).
- improvement: Kv: 'prefix' setting now respects sprintf (LOGSTASH-913,
#647; Richard Pijnenburg)
- checksum: sha128 was not a valid digest, removed from list
- feature: metrics: added clear_interval and flush_interval parameters for
setting flush rates and when to clear metrics (#545)
- new: collate: group events by time and/or count into a single event. (#609,
Neway Liu)
- feature: date: now supports a 'target' field for writing the timestamp into
a field other than @timestamp. (#625, Jonathan Van Eenwyk)
- bugfix: riemann: event tagging works again (#631, Marc Fournier)
- improvement: grok: IPV6 pattern (#623, Matt Dainty)
- improvement: metrics: add clear_interval and flush_interval settings (#545,
Juarez Bochi)
- improvement: useragent: include operating system details (#656, Philip
Kubat)
- improvement: csv: new quote_char setting (#725, Alex Markham)
## outputs
- feature: all outputs have a 'worker' setting now that allows you to
perform more work at the same time. This is useful for plugins like
elasticsearch_http, redis, etc, which can bottleneck on waiting for
requests to complete but would otherwise be happy processing more
simultaneous requests. (#708, Jordan Sissel)
- bugfix: elasticsearch: requests are now synchronous. This avoid overloading
the client and server with unlimited in-flight requests. (#688, Jordan
Sissel)
- bugfix: elasticsearch_http: fix bug when sending multibyte utf-8 events
(LOGSTASH-1328, #678, #679, #695; Steve Merrill, Christian Winther,
NickEthier, Jordan Sissel)
- performance: elasticsearch_http: http client library uses TCP_NODELAY now
which dramatically improves performance. (#696, Jordan Sissel)
- feature: elasticsearch_http now supports a 'replication' setting to
allow you to choose how you wait for the response. THe default is 'sync'
which waits for all replica shards to be written. If you set it to 'async'
then all index requests will respond once only the primary shards have been
written and the replica shards will be written later. This can improve
throughput. (#700, Nick Ethier, Jordan Sissel)
- bugfix: elasticsearch: the default port range is now 9300-9305; the older
range up to 9400 was unnecessary and could cause problems for the
elasticsearch cluster in some cases.
- improvement: aws-based outputs (e.g. cloudwatch) now support proxy uri.
- bugfix: rabbitmq: disable automatic connection recovery (LOGSTASH-1350)
(#642)
- bugfix: riemann: fixed tagging of riemann events (#631)
- bugfix: s3: fix LOGSTASH-1321 and LOGSTASH-1319 (#636, #645; Richard
Pijnenburg)
- bugfix: mongodb: Fix mongodb auth (LOGSTASH-1371, #659; bitsofinfo)
- bugfix: datadog: Fix time conversion (LOGSTASH-1427, #690; Bernd Ahlers)
- bugfix: statsd: Permit plain floating point values correctly in the
config. Example: sample_rate => 0.5 (LOGSTASH-1441, #705; Jordan Sissel)
- bugfix: syslog: Fix timestamp date formation. 'timestamp' setting is now
deprecated and the format of the time depends on your rfc selection.
(LOGSTASH-1423, #692, #739; Jordan Sissel, Bernd Ahlers)
## patterns
- improvement: added IPV6 suppot to IP pattern (#623)
1.2.1 (September 7, 2013)
# general
- This is primarily a bugfix/stability release based on feedback from 1.2.0
- web: kibana's default dashboard now works with the new logstash 1.2 schema.
- docs: updated the tutorials to work in logstash 1.2.x
- agent: Restored the --configtest flag (unintentionally removed from 1.2.0)
- deprecation: Using deprecated plugin settings can now advise you on a
corrective path to take. One example is the 'type' setting on filters and
outputs will now advise you to use conditionals and give an example.
- conditionals: The "not in" operator is now supported.
## inputs
- bugfix: pipe: reopen the pipe and retry on any error. (#619, Jonathan Van
Eenwyk)
- bugfix: syslog: 'message' field no longer appears as an array.
- bugfix: rabbitmq: can now bind the queue to the exchange (#624, #628,
LOGSTASH-1300, patches by Jonathan Tron and Jonathan Van Eenwyk)
## codecs
- compatibility: json: if data given is not valid as json will now be used as
the "message" of an event . This restores the older behavior when using
1.1.13's "format => json" feature on inputs. (LOGSTASH-1299)
- new: netflow: process netflow data (#580, patches by Nikolay Bryskin and
Matt Dainty)
## filters
- bugfix: multiline: the multiline filter returns! It was unintentionally
removed from the previous (1.2.0) release.
- bugfix: json_encode: fix a syntax error in the code. (LOGSTASH-1296)
- feature: kv: now captures duplicate field names as a list, so 'foo=bar
foo=baz' becomes the field 'foo' with value ['bar', 'baz'] (an array).
(#622, patch by Matt Dainty)
## outputs
- new: google_cloud_storage: archive logs to Google Cloud Storage (#572,
Rodrigo De Castro)
- bugfix: fixed bug with 'tags' and 'exclude_tags' on outputs that would
crash if the event had no tags. (LOGSTASH-1286)
1.2.0 (September 3, 2013)
# general
- The logstash json schema has changed. (LOGSTASH-675)
For prior logstash users, you will be impacted one of several ways:
* You should check your elasticsearch templates and update them accordingly.
* If you want to reindex old data from elasticsearch with the new schema,
you should be able to do this with the elasticsearch input. Just make
sure you set 'codec => oldlogstashjson' in your elasticsearch input.
- The old logstash web ui has been replaced by Kibana 3. Kibana is a far
superior search and analytics interface.
- New feature: conditionals! You can now make "if this, then ..." decisions
in your filters or outputs. See the docs here:
http://logstash.net/docs/latest/configuration#conditionals
- A new syntax exists for referencing fields (LOGSTASH-1153). This replaces
the prior and undocumented syntax for field access (was 'foo.bar' and is
now '[foo][bar]'). Learn more about this here:
http://logstash.net/docs/latest/configuration#fieldreferences
- A saner hash syntax in the logstash config is now supported. It uses the
perl/ruby hash-rocket syntax: { "key" => "value", ... } (LOGSTASH-728)
- ElasticSearch version 0.90.3 is included. (#486, Gang Chen)
- The elasticsearch plugin now uses the bulk index api which should result
in lower cpu usage as well as higher performance than the previous
logstash version.
- Many deprecated features have been removed. If your config caused
deprecation warnings on startup in logstash v1.1.13, there is a good
chance that these deprecated settings are now absent.
- 'type' is no longer a required setting on inputs.
- New plugin type: codec. Used to implement decoding of events for inputs and
encoding of events for outputs. Codecs allow us to separate transport (like
tcp, redis, rabbitmq) from serialization (gzip text, json, msgpack, etc).
- Improved error messages that try to be helpful. If you see bad or confusing
error messages, it is a bug, so let us know! (Patch by Nick Ethier)
- The old 'plugin status' concept has been replaced by 'milestones'
(LOGSTASH-1137)
- SIGHUP should cause logstash to reopen it's logfile if you are using the
--log flag
## inputs
- new: s3: reads files from s3 (#537, patch by Mathieu Guillaume)
- feature: imap: now marks emails as read (#542, Raffael Schmid)
- feature: imap: lets you delete read email (#591, Jonathan Van Eenwyk)
- feature: rabbitmq: now well-supported again (patches by Michael Klishin)
- bugfix: gelf: work around gelf parser errors (#476, patch by Chris McCoy)
- broken: the twitter input is disabled because the twitter stream v1 api is
no longer supported and I couldn't find a replacement library that works
under JRuby.
- new: sqlite input (#484, patch by Evan Livingston)
- improvement: snmptrap: new 'yamlmibdir' setting for specifying an external
source for MIB definitions. (#477, patch by Dick Davies)
- improvement: stomp: vhost support (#490, patch by Matt Dainty)
- new: unix: unix socket input (#496, patch by Nikolay Bryskin)
- new: wmi: for querying wmi (windows). (#497, patch by Philip Seidel)
- improvement: sqs: new id_field and md5_field settings (LOGSTASH-1118, Louis
Zuckerman)
## filters
- feature: grok: 'singles' now defaults to true.
- bugfix: grep: allow repeating a field in the hash config (LOGSTASH-919)
- feature: specify timezone in date filter (#470, patch by Philippe Weber)
- feature: grok setting 'overwrite' now lets you overwrite fields instead
of appending to them.
- feature: the useragent filter now defaults to writing results to the top
level of the event instead of "ua"
- feature: grok now defaults 'singles' to true, meaning captured fields are
stored as single values in most cases instead of the old behavior of being
captured as an array of values.
- new: json_encoder filter (#554, patch by Ralph Meijer)
- new: cipher: gives you many options for encrypting fields (#493, patch by
saez0pub)
- feature: kv: new settings include_fields and exclude_fields. (patch by
Piavlo)
- feature: geoip: new 'target' setting for where to write geoip results.
(#491, patch by Richard Pijnenburg)
- feature: dns: now accepts custom nameservers to query (#495, patch by
Nikolay Bryskin)
- feature: dns: now accepts a timeout setting (#507, patch by Jay Luker)
- bugfix: ruby: multiple ruby filter instances now work (#501, patch by
Nikolay Bryskin)
- feature: uuid: new filter to add a uuid to each event (#531, Tomas Doran)
- feature: useragent: added 'prefix' setting to prefix field names created
by this filter. (#524, patch by Jay Luker)
- bugfix: mutate: strip works now (#590, Jonathan Van Eenwyk)
- new: extractnumbers: extract all numbers from a message (#579, patch by
Pablo Barrera)
## outputs
- new: jira: create jira tickets from an event (#536, patch by Martin Cleaver)
- feature: rabbitmq: now well-supported again (patches by Michael Klishin)
- improvement: stomp: vhost support (Patch by Matt Dainty)
- feature: elasticsearch: now uses the bulk index api and supports
a tunable bulk flushing size.
- feature: elasticsearch_http: will now flush when idle instead of always
waiting for a full buffer. This helps in slow-sender situations such
as testing by hand.
- feature: irc: add messages_per_second tunable (LOGSTASH-962)
- bugfix: email: restored initial really useful documentation
- improvement: emails: allow @message, @source, @... in match (LOGSTASH-826,
LOGSTASH-823)
- feature: email: can now set Reply-To (#540, Tim Meighen)
- feature: mongodb: replica sets are supported (#389, patch by Mathias Gug)
- new: s3: New plugin to write to amazon S3 (#439, patch by Mattia Peterle)
- feature: statsd: now supports 'set' metrics (#513, patch by David Warden)
- feature: sqs: now supports batching (#522, patch by AaronTheApe)
- feature: ganglia: add slope and group settings (#583, patch by divanikus)
1.1.13 (May 28, 2013)
## general
- fixed bug in static file serving for logstash web (LOGSTASH-1067)
## outputs
- feature: irc: add messages_per_second tunable (LOGSTASH-962)
1.1.12 (May 7, 2013)
## filters
- bugfix: useragent filter now works correctly with the built-in regexes.yaml
- bugfix: mail output with smtp now works again
1.1.11 (May 7, 2013)
## general
- This release is primarily a bugfix release for bugs introduced by the
previous release.
- Support for Rubinius and MRI exists once again.
## inputs
- bugfix: lumberjack now respects field data again (lumberjack --field foo=bar)
- bugfix: rabbitmq was broken by the previous release (LOGSTASH-1003,
LOGSTASH-1038; Patch by Jason Koppe)
- bugfix: relp: allow multiple client socket connections to RELP input
(LOGSTASH-707, LOGSTASH-736, LOGSTASH-921)
## filters
- bugfix: geoip was broken by the previous release (LOGSTASH-1013)
- feature: sleep now accepts an 'every' setting which causes it to
sleep every N events. Example; sleep every 10 events: every => 10.
- feature: grok now permits dashes and dots in captures, such as
%{WORD:foo-bar}.
- bugfix: useragent filter now ships with a default regexes.yaml file
that is used by default unless you otherwise specify (LOGSTASH-1051)
- bugfix: add_field now correctly sets top-level fields like @message
- bugfix: mutate 'replace' now sets a field regardless of whether or not
it exists.
- feature: new mutate 'update' setting to change a field's value but
only if that field exists already.
## outputs
- feature: irc output now supports 'secure' setting to use ssl (LOGSTASH-139)
- feature: nagios_nsca has new setting 'message_format'
- bugfix: fix graphite plugin broken in 1.1.10 (LOGSTASH-968)
- bugfix: elasticsearch_http was broken in 1.1.10 (LOGSTASH-1004)
- bugfix: rabbitmq was broken by the previous release (LOGSTASH-1003,
LOGSTASH-1038; Patch by Jason Koppe)
- feature: hipchat 'notify' setting now called 'trigger_notify' (#467, patch
by Richard Pijnenburg)
1.1.10 (April 16, 2013)
## general
- On linux, all threads will set their process names so you can identify
threads in tools like top(1).
- Java 5 is no longer supported (You must use Java 6 or newer).
- Windows line terminators (CRLF) are now accepted in config files.
- All AWS-related plugins now have the same configuration options:
region, access_key_id, secret_access_key, use_ssl, and
aws_credentials_file. Affected plugins: cloudwatch output,
sns output, sqs output, sqs input. (LOGSTASH-805)
- Lots of documentation fixes (James Turnbull, et al)
- The amqp plugins are now named 'rabbitmq' because it *only* works
with rabbitmq. The old 'amqp' name should still work, but it will
be removed soon while 'rabbitmq' will stay. (Patches by Michael Zaccari)
- New flag '--configtest' to test config and exit. (Patch by Darren Patterson)
- Improved error feedback logstash gives to you as a user.
## inputs
- new: elasticsearch: this input allows you to stream search results from
elasticsearch; it uses the Scroll API.
- new: websocket. Currently supports acting as a websocket client.
- new: snmptrap, to receive SNMP traps (patch by Paul Czar)
- new: varnishlog input to read from the Varnish Cache server's shared memory
log (LOGSTASH-978, #422; Louis Zuckerman)
- new: graphite input. Supports the plain text carbon tcp protocol.
- new: imap input. Read mail!
- feature: twitter: supports http proxying now (#276, patch by Richard
Pijnenburg)
- feature: loggly: supports http proxying now (#276, patch by Richard
Pijnenburg)
- feature: tcp: ssl now supported! (#318, patch by Matthew Richardson)
- feature: redis: now supports 'batch_count' option for doing bulk fetches
from redis lists. Requires Redis 2.6.0 or higher. (#320, patch by Piavlo)
- feature: irc: will use ssl if you set 'secure' (#393, patch by Tomas Doran)
- bugfix: log4j: respect add_fields (LOGSTASH-904, #358)
- bugfix: eventlog: input should now work
- bugfix: irc: passwords now work (#412, Nick Ethier)
## filters
- new: useragent: parses user agent strings in to structured data based on
BrowserScope data (#347, patch by Dan Everton)
- new: sleep: sleeps a given amount of time before passing the event.
Useful for rate limiting or replay simulation.
- new: ruby: experimental ruby plugin that lets you call custom ruby code
on every event.
- new: translate: for mapping values (#335, patch by Paul Czar)
- new: clone: creates a copy of the event.
- feature: grok: Adds tag_on_failure setting so you can prevent grok from
tagging events on failure. (#328, patch by Neil Prosser)
- deprecated: grok: deprecated the --grok-patterns-path flag (LOGSTASH-803)
- feature: date: nested field access is allowed now
- feature: csv, xml, kv, json, geoip: new common settings!
(LOGSTASH-756, #310, #311, #312, #383, #396; patches by Richard Pijnenburg)
source - what field the text comes from
target - where to store the parse result.
- feature: csv: new setting: columns - labels for each column parsed.
- bugfix: geoip: The built-in geoip database should work now (#326, patch
by Vincent Batts)
- bugfix: kv filter now respects add_tag, etc (LOGSTASH-935)
## outputs
- new: hipchat output (#428, Cameron Stokes)
- bugfix: mongo would fail to load bson_java support (LOGSTASH-849)
- bugfix: tags support to gelf output. Returns tags as _tags field
(LOGSTASH-880, patch by James Turnbull)
- bugfix: elasticsearch: Fix a race condition. (#340, patch by Raymond Feng)
- improvement: http: now supports a custom 'message' format for building your
own http bodies from an event. (#319, patch by Christian S)
- bugfix: Fix opentsdb output (LOGSTASH-689, #317; patch by Emmet Murphy)
- improvement: http output now supports a custom message format with
the 'message' setting (Patch by Christian Schröder)
- graphite output now lets you ship the whole (or part) of an event's fields
to graphite as metric updates. (#350, patch by Piavlo)
- email output now correctly defaults to not using authentication
(LOGSTASH-559, #365; patch by Stian Mathiassen)
- bugfix: file output now works correctly on fifos
- bugfix: irc passwords now work (#412, Nick Ethier)
- improvement: redis output now supports congestion detection. If
it appears nothing is consuming from redis, the output will stall
until that problem is resolved. This helps prevent a dead reader
from letting redis fill up memory. (Piavlo)
- feature: boundary: New 'auto' setting. (#413, Alden Jole)
1.1.9 (January 10, 2013)
## inputs
- bugfix: all inputs: fix bug where some @source values were not valid urls
## filters
- bugfix: mutate: skip missing fields in 'convert' (#244, patch by Ralph Meijer)
## outputs
- improvement: gelf: new tunable 'ignore_metadata' flag to set which fields
to ignore if ship_metadata is set. (#244, patch by Ralph Meijer)
- improvement: gelf: make short_message's field name tunable (#244, patch by
Ralph Meijer)
1.1.8 (January 10, 2013)
## general
- patched another work around for JRUBY-6970 (LOGSTASH-801)
## inputs
- bugfix: tcp: 'Address in use' errors now report the host/port involved.
(LOGSTASH-831)
- bugfix: zeromq: fix bug where an invalid url could be given as a source
(LOGSTASH-821, #306)
## outputs
- bugfix: elasticsearch_river: it now resolves evaluates %{} variables in
index and index_type settings. (LOGSTASH-819)
1.1.7 (January 3, 2013)
## inputs
- fix bug where @source_host was set to 'false' in many cases.
## outputs
- improvement: redis: shuffle_hosts is now enabled by default
1.1.6 (January 2, 2013)
## Overview of this release:
- new inputs: drupal_dblog.
- new filters: anonymize, metrics.
- new outputs: syslog, cloudwatch.
- new 'charset' setting for all inputs. This should resolve all known encoding
problems. The default charset is UTF-8.
- grok now captures (?<somename>...) regexp into 'somename' field
- Elasticsearch 0.20.2 is included. This means you are required to upgrade
your elasticsearch cluster to 0.20.2. If you wish to continue using an old
version of elasticsearch, you should use the elasticsearch_http plugin
instead of the elasticsearch one.
## general
- fixed internal dependency versioning on 'addressable' gem (LOGSTASH-694)
- fixed another case of 'watchdog timeout' (LOGSTASH-701)
- plugin flags are now deprecated. The grok filter (--grok-pattern-path) was
the only plugin to make use of this.
- the grok filter has improved documentation
- lots of documentation fixes (James Turnbull, Louis Zuckerman)
- lots of testing improvements (Philippe Weber, Laust Rud Jacobsen)
- all 'name' settings have been deprecated in favor of more descriptive
settings (LOGSTASH-755)
- JRuby upgraded to 1.7.1
- removed use of bundler
- Fixed timestamp parsing in MRI (patch by Rene Lengwinat)
## inputs
- All inputs now have a 'charset' setting to help you inform logstash of the
text encoding of the input. This is useful if you have Shift_JIS or CP1251
encoded log files. This should help resolve the many UTF-8 bugs that were
reported recently. The default charset is UTF-8.
- new: drupal_dblog: read events from a DBLog-enabled Drupal. (#251, Patch by
theduke)
- bugfix: zeromq: 'topology' is now a required setting
- bugfix: lumberjack: client connection closing is now handled properly.
(Patch by Nick Ethier)
- misc: lumberjack: jls-lumberjack gem updated to 0.0.7
- bugfix: stomp: fix startup problems causing early termination (#226
- bugfix: tcp: the 'source host' for events is now the client ip:port that
sent it, instead of the listen address that received it. (LOGSTASH-796)
- improvement: tcp: the default data_timeout is now -1 (never timeout).
This change was made because read timeouts were causing data loss, and
logstash should avoid losing events by default.
- improvement: amqp: the 'name' setting is now called 'queue' (#274)
- improvement: eventlog: the 'name' setting is now called 'logfile' (#274)
- bugfix: log4j: fix stacktrace reading (#253, patch by Alex Arutyunyants)
## filters
- new: anonymize: supports many hash mechanisms (murmur3, sha1, md5, etc) as
well as IP address anonymization (#280, #261; patches by Richard Pijnenburg
and Avishai Ish-Shalom)
- new: metrics: allows you to aggregate metrics from events and emit them
periodically. Think of this like 'statsd' but implemented as a logstash
filter instead of an external service.
- feature: date: now accepts 'match' as a setting. Use of this is preferable
to the old syntax. Where you previously had 'date { somefield =>
"somepattern" }' you should now do: 'date { match => [ "somefield",
"somepattern" ] }'. (#248, LOGSTASH-734, Patch by Louis Zuckerman)
- feature: grok: now accepts (?<foo>...) named captures. This lets you
compose a pattern in the grok config without needing to define it in a
patterns file. Example: (?<hostport>%{HOST}:%{POSINT}) to capture 'hostport'
- improvement: grok: allow '$' in JAVACLASS pattern (#241, patch by Corry
Haines)
- improvement: grok: can now match against number types. Example, if you're
sending a json format event with { "status": 403 } you can now grok that
field. The number is represented as a string "403" before pattern matching.
- bugfix: date: Fix a bug that would crash the pipeline if no date pattern
matched. (LOGSTASH-705)
- feature: kv: Adds field_split, value_split, prefix, and container
settings. (#225, patch by Alex Wheeler)
- bugfix: mutate: rename on a nonexistant field now does nothing as expected.
(LOGSTASH-757)
- bugfix: grok: don't tag an event with _grokparsefailure if it's already so
(#248, patch by Greg Brockman)
- feature: mutate: new settings - split, join, strip. "split" splits a field
into an array. "join" merges an array into a string. "strip" strips leading and
trailing whitespace. (Patch by Avishai Ish-Shalom)
## outputs
- new: syslog output supporting both RFC3164 and RFC5424 (#180, patch by
Rui Alves)
- new: cloudwatch output to emit metrics and other events to Amazon CloudWatch.
(LOGSTASH-461, patch by Louis Zuckerman)
- feature: stdout: added 'message' setting for configuring the output message
format. The default is same behavior as before this feature.
- feature: http: added 'format' option to select 'json' or form-encoded
request body to send with each request.
- feature: http: added 'content_Type' option set the Content-Type header.
This defaults to "application/json" if the 'format' is 'json'. Will default
to 'application/x-www-form-urlencoded' if the 'format' is 'form'
- bugfix: zeromq: 'topology' is now a required setting
- feature: mongodb: new setting 'isodate' that, when true, stores the
@timestamp field as a mongodb date instead of a string. (#224, patch by
Kevin Amorin)
- improvement: gelf: Allow full_message gelf property to be overridden (#245,
patch by Sébastien Masset)
- misc: lumberjack: jls-lumberjack gem updated to 0.0.6
- feature: nagios: New 'nagios_level' setting to let you change the level
of the passive check result sent to nagios. (#298, Patch by James Turnbull)
- feature: elasticsearch, elasticsearch_http, elasticsearch_river: new setting
'document_id' for explicitly setting the document id in each write to
elasticsearch. This is useful for overwriting existing documents.
- improvement: elasticsearch_river: 'name' is now 'queue' (#274)
- improvement: amqp: 'name' is now 'exchange' (#274)
- bugfix: the websocket output works again (supports RFC6455)
1.1.5 (November 10, 2012)