Generated RFC 3164 messages are not valid CEF messages (was "toRfc3164SyslogMessage appends bogus ':' suffix to syslog header")

[asaf-romano]

toRfc3164SyslogMessage appends bogus ": " suffix to syslog headers. This makes it impossible to send proper CEF messages (which expect " CEF: " string right after the header).

[cyrille-leclerc]

• Could you share an example of message with explaining where the problem is so that we can write a test case? You can enable logging enabling on java.util.logging "com.cloudbees.syslog.sender" at the FINEST level
• Do you think the ":" that is causing you problems is this one: https://github.com/CloudBees-community/syslog-java-client/blob/syslog-java-client-1.1.1/src/main/java/com/cloudbees/syslog/SyslogMessage.java#L356

[asaf-romano]

Yes, that's precisely the line.

It basically looks like this:
<113>Oct 28 01:35:36 MacBook-Pro-2.local : CEF...
rather than:
<113>Oct 28 01:27:40 MacBook-Pro-2.local CEF

[cyrille-leclerc]

My understanding is that the Common Event Format (CEF) and RFC 3164 are two distinct formats and that we should implement an additional format in the syslog-java-client to support your use case.

Do you agree with this statement?

References:

• Common Event Format - ArcSight, Inc.
• SYSTEM LOGGING: LOG MESSAGES FORMAT FOR YOUR SIEM - RFC 3164 OR CEF?
• npmjs: nsyslog-parser

[asaf-romano]

As far as I know CEF is completely compatible with RFC 3164, as it only formats the message part. The ":" symbol simply is not in RFC 3164.

[asaf-romano]

And in any event, popular parsers don't expect it, e.g.
https://github.com/slmnhq/flume/blob/master/flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java

[asaf-romano]

Here's one of the valid examples from RFC 3164:
<13>Feb 5 17:32:18 10.0.0.99 Use the BFG!

Admittedly, they do also have examples with ":" - and of course that's valid, but, spec wise, it's just the first character in the message in that case. I see no reason for a library to enforce such convention.

[asaf-romano]

For convenience, I'm including relevant sections from RFC 3164:

 4.1.2 HEADER Part of a syslog Packet

   The HEADER part contains a timestamp and an indication of the
   hostname or IP address of the device.  [...]

   The HEADER contains two fields called the TIMESTAMP and the HOSTNAME.
   The TIMESTAMP will immediately follow the trailing ">" from the PRI
   part and single space characters MUST follow each of the TIMESTAMP
   and HOSTNAME fields. [...].  

   [...]

   A single space character MUST follow the TIMESTAMP field.

   [...] A single space character MUST also follow the HOSTNAME field.

This is where the header ends. But it looks like the ":" convention has some grounds as part of the message field, where it can follow the TAG if there is one (as done in CEF. In that sense, "CEF" is the tag):

4.1.3 MSG Part of a syslog Packet

   The MSG part will fill the remainder of the syslog packet. [...]

   [...]
 
  The MSG part has two fields known as the TAG field and the CONTENT
   field.  The value in the TAG field will be the name of the program or
   process that generated the message.  The CONTENT contains the details
   of the message.  This has traditionally been a freeform message that
   gives some detailed information of the event.  The TAG is a string of
   ABNF alphanumeric characters that MUST NOT exceed 32 characters.  Any
   non-alphanumeric character will terminate the TAG field and will be
   assumed to be the starting character of the CONTENT field.  Most
   commonly, the first character of the CONTENT field that signifies the

   conclusion of the TAG field has been seen to be the left square
   bracket character ("["), a colon character (":"), or a space
   character.  This is explained in more detail in Section 5.3.

   [...]

5.3 Originating Process Information

   It has also been considered to be a good practice to include some
   information about the process on the device that generated the
   message - if that concept exists.  This is usually the process name
   and process id (often known as the "pid") for robust operating
   systems.  The process name is commonly displayed in the TAG field.
   Quite often, additional information is included at the beginning of
   the CONTENT field.  The format of "TAG[pid]:" - without the quote
   marks - is common.  The left square bracket is used to terminate the
   TAG field in this case and is then the first character in the CONTENT
   field.  If the process id is immaterial, it may be left off.

   In that case, a colon and a space character usually follow the TAG.
   This would be displayed as "TAG: " without the quotes.  In that case,
   the colon is the first character in the CONTENT field.

Various references in the spec imply that TAG is optional:

4.2 Original syslog Packets Generated by a Device

   There are no set requirements on the contents of the syslog packet as
   it is originally sent from a device.  [...]

   For implementers that do choose to construct syslog messages with the
   RECOMMENDED format, the following guidance is offered.

         [...]

         If the originally formed message has a TAG value, then that
         will be the name of the program or process that generated the
         message.

[...]

 Example 2

        Use the BFG!

   While this is a valid message, it has extraordinarily little useful
   information.  This message does not have any discernable PRI part. It
   does not contain a timestamp or any indication of the source of the
   message.  If this message is stored on paper or disk, subsequent
   review of the message will not yield anything of value.

   This example is obviously an original message from a device.  A relay
   MUST make changes to the message as described in Section 4.3 before
   forwarding it.  The resulting relayed message is shown below.

        <13>Feb  5 17:32:18 10.0.0.99 Use the BFG!

All in all, my suggestion would be to either consider the msg field in SyslogMessage a blackbox, or split it to tag and content, and only append ":" to tag if it's set.

[cyrille-leclerc]

I may have misread this spec when I added the ':' or used a misleading sample. I'll check without ':' some common integrations (papertrail...)

[cyrille-leclerc]

• https://www.loggly.com/ does NOT support RFC 3164 https://www.loggly.com/docs/streaming-syslog-without-using-files/ . Problem does not apply
• https://papertrailapp.com/ works well without the :
• https://logdna.com/ MUST have : char to separate the application name from the message

[asaf-romano]

Well, there's no app name in my case - I would expect a colon if there was one. In any event, this behavior breaks RSA NetWitenss parsing, at the very least. It doesn't break ArcSight, which is much more liberal, generally speaking.

[cyrille-leclerc]

there's no app name in my case

Could you please do a quick test setting an application name to see if there is something specific to the case where the application name is omitted ?

[cyrille-leclerc]

Hello @asaf-romano, did you have by any chance the opportunity to do this test?

[cyrille-leclerc]

@asaf-romano friendly ping

[LuboViluda]

Hi all,

I saw your changes in "#18 RFC 3164 message format: remove ":"" commit. Now there is no ":" even if app name is specified and there still will is "-" like nullable value, if I understand correctly. What about behavior like:

• There is added app name before a message with ": " if app name isn't
  empty or null
• There is no ":" added before message part, if app name is null or empty
• No "nillvalue" "-" used if app name is empty or null

This seems like the best options for me, @asaf-romano issues will be solved when sending the message with an empty or null app name. @asaf-romano Could you confirm this? And there will be still the possibility to append app name with ": " part. (which could be desired, fe. paper trail parses this somehow and uses the first world if a message has no app name part). I committed the changes into my fork of the repository and I could create a merge request if you confirm this as desired behavior.

Regards,
Lubo

[LakshmiRavali]

Is it possible to use "syslog-java-client" with Common Event Format?