Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logging enhancements #69

Open
wants to merge 15 commits into
base: master
Choose a base branch
from

Conversation

vavarachen
Copy link
Contributor

I use fakenet-ng as a sinkhole and one of our primary needs is to be able to monitor traffic hitting the sinkhole and capture useful details such as HTTP headers, requested resources, credentials etc.

P.S I tried to minimize drastic changes to the upstream code.

1. Implemented remote logging capabilities.

  • Feature can be easily enabled/disabled/configured via the Fakenet ini. Examples have been added.
  • Logging can be controlled at a per listener level (RemoteLogging = 1|0)
  • JSON output can be included or excluded form any of the remote logger, including syslog.
  • TCP/UDP Syslog support
  • Splunk HTTP Event Collector (HEC) support added. For Splunk HEC, I use splunk_http_handler. In case you are using Splunk HEC listener with SSL, the splunk_http_handler module in python repo will not work. Instead use my updates to the same library.
  • When possible, Common Information Model compliant field names are used when constructing messages for remote logging
  • Events logged to Splunk are filtered using logging.filter to only send json formatted events.
  • Each listener is configured with its own dedicated remote logger

2. HTTPListener/BITSListener
Logging implemented by overloading BaseHTTPRequestHander.log_message.

Sample get request logged in json format
image

3. FTPListener
Logging implemented by overloading FTPHandler.log_cmd function (Log commands and responses in a standardized format.).

We can reconstruct the entire transaction (index=hec source="listeners.FTPListener*" | transaction src, src_port) using user,src and src_port. For example, the transaction below show the user login, download a file, download a file and then quit.

{
	"dest_port": 2121,
	"ftp_cmd": "AUTH",
	"ftp_cmd_args": "SSL",
	"ftp_respcode": 234,
	"listener": "listeners.FTPListener",
	"log_level": "INFO",
	"src": "127.0.0.1",
	"src_port": 37256,
	"user": ""
} {
	"dest_port": 2121,
	"ftp_cmd": "USER",
	"ftp_cmd_args": "JAMES",
	"ftp_respcode": 331,
	"listener": "listeners.FTPListener",
	"log_level": "INFO",
	"src": "127.0.0.1",
	"src_port": 37256,
	"user": "JAMES"
} {
	"dest_port": 2121,
	"ftp_cmd": "PASS",
	"ftp_cmd_args": "BOND",
	"ftp_respcode": 230,
	"listener": "listeners.FTPListener",
	"log_level": "INFO",
	"src": "127.0.0.1",
	"src_port": 37256,
	"user": "JAMES"
} {
	"dest_port": 2121,
	"ftp_cmd": "SYST",
	"ftp_cmd_args": "",
	"ftp_respcode": 215,
	"listener": "listeners.FTPListener",
	"log_level": "INFO",
	"src": "127.0.0.1",
	"src_port": 37256,
	"user": "JAMES"
} {
	"dest_port": 2121,
	"ftp_cmd": "TYPE",
	"ftp_cmd_args": "I",
	"ftp_respcode": 200,
	"listener": "listeners.FTPListener",
	"log_level": "INFO",
	"src": "127.0.0.1",
	"src_port": 37256,
	"user": "JAMES"
} {
	"dest_port": 2121,
	"ftp_cmd": "EPRT",
	"ftp_cmd_args": "[127.0.0.1]:47321",
	"ftp_respcode": 200,
	"listener": "listeners.FTPListener",
	"log_level": "INFO",
	"src": "127.0.0.1",
	"src_port": 37256,
	"user": "JAMES"
} {
	"dest_port": 2121,
	"ftp_cmd": "RETR",
	"ftp_cmd_args": "/git/flare-fakenet-ng/fakenet/defaultFiles/fake.file.txt",
	"ftp_respcode": 125,
	"listener": "listeners.FTPListener",
	"log_level": "INFO",
	"src": "127.0.0.1",
	"src_port": 37256,
	"user": "JAMES"
}

3. ProxyListener/RawListener
Logging implemented by extending Threated[TCP|UDP]RequestHandler classes. In the case of RawListener, log message preserves the hex output.

image

4. SMTPListener/TFTPListener
Logging implemented by extending handle() method of Threaded[TCP|UDP]RequestHander classes.

{
	"dest_port": 2525,
	"domain": ["EHLO microsoft.com%0D%0A"],
	"listener": "listeners.SMTPListener",
	"log_level": "INFO",
	"smtp_cmd": "EHLO",
	"src": "127.0.0.1",
	"src_port": 43170
} {
	"dest_port": 2525,
	"listener": "listeners.SMTPListener",
	"log_level": "INFO",
	"smtp_cmd": "MAIL",
	"src": "127.0.0.1",
	"src_port": 43170,
	"src_user": ["MAIL FROM: [email protected]%0D%0A"]
} {
	"dest_port": 2525,
	"listener": "listeners.SMTPListener",
	"log_level": "INFO",
	"recipient": ["RCPT TO: [email protected]%0D%0A"],
	"smtp_cmd": "RCPT",
	"src": "127.0.0.1",
	"src_port": 43170
} {
	"dest_port": 2525,
	"listener": "listeners.SMTPListener",
	"log_level": "INFO",
	"recipient": ["RCPT TO: [email protected]%0D%0A"],
	"smtp_cmd": "RCPT",
	"src": "127.0.0.1",
	"src_port": 43170
} {
	"dest_port": 2525,
	"listener": "listeners.SMTPListener",
	"log_level": "INFO",
	"message": "%0D%0A%0D%0AType AUTH LOGIN. The server responds with an encrypted prompt for your user name.%0D%0A%0D%0AEnter your user name encrypted in base 64. You can use one of several tools that are available to encode your user name.%0D%0A%0D%0AThe server responds with an encrypted base 64 prompt for your password. Enter your password encrypted in base 64.%0D%0A%0D%0AType MAIL FROM:<[email protected]>%2C and then press ENTER. If the sender is not permitted to send mail%2C the SMTP server returns an error.%0D%0A%0D%0AType RCPT TO:<recipient@remotedomai%0D%0A%0D%0A.%0D%0A",
	"smtp_cmd": "DATA",
	"src": "127.0.0.1",
	"src_port": 43170
} {
	"dest_port": 2525,
	"listener": "listeners.SMTPListener",
	"log_level": "INFO",
	"message": ["QUIT%0D%0A"],
	"smtp_cmd": "QUIT",
	"src": "127.0.0.1",
	"src_port": 43170
}

5. ListenerBase
Remote logger functions are implemented in the ListenerBase, which has been imported into all listeners. I had to add the 'remotelogger_config' to 'listener_config' in order to avoid drastic changes.

To Do

  • Improve user input handling before blindly handing off to loggers.
  • Bring FakeNet daemon operational logs into the fold for health monitoring

** Known Issues**

  • A lot of the redundant code could be avoided by developing a higher level Listener class which could incorporate many of the logging related details. I did not want to made drastic changes.

ProxyListener - Initialize dport to none if diverter.getOriginalDestPort does not return a destination port.

Feature enhancements
- Added ability to send logs to Splunk using HEC
- Modified the listeners to use the same logger instance
- Modified listeners to log key session information using json format

Known issues
- logger.name is not unique to each listener, which makes following the logs difficult
- Logging json formatted logs to streamhandler/filehandler is not very useful.
- Extended logging capabilities to include remote syslog and Splunk loggers.
- Implemented remote logger feature to ListenerBase
- Splunk loggers are configure with a filter to drop all non-json messages
- Log messages for remote loggers are created using Common Information Model when applicable
- Improved logging consistency across most listeners
- Each listener instance is created with its own stream and remote logger (if configured)
- Added log and log_cmd to TLS_FakeFTPHander

Bugfix
- Fixed SSL configuration of SMTP listener

ToDo
- Improve banner and server configuration for SMTP listener, similar to FTPListener
…enerBase.

- Added ability to specify more than one remote loggers
- Added json_only flag to offer the ability to avoid logging JSON to syslog.
…ger = [1|0]

Fixed type in ProxyListener.  Renamed log_mesage to log_message.
…ilter out JSON.

Additionally, addressed the issue of excluding json output in listener stream handler.
…gEnhancements

# Conflicts:
#	fakenet/listeners/ListenerBase.py
@vavarachen vavarachen mentioned this pull request Feb 20, 2018
@strictlymike
Copy link
Collaborator

strictlymike commented May 2, 2018

Hi @vavarachen, thank you for your contribution, I am impressed by the depth here and I perceive this to be of significant value. I've wanted to come up with a more structured kind of logging for quite a while. I'll evaluate this as a candidate for that.

Unfortunately, we had to prioritize refactoring FakeNet-NG to make it tenable to continue maintaining and adding features to FakeNet-NG on multiple operating systems and this has created some conflicts. I'm sensitive to how frustrating that can be. This is on us for not telegraphing what our plans were for the codebase. All that aside, depending on what I find in code review, would you be available this month to help me resolve the conflicts?

I also have a few questions if we're going to move forward:

  • Is the JSON-formatted logging available for use by anything other than Splunk/syslog? (looks like it, but unsure)
  • Does this affect making a frozenpy release? (note: right now I'm having an issue with the latest WinDivert that might make it difficult for you to answer this question)
  • How does this affect the setup.py install process?
  • Why a dedicated remote logger per listener instead of a single FakeNet-NG logger?
  • HTTP: I see that HTTP headers are bound to a kind of schema/nomenclature, what happens with custom or rare headers?
  • Why don't fakenet.py:167 and fakenet.py:174 pass self.logging_level?
  • Why is ListenerBase so named when it appears that it is not a base class but a set of module-level methods including one (add_remote_logger) that is called by fakenet.py?

At a glance, I see one thing that might need revision:

  • I think fakenet.py:94 should be elif section.startswith('RemoteLogger:'): to capture the colon and ensure it is there

Other notes:
I agree that listeners in general should have a ListenerBase to codify an interface people should program to when writing a listener and to encapsulate common code.

I'm going to take a look at the code you've put forth. Pending that, please let me know if you're available to help hit it home.

@acstanton515
Copy link

@strictlymike @vavarachen I have attempted to use this code, and have ran into multiple problems. I think there is a significant issue with splunk_http_handler in that it may take some time to complete the HTTP POST (requests is not asynchronous). In even running the simple test from that project, the code loops as logging tries to move on before the emit is complete. I think the most valuable thing here is to modify the type of log output (like JSON). I would steer away from introducing a direct integration with Splunk. Rather let the Splunk admin/user find a way to get syslog or log file HEC'd into Splunk. We just need good log output.

@vavarachen
Copy link
Contributor Author

vavarachen commented Nov 15, 2018 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants