working on idaholab#266, Malcolm ISO should format bigger drives for …

…index and artifact storage, NOT DONE YET

README.md

@@ -4,7 +4,7 @@
 
 Malcolm is a powerful network traffic analysis tool suite designed with the following goals in mind:
 
-* **Easy to use** – Malcolm accepts network traffic data in the form of full packet capture (PCAP) files and Zeek (formerly Bro) logs. These artifacts can be uploaded via a simple browser-based interface or captured live and forwarded to Malcolm using lightweight forwarders. In either case, the data is automatically normalized, enriched, and correlated for analysis.
+* **Easy to use** – Malcolm accepts network traffic data in the form of full packet capture (PCAP) files and Zeek logs. These artifacts can be uploaded via a simple browser-based interface or captured live and forwarded to Malcolm using lightweight forwarders. In either case, the data is automatically normalized, enriched, and correlated for analysis.
 * **Powerful traffic analysis** – Visibility into network communications is provided through two intuitive interfaces: OpenSearch Dashboard, a flexible data visualization plugin with dozens of prebuilt dashboards providing an at-a-glance overview of network protocols; and Arkime (formerly Moloch), a powerful tool for finding and identifying the network sessions comprising suspected security incidents.
 * **Streamlined deployment** – Malcolm operates as a cluster of Docker containers – isolated sandboxes that each serve a dedicated function of the system. This Docker-based deployment model, combined with a few simple scripts for setup and run-time management, makes Malcolm suitable to be deployed quickly across a variety of platforms and use cases; whether it be for long-term deployment on a Linux server in a security operations center (SOC) or for incident response on a Macbook for an individual engagement.
 * **Secure communications** – All communications with Malcolm, both from the user interface and from remote log forwarders, are secured with industry standard encryption protocols.

docs/arkime.md

@@ -17,7 +17,7 @@ The Arkime interface will be accessible over HTTPS on port 443 at the docker hos
 
 ## <a name="ArkimeZeek"></a>Zeek log integration
 
-A stock installation of Arkime extracts all its network connection ("session") metadata ("SPI" or "Session Profile Information") from full packet capture artifacts (PCAP files). Zeek (formerly Bro) generates similar session metadata, linking network events to sessions via a connection UID. Malcolm aims to facilitate analysis of Zeek logs by mapping values from Zeek logs to the Arkime session database schema for equivalent fields, and by creating new "native" Arkime database fields for all other Zeek log values for which there is not currently an equivalent in Arkime:
+A stock installation of Arkime extracts all its network connection ("session") metadata ("SPI" or "Session Profile Information") from full packet capture artifacts (PCAP files). Zeek generates similar session metadata, linking network events to sessions via a connection UID. Malcolm aims to facilitate analysis of Zeek logs by mapping values from Zeek logs to the Arkime session database schema for equivalent fields, and by creating new "native" Arkime database fields for all other Zeek log values for which there is not currently an equivalent in Arkime:
 
 ![Zeek log session record](./images/screenshots/arkime_session_zeek.png)
 

docs/malcolm-hedgehog-e2e-iso-install.md

@@ -463,7 +463,7 @@ To specify which files should be extracted, specify the Zeek file carving mode:
 
 If unsure what mode to choose, both **mapped (except common plain text files)** (to carve and scan almost all files) and **interesting** (to only carve and scan files with [mime types of common attack vectors]({{ site.github.repository_url }}/blob/{{ site.github.build_revision }}/hedgehog-iso/interface/sensor_ctl/zeek/extractor_override.interesting.zeek)) are probably good choices.
 
-Next, specify which carved files to preserve (saved on the sensor under `/capture/bro/capture/extract_files/quarantine` by default). In order to not consume all the sensor's available storage space, the oldest preserved files will be pruned along with the oldest Zeek logs as described below with **AUTOSTART_PRUNE_ZEEK** in the [autostart services](#HedgehogConfigAutostart) section.
+Next, specify which carved files to preserve (saved on the sensor under `/capture/zeek/capture/extract_files/quarantine` by default). In order to not consume all the sensor's available storage space, the oldest preserved files will be pruned along with the oldest Zeek logs as described below with **AUTOSTART_PRUNE_ZEEK** in the [autostart services](#HedgehogConfigAutostart) section.
 
 Users will prompted to specify which engine(s) to use to analyze extracted files. Extracted files can be examined through any of three methods:
 

hedgehog-iso/interface/sensor_ctl/control_vars.conf

@@ -32,7 +32,7 @@ export FLUENTBIT_METRICS_INTERVAL=30
 export FLUENTBIT_THERMAL_INTERVAL=10
 export FLUENTBIT_AIDE_INTERVAL=86400
 
-export ZEEK_LOG_PATH=/home/sensor/bro_logs
+export ZEEK_LOG_PATH=/home/sensor/zeek_logs
 export ZEEK_MAX_DISK_FILL=90
 export ZEEK_PRUNE_CHECK_SECONDS=90
 

hedgehog-iso/interface/sensor_ctl/filebeat/filebeat.yml

@@ -5,8 +5,8 @@ logging.metrics.enabled: false
 filebeat.inputs:
 - type: log
   paths:
-    - ${BEAT_LOG_PATTERN:/home/sensor/bro_logs/*.log}
-    - ${BEAT_STATIC_LOG_PATTERN:/home/sensor/bro_logs/static/*.log}
+    - ${BEAT_LOG_PATTERN:/home/sensor/zeek_logs/*.log}
+    - ${BEAT_STATIC_LOG_PATTERN:/home/sensor/zeek_logs/static/*.log}
   symlinks: true
   fields_under_root: true
   tags: ["_filebeat_zeek_hedgehog_live"]
@@ -24,7 +24,7 @@ filebeat.inputs:
 
 - type: log
   paths:
-    - ${BEAT_SURICATA_LOG_PATTERN:/home/sensor/bro_logs/suricata/eve*.json}
+    - ${BEAT_SURICATA_LOG_PATTERN:/home/sensor/zeek_logs/suricata/eve*.json}
   symlinks: true
   fields_under_root: true
   tags: ["_filebeat_suricata_hedgehog_live"]

hedgehog-iso/interface/sensor_ctl/filebeat/sensor_filebeat_local.sh

@@ -3,10 +3,10 @@
 # Copyright (c) 2024 Battelle Energy Alliance, LLC.  All rights reserved.
 
 if [[ -z "$ZEEK_CAPTURE_PATH" ]]; then
-  ZEEK_CAPTURE_PATH="$HOME/bro_logs"
+  ZEEK_CAPTURE_PATH="$HOME/zeek_logs"
 fi
 if [[ -z "$SURICATA_CAPTURE_PATH" ]]; then
-  SURICATA_CAPTURE_PATH="$HOME/bro_logs/suricata"
+  SURICATA_CAPTURE_PATH="$HOME/zeek_logs/suricata"
 fi
 export ZEEK_CAPTURE_PATH
 export SURICATA_CAPTURE_PATH

hedgehog-iso/interface/sensor_interface/static/js/custom.js

@@ -28,7 +28,7 @@ function stop_all() {
 
 }
 
-function start_bro() {
+function start_zeek() {
     var xhttp = new XMLHttpRequest();
     loadingBar('on');
     xhttp.onreadystatechange = function () {
@@ -41,7 +41,7 @@ function start_bro() {
     xhttp.send();
 }
 
-function stop_bro() {
+function stop_zeek() {
     var xhttp = new XMLHttpRequest();
     loadingBar('on');
     xhttp.onreadystatechange = function () {

hedgehog-iso/interface/sensor_interface/templates/buttons.html

@@ -55,7 +55,7 @@ <h3 class="mdl-card__title-text">
                 </div>
                 <div class="mdl-card__actions mdl-card--border">
                     <a class="mdl-button mdl-button--colored mdl-js-button mdl-js-ripple-effect"
-                       onclick="start_bro()">
+                       onclick="start_zeek()">
                         Start
                     </a>
                 </div>
@@ -116,7 +116,7 @@ <h3 class="mdl-card__title-text">
                 </div>
                 <div class="mdl-card__actions mdl-card--border">
                     <a class="mdl-button mdl-button--colored mdl-js-button mdl-js-ripple-effect"
-                       onclick="stop_bro()">
+                       onclick="stop_zeek()">
                         Stop
                     </a>
                 </div>

pcap-capture/scripts/supervisor.sh

@@ -19,7 +19,7 @@ function SetCaptureCapabilities() {
 
 # Create config files for each capture interface for the various capture programs (tcpdump, netsniff)
 # so that supervisord can manage instances of each of these programs for each interface.
-# bro is now managed by broctl (via brodeploy.sh) rather than individually by supervisord so that
+# zeek is now managed by zeekctl (via zeekdeploy.sh) rather than individually by supervisord so that
 # we can use pf_ring
 function CreateCaptureConfigs() {
 

shared/bin/os-disk-config.py

@@ -27,7 +27,7 @@
 OS_MODE_MALCOLM = 'malcolm'
 
 HEDGEHOG_PCAP_DIR = "pcap"
-HEDGEHOG_ZEEK_DIR = "bro"
+HEDGEHOG_ZEEK_DIR = "zeek"
 MALCOLM_DB_DIR = "datastore"
 MALCOLM_PCAP_DIR = "pcap"
 MALCOLM_LOGS_DIR = "logs"
@@ -204,7 +204,7 @@ def main():
         nargs='?',
         const=True,
         default=False,
-        help="Unmount capture directories before determining candidate drives",
+        help="Unmount storage directories before determining candidate drives",
     )
     parser.add_argument(
         '-v', '--verbose', dest='debug', type=str2bool, nargs='?', const=True, default=False, help="Verbose output"
@@ -250,9 +250,9 @@ def main():
 
     # unmount existing mounts if requested
     if args.umount and (not args.dryrun):
-        if (not args.interactive) or YesOrNo('Unmount any mounted capture path(s)?'):
+        if (not args.interactive) or YesOrNo('Unmount any mounted storage path(s)?'):
             if debug:
-                eprint("Attempting unmount of capture path(s)...")
+                eprint("Attempting unmount of storage path(s)...")
             for subdir in OS_PARAMS[osMode][MOUNT_DIRS]:
                 run_subprocess(f"umount {os.path.join(OS_PARAMS[osMode][MOUNT_ROOT_PATH], subdir)}")
             run_subprocess(f"umount {OS_PARAMS[osMode][MOUNT_ROOT_PATH]}")
@@ -271,7 +271,7 @@ def main():
                         eprint(f"\t{line}")
             _, reloadOut = run_subprocess("systemctl daemon-reload")
 
-    # check existing mounts, if the capture path(s) are already mounted, then abort
+    # check existing mounts, if the path(s) are already mounted, then abort
     with open('/proc/mounts', 'r') as f:
         for line in f.readlines():
             mountDetails = line.split()
@@ -366,7 +366,7 @@ def main():
 
     if len(candidateDevs) > 0:
         if args.encrypt:
-            # create keyfile (will be on the encrypted system drive, and used to automatically unlock the encrypted capture drives)
+            # create keyfile (will be on the encrypted system drive, and used to automatically unlock the encrypted drives)
             with open(OS_PARAMS[osMode][CRYPT_KEYFILE], 'wb') as f:
                 f.write(os.urandom(4096))
             os.chown(OS_PARAMS[osMode][CRYPT_KEYFILE], 0, 0)
@@ -574,17 +574,17 @@ def main():
         # reload tab files with systemctl
         _, reloadOut = run_subprocess("systemctl daemon-reload")
 
-        # get the GID of the group of the user(s) that will be doing the capture
+        # get the GID of the group of the user(s) under which the processes will be run
         try:
             ecode, guidGetOut = run_subprocess(
                 f"getent group {OS_PARAMS[osMode][GROUP_OWNER]}", stdout=True, stderr=True
             )
             if (ecode == 0) and (len(guidGetOut) > 0):
-                netdevGuid = int(guidGetOut[0].split(':')[2])
+                ownerGuid = int(guidGetOut[0].split(':')[2])
             else:
-                netdevGuid = -1
+                ownerGuid = -1
         except Exception:
-            netdevGuid = -1
+            ownerGuid = -1
 
         # rmdir any mount directories that might be interfering from previous configurations
         if os.path.isdir(OS_PARAMS[osMode][MOUNT_ROOT_PATH]):
@@ -599,7 +599,7 @@ def main():
             if debug:
                 eprint(f"Creating {OS_PARAMS[osMode][MOUNT_ROOT_PATH]}")
             os.makedirs(OS_PARAMS[osMode][MOUNT_ROOT_PATH], exist_ok=True)
-            os.chown(OS_PARAMS[osMode][MOUNT_ROOT_PATH], -1, netdevGuid)
+            os.chown(OS_PARAMS[osMode][MOUNT_ROOT_PATH], -1, ownerGuid)
             os.chmod(OS_PARAMS[osMode][MOUNT_ROOT_PATH], OS_PARAMS[osMode][DIR_PERMS])
 
         # add crypttab entries
@@ -656,23 +656,23 @@ def main():
                     for subdir in OS_PARAMS[osMode][MOUNT_DIRS]:
                         userDirs.append(os.path.join(par.mount, subdir))
                 else:
-                    # we're mounted somewhere *underneath* /capture, so create a user-writeable subdirectory where we are
+                    # we're mounted somewhere *underneath* /{MOUNT_ROOT_PATH}, so create a user-writeable subdirectory where we are
                     userDirs.append(os.path.join(par.mount, OS_PARAMS[osMode][MOUNT_ROOT_PATH].strip(os.path.sep)))
 
                 # set permissions on user dirs
                 createdUserDirs = defaultdict(lambda: None)
                 for userDir in userDirs:
                     os.makedirs(userDir, exist_ok=True)
-                    os.chown(userDir, OS_PARAMS[osMode][USER_UID], netdevGuid)
+                    os.chown(userDir, OS_PARAMS[osMode][USER_UID], ownerGuid)
                     os.chmod(userDir, OS_PARAMS[osMode][SUBDIR_PERMS])
                     if debug:
-                        eprint(f'Created "{userDir}" for writing by capture user')
+                        eprint(f'Created "{userDir}" for writing by unprivileged user')
                     for subdir in OS_PARAMS[osMode][MOUNT_DIRS]:
                         if f"{os.path.sep}{subdir}{os.path.sep}" in userDir:
                             createdUserDirs[subDir] = userDir
                             break
 
-                # replace capture paths in-place in control_vars.conf
+                # replace paths in-place in control_vars.conf
                 if (osMode == OS_MODE_HEDGEHOG) and os.path.isfile(OS_PARAMS[osMode][SYSTEM_CONFIG_FILE]):
                     capture_re = re.compile(r"\b(?P<key>PCAP_PATH|ZEEK_LOG_PATH)\s*=\s*.*?$")
                     with fileinput.FileInput(OS_PARAMS[osMode][SYSTEM_CONFIG_FILE], inplace=True, backup='.bak') as f: