Bump json5 from 1.0.1 to 1.0.2 in /website

.gitignore

@@ -3,10 +3,20 @@
 .*.swp
 conf/systemd/deploy
 ~
+
 .env
-.DS_Store
-*.pyc
-data
+docker-compose.yml
+docker-compose.override.yml
+userConfig
+bin/docker-netsage-downloads.sh
+cron.d/docker-netsage-downloads.cron
+bin/restart-logstash-container.sh
+cron.d/restart-logstash-container.cron
+conf-pmacct/*_1*
+conf-pmacct/*_2*
+conf-logstash/support/sensor_groups.json
+conf-logstash/support/sensor_types.json
+
 # Dependencies
 /website/node_modules
 
@@ -29,10 +39,16 @@ yarn-debug.log*
 yarn-error.log*
 build
 
+*.pyc
 .vscode
 .history
 .idea
 replayData
-userConfig
-docker-compose.override.yml
 node_modules
+
+Makefile
+grnoc-netsage-pipeline-2.0.0.tar.gz
+blib
+blib/*
+pm_to_blib
+

CHANGES.md

@@ -1,8 +1,99 @@
+------------------------------------------------------
+## GRNOC NetSage Pipeline 2.0.0 --, 2022
+NEW PACKAGE NAME; USING PMACCT INSTEAD OF NFDUMP AND IMPORTER
+Docker will pull 7Jun2022 images we made for nfacctd and sfacctd images from Github Container Registry
+------------------------------------------------------
+Features:
+ * Renamed package to grnoc-netsage-pipeline
+ * Got rid of old importer references, requirements, files, etc. 
+ * Used the %post section in the spec file to check to see if pmacct is installed.
+ * Added systemd unit files for bare-metal sfacctd and nfacctd  (default will be 1 sflow, 1 netflow source, for docker installs)
+ * Revised docker-compose.yml file, etc. to work with pmacct containers.
+ * Revised parts of the .env file, including adding variables for number of sflow and netflow sensors. 
+ * Added example/default sfacct and nfacct config files in conf-pmacct/ (.ORIG files to be copied)
+ * Added setup-pmacct-compose.sh script which the user runs to create pmacct config files and docker-compose.yml, based on
+   docker-compose.example.yml and .env files.  (pmacct configs cannot use env vars directly, so script fills them in.)
+ * The number of sflow or netflow sensors can be 0. In this case, the setup script does not include any of the unneeded services 
+   in the docker-compose.yml file.
+ * Added 05-translate-pmacct.conf logstash config to translate pmacct fields to ones the pipeline uses.
+ * Revised 40-aggregation.conf to deal with pmacct; there are separate sections for sflow and netflow. 
+ * For netflow, in 40-aggregation.conf, the start time of incoming flows will be adjusted if duration is greater than the active timeout 
+   (ie, for "updates" to long lasting flows)
+ * The default inactive timeout for logstash aggregation has been set to 6 minutes (to go with 5 minute sflow aggregation by sfacctd)
+ * Added 41-thresholds.conf - applies size threshold of 10 MB (will drop smaller flows) and duration threshold of 1.0 sec (will set 
+   duration and rates to 0 if shorter) after aggregation is finished.
+ * Added new field: @sampling_corrected = yes/no. If sampling rate correction has been applied by pmacct or logstash, value will be yes.
+ * Sampling rate corrections will be done in logstash when requested (ie, flag is set in the env file) but 
+   ONLY IF a correction has not yet been applied (by pmacct). 
+ * Sensor list for sampling rate corrections in the env file is now semicolon-delimited.
+ * Allowed "ALL" when specifying sensors for sampling rate corrections.
+ * When a sampling rate correction is applied by logstash, add a tag with the rate. 
+ * Added an option to skip de-identification. Set it in .env.
+ * 0.0.0.x and 0.0.0.0 flows are tagged and dropped by default. (Unadvertised option to keep them is available in the env file.)
+ * Added setup-cron.sh script which copies .ORIG .cron and .sh files and writes in username and the location of the git checkout. 
+   The user must copy cron files to /etc/cron.d/.
+ * One cron file runs a script to download all files (caida, maxmind, etc) from scienceregistry.grnoc once/wk.
+ * Another cron file restarts the logstash container each day.
+ * Changed sensor_groups.json and sensor_types.json in the git checkout to .example files. From now on, our particular files/regexes 
+   will be downloaded from scienceregistry.grnoc by a cron job (different cron jobs for docker and bare-metal installations).
+ * Docker-compose.yml ensures logstash runs with uid 1000, while setup-cron.sh sets the owner of logstash-temp/ to 1000, 
+   so logstash can write and read aggregation map files when it stops and starts. (User 1000 could be anyone on the host; name doesn't matter.)
+ * AARNET privatization is no longer needed, so added .disabled to 80-privatize-org.conf, and made it into a generalized version as an 
+   example. Moved lines making the AARNET org name consistent to 95-cleanup.conf.
+
+ * Documentation updates
+ * Dependabot automatic remediations of vulnerabilites (for docusaurus)
+
+Bugs:
+ * Fixed ifindex filtering to be able to filter only specified sensors and keep all flows for other sensors; allow "ALL" for sensor names or interfaces.
+
+------------------------------------------------------
+## GRNOC NetSage Deidentfier 1.2.12 -- Jan 4, 2022
+------------------------------------------------------
+Usage note: With this release, we will move to using logstash 7.16.2 to fix a Log4j vulnerability.
+Bare-metal installations will need to upgrade logstash manually.
+(Dec 14,2021- original 1.2.12 release with logstash 7.16.1 in the pipeline_logstash Dockerfile)
+
+Features:
+  * In the dockerfile, increased the version of logstash on which the pipeline_logstash container is based 
+  * Added LEARN to the regexes in the sensor groups and types support files
+
+------------------------------------------------------
+## GRNOC NetSage Deidentfier 1.2.11 -- Sept 3, 2021
+------------------------------------------------------
+Features:
+  * Made filtering by ifindex (optionally) sensor-specific
+  * Added tags to flows with src and dst IPs = 0.0.0.x  (user can set outputs filter to do something based on tags)
+  * When duration <= 0.002 sec, set duration, bits/s, and packets/s to 0 as rates are inaccurate for small durations
+  * Added NORDUnet* and tacc_netflows to sensor group and type regexes
+  * Added onenet-members-list.rb to the members-list files to download
+  * Increased version numbers for some website-related packages is response to Dependabot
+  * Documentation improvements
+
+Bugs:
+  * Fixed es_doc_id. The hash had been missing meta.id due to a bug.
+  * At the beginning of the pipeline, set missing IPs to 0.0.0.0, missing ifindexes to -10, missing durations to 0.
+
+------------------------------------------------------
+## GRNOC NetSage Deidentfier 1.2.10 -- May 10 2021
+------------------------------------------------------
+Usage note: With this release, we will move to using nfdump v1.6.23. 
+This includes a fix for IPs not being parsed in MPLS flows, as well as the fix for missing ASNs from April. 
+  * docker-compose.override_example.yml has been updated to refer to this version.
+
+Features:
+  * 15-sensor-specific-changes.conf can now be used to drop all flows from a certain sensor except those from listed ifindexes.
+  * 0.0.0.0 flows are no longer dropped
+  * Will now tag flows with the pipeline version number (@pipeline_ver)
+  * Added a sript (to util/) that can be used to process as-org files from CAIDA into the ASN lookup files that we need.
+  * Documentation updates
+
 ------------------------------------------------------
 ## GRNOC NetSage Deidentfier 1.2.9 -- Apr 7 2021
 ------------------------------------------------------
-Usage note: With this release, we are also moving to using a version of nfdump built from github master which includes commits through Feb 20, 2021. This includes a fix for incorrect ASNs being added to flows when the ASN data is actually missing.
-  * To go along with this, docker-compose.override_example.yml refers to a "nightly" tag of nfdump (this is not updated nightly!)
+Usage note: With this release, we are also moving to using a version of nfdump built from github master which includes
+commits through Feb 20, 2021. This includes a fix for incorrect ASNs being added to flows when the ASN data is actually missing.
+  * To go along with this, docker-compose.override_example.yml refers to a "nightly" tag of nfdump (this is not actually updated nightly!)
 
 Features:
   * The installed version of 15-sensor-specific-changes.conf now accomodates environment variables for 
@@ -17,13 +108,14 @@ Bugs
   * Flow-filter changes have been made to accomodate changes to simp
   * Flows with IPs of 0.0.0.0 are dropped
   * For Docker installs, rabbit host name will be fixed 
-  * Docusaurus and some packages flagged by depndabot were upgraded
+  * Docusaurus and some packages flagged by dependabot were upgraded
 
 ------------------------------------------------------
 ## GRNOC NetSage Deidentfier 1.2.8 -- Jan 28 2021
 ------------------------------------------------------
 Features:
-  * Added 15-sensor-specific-changes.conf with multiplication by mirroring-sampling rate for a pacificwave sensor and changing of the sensor name for NEAAR flows using a certain ifindex.
+  * Added 15-sensor-specific-changes.conf with multiplication by mirroring-sampling rate for a pacificwave sensor and changing of 
+    the sensor name for NEAAR flows using a certain ifindex.
   * Started saving ifindexes to ES (at least for now)
   * Added consideration of continents to possibly get a country_scope value when a country is missing.
   * Stopped saving old 'projects' array field to ES

MANIFEST

@@ -1,77 +1,45 @@
-bin/netsage-flow-filter-daemon
-bin/netsage-netflow-importer-daemon
-bin/restart-logstash.sh
+grnoc-netsage-pipeline.spec
 CHANGES.md
+bin/restart-logstash-service.sh
+conf-pmacct/sfacctd.conf.ORIG
+conf-pmacct/nfacctd.conf.ORIG
+conf-pmacct/sfacctd-pretag.map.ORIG
+conf-pmacct/nfacctd-pretag.map.ORIG
 conf-logstash/01-input-jsonfile.conf.disabled
 conf-logstash/01-input-multiline-json-file.conf.disabled
 conf-logstash/01-input-rabbit.conf
+conf-logstash/05-translate-pmacct.conf
 conf-logstash/10-preliminaries.conf
 conf-logstash/15-sensor-specific-changes.conf
 conf-logstash/20-add-id.conf
 conf-logstash/40-aggregation.conf
+conf-logstash/41-thresholds.conf
 conf-logstash/45-geoip-tagging.conf
 conf-logstash/50-asn.conf
 conf-logstash/53-caida-org.conf
 conf-logstash/55-member-orgs.conf
 conf-logstash/60-scireg-tagging-fakegeoip.conf
 conf-logstash/70-deidentify.conf
-conf-logstash/80-privatize-org.conf
+conf-logstash/80-privatize-org.conf.disabled
 conf-logstash/88-preferred-location-org.conf
 conf-logstash/90-additional-fields.conf
 conf-logstash/95-cleanup.conf
 conf-logstash/98-post-process.conf
+conf-logstash/99-output-rabbit.conf
+conf-logstash/99-output-file.conf.disabled
 conf-logstash/99-output-elastic.conf.disabled
-conf-logstash/99-output-jsonlog.conf.disabled
-conf-logstash/99-output-multiline-json.conf.disabled
 conf-logstash/99-output-stdout.conf.disabled
-conf-logstash/99-output-rabbit.conf
 conf-logstash/ruby/anonymize_ipv6.rb
 conf-logstash/ruby/domestic.rb
-conf-logstash/support/sensor_groups.json  
-conf-logstash/support/sensor_types.json  
+conf-logstash/support/sensor_groups.json.example  
+conf-logstash/support/sensor_types.json.example  
 conf-logstash/support/networkA-members-list.rb.example
-conf/logging-debug.conf
-conf/logging.conf
-conf/netsage_flow_filter.xml
-conf/netsage_netflow_importer.xml
-conf/netsage_shared.xml
-cron.d/netsage-maxmind-update.cron
-cron.d/netsage-caida-update.cron
-cron.d/netsage-scireg-update.cron
-cron.d/netsage-logstash-restart.cron
-cron.d/netsage-memberlists-update.cron 
-grnoc-netsage-deidentifier.spec
-init.d/netsage-flow-filter-daemon
-init.d/netsage-netflow-importer-daemon
-systemd/netsage-flow-filter.service
-systemd/netsage-netflow-importer.service
+cron.d/baremetal-netsage-downloads.cron
+cron.d/restart-logstash-service.cron
 systemd/logstash.service
-lib/GRNOC/NetSage/Deidentifier.pm
-lib/GRNOC/NetSage/Deidentifier/FlowFilter.pm
-lib/GRNOC/NetSage/Deidentifier/NetflowImporter.pm
-lib/GRNOC/NetSage/Deidentifier/Pipeline.pm
-lib/GRNOC/NetSage/Deidentifier/WorkerManager.pm
+systemd/sfacctd.service
+systemd/nfacctd.service
 Makefile.PL
 MANIFEST
 README.md
 website/docs/deploy/bare_metal_install.md
-reporting/flow-mongo-stats.pl
-reporting/getdata-cron.pl
-reporting/getdata.pl
-reporting/queues.txt
-reporting/queuestats.pl
-test-data/data1.json
-test-data/data2.json
-test-data/scireg2.json
-util/export-tsds
-util/generate_data.pl
-util/header.pl
-util/hist-export.pl
-util/json2lines
-util/json_to_rabbit.pl
-util/lines2json
-util/netsage-raw-data-importer
-util/netsage_raw_data_importer.xml.example
-util/nfcache
-util/RawDataImporter.pm
-util/tstat-flow-copier

Makefile.PL

@@ -6,29 +6,31 @@ use ExtUtils::MakeMaker;
 sub MY::postamble { <<'END'; }
 
 rpm:	dist
-	rpmbuild -ta grnoc-netsage-deidentifier-$(VERSION).tar.gz
+	rpmbuild -ta grnoc-netsage-pipeline-$(VERSION).tar.gz
 END
 
-sub MY::test
-	{
-    q(
-TEST_VERBOSE=1
+#sub MY::test
+#	{
+#    q(
+#TEST_VERBOSE=1
+#
+#test :	pure_all
+#	$(FULLPERL) t/TEST $(TEST_VERBOSE)
+#
+#test_jenkins : pure_all
+#	$(FULLPERL) t/TEST $(TEST_VERBOSE) --formatter=TAP::Formatter::Console
+#    );
+#}
 
-test :	pure_all
-	$(FULLPERL) t/TEST $(TEST_VERBOSE)
-
-test_jenkins : pure_all
-	$(FULLPERL) t/TEST $(TEST_VERBOSE) --formatter=TAP::Formatter::Console
-    );
-}
+#              VERSION_FROM        => 'lib/GRNOC/NetSage/Deidentifier.pm',
 
 WriteMakefile(
-              NAME                => 'grnoc-netsage-deidentifier',
+              NAME                => 'grnoc-netsage-pipeline',
               AUTHOR              => 'GRNOC Software Engineering <[email protected]>',
-              VERSION_FROM        => 'lib/GRNOC/NetSage/Deidentifier.pm',
+              VERSION             => '2.0.0',
               PL_FILES            => {},
               PREREQ_PM => {
                            },
               dist                => { COMPRESS => 'gzip -9f', SUFFIX => 'gz', },
-              clean               => { FILES => 'grnoc-netsage-deidentifier-*' },
+              clean               => { FILES => 'grnoc-netsage-pipeline-*' },
              );

README.md

@@ -3,9 +3,9 @@
 
 [![Build Status](https://travis-ci.com/netsage-project/netsage-pipeline.svg?branch=master)](https://travis-ci.com/netsage-project/netsage-pipeline)
 
-The Netsage Flow Processing Pipeline includes several components for processing network flow data, including importing, deidentification, metadata tagging, flow stitching, etc.
+The Netsage Flow Processing Pipeline includes several components for processing network flow data, including collection, deidentification, metadata tagging, flow stitching, etc.
 
-Detailed documentation is available [here](https://netsage-project.github.io/netsage-pipeline/)
+Detailed documentation is available [here]. (https://netsage-project.github.io/netsage-pipeline/)
 
 
 

bin/docker-netsage-downloads.sh.ORIG

@@ -0,0 +1,42 @@
+#!/bin/bash
+
+# Download possibly-updated files required by the Netsage Pipeline
+# Use touch to change the file time to the time of download
+
+# -for docker installations - 
+#   DOWNLOAD_PATH="/PATH-TO-GIT-CHECKOUT/downloads"
+#   SUPPORT_PATH="/PATH-TO-GIT-CHECKOUT/conf-logstash/support"
+
+# -for bare metal installations - run with sudo and use
+#   DOWNLOAD_PATH="/var/lib/grnoc/netsage"
+#   SUPPORT_PATH="/etc/logstash/conf.d/support"
+
+DOWNLOAD_PATH="-PATH-TO-GIT-CHECKOUT-/logstash-downloads"
+SUPPORT_PATH="-PATH-TO-GIT-CHECKOUT-/conf-logstash/support"
+
+# MAXMIND ASN 
+/usr/bin/wget https://scienceregistry.grnoc.iu.edu/exported/GeoLite2-ASN.mmdb -q -O $DOWNLOAD_PATH/GeoLite2-ASN.mmdb && touch $DOWNLOAD_PATH/GeoLite2-ASN.mmdb
+
+# MAXMIND CITY 
+/usr/bin/wget https://scienceregistry.grnoc.iu.edu/exported/GeoLite2-City.mmdb  -q -O $DOWNLOAD_PATH/GeoLite2-City.mmdb && touch $DOWNLOAD_PATH/GeoLite2-City.mmdb
+
+# CAIDA file
+/usr/bin/wget https://scienceregistry.grnoc.iu.edu/exported/CAIDA-org-lookup.csv -q -O $DOWNLOAD_PATH/CAIDA-org-lookup.csv && touch $DOWNLOAD_PATH/CAIDA-org-lookup.csv
+
+# SCIENCE REGISTRY
+/usr/bin/wget  https://scienceregistry.netsage.global/exported/scireg.mmdb -q -O $DOWNLOAD_PATH/scireg.mmdb && touch $DOWNLOAD_PATH/scireg.mmdb
+
+# FRGP MEMBER LIST 
+/usr/bin/wget https://scienceregistry.grnoc.iu.edu/exported/FRGP-members-list.rb -q -O $SUPPORT_PATH/FRGP-members-list.rb && touch $SUPPORT_PATH/FRGP-members-list.rb
+
+# ILIGHT MEMBER LIST 
+/usr/bin/wget https://scienceregistry.grnoc.iu.edu/exported/ilight-members-list.rb -q -O $SUPPORT_PATH/ilight-members-list.rb && touch $SUPPORT_PATH/ilight-members-list.rb
+
+# ONENET MEMBER LIST 
+/usr/bin/wget https://scienceregistry.grnoc.iu.edu/exported/onenet-members-list.rb -q -O $SUPPORT_PATH/onenet-members-list.rb && touch $SUPPORT_PATH/onenet-members-list.rb
+
+# SENSOR_GROUPS 
+/usr/bin/wget https://scienceregistry.grnoc.iu.edu/exported/sensor_groups.json -q -O $SUPPORT_PATH/sensor_groups.json && touch $SUPPORT_PATH/sensor_groups.json
+
+# SENSOR_TYPES 
+/usr/bin/wget https://scienceregistry.grnoc.iu.edu/exported/sensor_types.json -q -O $SUPPORT_PATH/sensor_types.json && touch $SUPPORT_PATH/sensor_types.json