v23.11.00

🚨 Breaking Changes

• Separate Pipeline type inference/checking & MRC pipeline construction (#1233) @dagardner-nv
• Remove multiprocess download option (#1189) @efajardo-nv

🐛 Bug Fixes

• CVE-2023-47248 Mitigation (#1399) @mdemoret-nv
• Fixing the hammah and phishing validation pipelines (#1398) @mdemoret-nv
• Fix the SID Viz workflow shutdown process with the new pipeline shutdown process (#1392) @mdemoret-nv
• Fix race condition in the C++ impl for the pre-process fil stage (#1390) @dagardner-nv
• Fixing the conda-build with DOCA enabled and upgrading to CMake 3.25 (#1386) @mdemoret-nv
• Add missing milvus marker (#1385) @dagardner-nv
• Register DataBricksDeltaLakeSourceStage with the CLI (#1384) @dagardner-nv
• Guard optional dependencies in try/except blocks (#1382) @dagardner-nv
• Fix test_vdb_upload_pipe (#1381) @dagardner-nv
• DFP container updates (#1347) @efajardo-nv
• Removed Mutex Related Milvus Tests (#1325) @bsuryadevara
• Pin cuda-python to 11.8.2 as a work around for 11.8.3 incompatibility. (#1320) @drobison00
• Forward-merge branch-23.07 to branch-23.11 [resolved conflicts] (#1246) @dagardner-nv
• Fix tests to support mlflow v2.7 (#1220) @dagardner-nv
• Update DFP training model_kwargs (#1216) @efajardo-nv
• Fix Kafka offset checking test (#1212) @dagardner-nv
• Add review_requested as a trigger & increased timeouts for camouflage (#1200) @dagardner-nv
• Remove multiprocess download option (#1189) @efajardo-nv
• Update feature length for test_abp_fil_e2e benchmark (#1188) @dagardner-nv
• Make manual_seed an autouse fixture for gnn_fraud_detection_pipeline tests (#1165) @dagardner-nv
• Update Downloader utility class to use static dask cluster (#1161) @efajardo-nv
• Update to handle GitHub CLI not installed (#1157) @efajardo-nv
• Update TimeSeries stage to also work with Production DFP (#1121) @efajardo-nv
• Fix issue where DFPFileToDataFrameStage logs messages about S3 even when S3 is not in use (#1120) @dagardner-nv
• Fix Tests broken by Test Reorganization (#1118) @cwharris
• Break circular reference issue causing a memory leak (#1115) @dagardner-nv
• Fix intermittent failures in test_dfencoder_distributed_e2e test (#1113) @dagardner-nv
• Resolve forward merger conflices for branch-23.11 (#1092) @mdemoret-nv
• Fix local CI building from a specific commit (#1083) @dagardner-nv

📖 Documentation

• Grafana example readme update (#1393) @efajardo-nv
• Align model card requirements (#1388) @drobison00
• Docs update to indicate use of conda-merge to generate install files (#1387) @drobison00
• Stage documentation improvements (#1362) @dagardner-nv
• Documentation patch for Examples (#1357) @pranavm7
• Update developer documentation to reflect new compute_schema changes (#1341) @dagardner-nv
• Create LICENSE.psycopg2 (#1295) @exactlyallan
• Fix documentation for morpheus.loaders.sql_loader (#1264) @dagardner-nv
• Phishing example fix (#1215) @efajardo-nv
• ABP PCAP detection readme update (#1205) @efajardo-nv
• Command line examples for module-based DFP pipelines (#1154) @efajardo-nv
• Update DFP E2E Benchmarks README to use dev container (#1125) @efajardo-nv
• Less intrusive doc builds (#1060) @dagardner-nv

🚀 New Features

• Add source & stage decorators (#1364) @dagardner-nv
• Add a Vector Database Service to allow stages to read and write to VDBs (#1225) @bsuryadevara
• CI test stage no longer depends on build stage (#1219) @dagardner-nv
• Updates for MRC/Morpheus to build in the same RAPIDS devcontainer environment (#1171) @cwharris
• KafkaSourceStage OAuth Callback Support (#1169) @cwharris
• GitHub Project Automation and Infra Updates (#1168) @jarmak-nv
• Elasticsearch Sink Module (#1163) @bsuryadevara
• RSS Source Stage for Reading RSS Feeds (#1149) @bsuryadevara
• Add parser_kwargs to FileSourceStage to support json files (#1137) @cwharris
• Add a --viz_direction flag to CLI (#1119) @dagardner-nv
• Adds support to read and write to Databricks delta tables (#630) @pthalasta

🛠️ Improvements

• LLM C++ test and doc updates (#1379) @efajardo-nv
• Merge fea-sherlock feature branch into branch-23.11 (#1359) @drobison00
• Make dfp_azure_pipeline inference output file configurable. (#1290) @drobison00
• Loosen nodejs version restriction (#1262) @dagardner-nv
• Use conda environment yaml's for training-tuning-scripts (#1256) @efajardo-nv
• Cherry pick to pull in august DFP enhancements (#1248) @drobison00
• [DRAFT] Add model and experiment template 'click' options to dfp example pipelines, and make model names Databricks compatible. (#1245) @drobison00
• Separate Pipeline type inference/checking & MRC pipeline construction (#1233) @dagardner-nv
• Adopt updated camouflage-server & fix test_dfp_mlflow_model_writer (#1195) @dagardner-nv
• Add development deps to dependencies.yaml (#1193) @cwharris
• Update to clang-16 & boost-1.82 (#1186) @dagardner-nv
• Scope Zookeeper & Kafka fixtures to session (#1160) @dagardner-nv
• Use copy-pr-bot (#1159) @ajschmidt8
• Update dfp_training stage to support ControlMessages or MultiDFPMessages (#1155) @drobison00
• Prefer conda package over pip dependencies (#1135) @cwharris
• Add tasks and metadata properties to python ControlMessage (#1134) @cwharris
• Eliminate redundant code blocks in modules and stages (#1123) @bsuryadevara
• update devcontainer base to 23.10 (#1116) @cwharris
• Slimmed down CI runners and published artifact urls (#1112) @dagardner-nv
• Updating tests to force .pyi files to be committed into the repo (#1111) @mdemoret-nv
• ABP nvsmi sample data generation (#1108) @efajardo-nv
• Reorganize C++ Tests (#1095) @cwharris
• Improve gitutils.py by using the Github CLI when available (#1088) @mdemoret-nv
• Fixing linting errors which could not be resolved in 23.07 (#1082) @mdemoret-nv
• Move testing utilities into tests/_utils (#1065) @dagardner-nv
• Update Versions for v23.11.00 (#1059) @mdemoret-nv
• Update GNN stellargraph with DGL (#1032) @tzemicheal
• Implement rest data loader (#986) @yuchenz427
• Adding HTTP sources & sinks (#977) @dagardner-nv

New Contributors

• @pthalasta made their first contribution in #630
• @exactlyallan made their first contribution in #1295
• @pranavm7 made their first contribution in #1357

Full Changelog: v23.07.03...v23.11.00