Release 3.4.0

Hopsworks 3.4.0

Bug

HWORKS-372 explicit provenance missing community stereotype

HWORKS-397 Links to Model Serving API reference are broken

HWORKS-426 Latest versions of managed Kubernetes may not support first-party-jwt

HWORKS-538 dataset sharing acl bug

HWORKS-610 HazelcastSerializationException on JupyterJWTManager

HWORKS-632 Command.from_response_json in hopsworks-api should parse based on items field

HWORKS-638 Kafka 3 removed --zookeeper option

HWORKS-639 Template cert store passwords in onlinefs-kafka.properties

HWORKS-640 Nightly workflow and load tests should not run as admin user

HWORKS-654 extra constraints migration does not respect the order of the columns

HWORKS-663 remote_file does not support not_if condition

HWORKS-665 Bump Zookeeper dependency in Hopsworks to 3.8.2 to fix CVE-2023-34462

HWORKS-666 Configuring HA Glassfish fails when importing certificates on worker nodes

HWORKS-670 Set Flink job staging and checkpoint directories per user

HWORKS-675 Exclude logback dependencies from zookeeper library in Hopsworks

HWORKS-680 Karamel with recipe parallelism restrictions schedules tasks although a previous task has failed

HWORKS-687 ZooKeepers fail to start because they cannot resolve other ZK consul domain name

HWORKS-688 Python library installation error message not latin1_general_cs

HWORKS-689 oauth logout uri is set to redirect uri

HWORKS-711 User deletion fails with FK constraint

HWORKS-731 Model deployment with a wrong predictor script path fails with a non-intuitive error

HWORKS-735 Chef attribute to configure ldap_groups_search_filter

HWORKS-736 ndb-chef download backup should skip host key verification

HWORKS-750 Pin traitlets in project python environment

Task

HWORKS-248 Publish all images to the internal docker registry

HWORKS-343 Add support for accessing a shared feature store using project.get_feature_store()

HWORKS-351 Add support for running generic Docker commands

HWORKS-354 Remove python_kernel variable

HWORKS-489 Register test code using register_assert_rewrite

HWORKS-544 Opensearch docs creation moved to Hopsworks

HWORKS-545 Move Tags and Keyword to Hopsworks db

HWORKS-546 Feature components - remove xattr - use Hopsworks db

HWORKS-547 featurestore search - static reindex

HWORKS-572 Add uptime to status-all-local-service.sh

HWORKS-576 Test upgrade from HA cluster to HA

HWORKS-589 Remove metastore foreign key from cached/streaming feature groups

HWORKS-618 Make integration test use HA cluster def

HWORKS-627 Log more information during Project creation

HWORKS-628 Skip project creation failure alert if a project with the same name exists

HWORKS-637 Add Release Candidate version to spark-chef for hsfs

HWORKS-642 Online feature group logs

HWORKS-646 Move airflow dags to hopsfs

HWORKS-647 Replace Chef with Cinc

HWORKS-667 MySQL X plugin socket should not be at /tmp

HWORKS-672 Remove angular file browser in old UI

HWORKS-673 Remove Hopsworks Kagent monitor

HWORKS-678 Allow configuration of RDRS with user supplied certificates

HWORKS-681 Flink apis for hopsworks python library

HWORKS-682 Include return type and description for model.download()

HWORKS-684 Separate job scheduler from feature monitoring PR

HWORKS-685 Add HOPS_SERVICE_USER account type to Python Environment endpoints

HWORKS-686 Zip and push airflow image tar to master in jenkins pipeline

HWORKS-696 Add documentation on project quota management

HWORKS-698 Precreate the project root dir in hopsfs

HWORKS-699 Update docker daemon security configuration

HWORKS-700 Remove kagent_disable check in conda::default and template kagent_utils there

HWORKS-701 Add date filter to python environment history

HWORKS-702 Automate setup of RonDB Global replication

HWORKS-703 Add consul/enabled flag to disable consul operations if consul is not available

HWORKS-704 hopsworks::install should create the hopsmon user if it doesn't exists

HWORKS-705 Chef should not pre-register the hosts

HWORKS-706 Switch opensearch-dashboard to use consul instead of IPs

HWORKS-708 Use git api to delete git repository in the hopsworks python SDK

HWORKS-709 hops-hadoop-chef should not overwrite the tls/enabled parameter if hopsworks::default is missing

HWORKS-710 Hopsworks database should create the ts_1 tablespace for the certificates

HWORKS-712 Add option to tag RonDB management node for onlinefs

HWORKS-713 Create a symlink to hopsfs-mount directory in the default working directory in jupyter terminal

HWORKS-715 Add hops::docker dependency to rdrs and onlinefs recipes

HWORKS-716 Add append2ChefTrustAnchors to elastic::default

HWORKS-717 Remove consul dependency in hops::client

HWORKS-718 Switch elastic clients to use consul fqdn and not ips

HWORKS-719 Add chef code to configure WAN federated consul

HWORKS-720 Remove Kafka waiter in Onlinefs systemd script if external kafka

HWORKS-723 Add rondb_replication to the list of forbidden project names

HWORKS-724 Implement multiregion controller in Hopsworks

HWORKS-726 Delete Chef cached attributes json in beginning of new installation

HWORKS-727 Bump Hops version to 3.2.0.11-RC0

HWORKS-728 Add secondary region attribute in Chef

HWORKS-734 Remove jdbc pool when initializing online feature store database for project

HWORKS-737 Deleted project leads to NPE when processing SearchFSCommand

HWORKS-738 Recipe to setup Judge service

HWORKS-740 Expat - disable provenance on existing projects

HWORKS-741 Upgrade Grafana to version 9.3.16

HWORKS-749 Remove chef-ulimit dependency from hopsworks-chef Berksfile

HWORKS-752 Flink workflow tests should get latest release candidate dynamically

Feature Store

Epic

FSTORE-709 Rest API to retrieve data from the online feature store

Bug

FSTORE-728 Improve error msgs when wrong API key provided for hopsworks.login()

FSTORE-824 Shows undefined in accept shared feature store prompt

FSTORE-842 Setting fg.insert(storage="online") adds data to offline as well

FSTORE-857 Appending to an online feature group fails after upgrade

FSTORE-879 Fix consecutive slashes (//) in file selector component

FSTORE-906 Training data shouldn't be recreated when recomputing stats

FSTORE-907 Recomputing statistics on feature groups without time-travel enabled fails

FSTORE-921 Resolve the set of serving key in get_feature_vector

FSTORE-928 When hitting limit of number of projects that one user can create, deleting a project doesn't work as expected

FSTORE-942 ADLS connector path issues

FSTORE-946 Orphan statistics are not removed from the DB

FSTORE-965 fastavro fails to build when installing HSFS

FSTORE-966 Removing spark options on update does not work on bigquery connector

FSTORE-969 Scheduled jobs are not triggering when expected

FSTORE-993 HSFS Java should use UTF-8 when creating the StringEntity

FSTORE-999 Cannot get feature vector from a shared feature view

FSTORE-1012 Hopsworks-api got an unexpected keyword argument 'feature_store_topic'

FSTORE-1016 Onlinefs schema issue

Task

FSTORE-50 Tutorial how to write custom transformation function for feature view

FSTORE-449 Use Caffeine cache for schemas

FSTORE-576 Tutorial for transformation functions in sklearn

FSTORE-654 Should throw proper error if data format is not provided.

FSTORE-672 Add shared feature store workflows tests

FSTORE-742 Statistics API

FSTORE-803 Beam support for HSFS java client

FSTORE-839 Single Kafka topic per project

FSTORE-840 Use existing Kafka cluster

FSTORE-860 EJB timer to delete orphan statistics

FSTORE-862 Allow naming of Kafka topics for data ingestion

FSTORE-865 Update online feature store logs

FSTORE-866 Add option to deprecate a feature group

FSTORE-877 Create Training Datasets in Flyingduck

FSTORE-884 Flying Duck - Write End-User Documentation

FSTORE-887 Compute statistics on both untransformed and transformed data when creating a Training Dataset

FSTORE-888 Upgrade flink 1.14 to flink 1.17

FSTORE-891 SQL and expat migrations for Statistics

FSTORE-899 Capability Write Up - Feature Engineering in Flink

FSTORE-903 External Warehouse Access Flyingduck

FSTORE-904 Flying Duck - Write Blog Article

FSTORE-908 Upgrade scheduler to near real-time standard and cron frequencies

FSTORE-926 Hopsworks BigQuery Storage Connector can't create and write data to external FG from Dataproc cluster

FSTORE-938 Remove enabled flag on Feature Monitoring Configuration to only use scheduler flag

FSTORE-939 Change name of Stream Feature Group's DeltaStreamer Job

FSTORE-940 Constraints for job and scheduler deletion

FSTORE-947 Tutorials Streamlit Fix

FSTORE-948 Add ASOF Joins to Hopsworks Query Constructor

FSTORE-952 Single Kafka topic per project

FSTORE-956 FS REST API logging and metric

FSTORE-958 Rename and split tutorial notebooks as feature, training and inference pipelines

FSTORE-961 tutorial for bytewax / hsfs integration

FSTORE-964 Exclude extra data from get all feature groups api

FSTORE-970 Include schedule in the job DTO

FSTORE-977 FlyingDuck - Slides Pydata

FSTORE-978 Hopsworks recommender system tutorial

FSTORE-979 Flink hsfs client workflow test

FSTORE-981 Make clear that from python env only stream=True is supported

FSTORE-982 Adopt serving key in feature store Rest API

FSTORE-984 FlyingDuck BigQuery Connector

FSTORE-988 Add PIT join tests to workflow/load tests

FSTORE-994 Use non-ascii characters in user first and last name in nightly tests

FSTORE-995 Return feature value in get_feature_vectors when entry is str instead of int

FSTORE-996 FS REST API Alert

FSTORE-1010 Don't leave orphaned subjects when deleting online enabled fg

FSTORE-1013 Hopsworks-api allow to specify feature_store_topic when creating project

FSTORE-1014 [Online FS REST API] Failed to return features when only primary key is selected

FSTORE-1020 Unable to delete a FeatureGroup with a FeatureView with filters

FSTORE-1027 FS docs fix

FSTORE-1028 POC for prospect customer

FSTORE-1031 Update Colab links in docs