v2.2.0

Release notes - Hopsworks - Version 2.2.0

Bug

HOPSWORKS-2482 Read Hudi Feature Group from Hive Engine fails with unknown fg0

HOPSWORKS-2481 Only default deployments must be enabled when KFServing is not installed

HOPSWORKS-2473 dataset read acl list does not get cleared on dataset delete

HOPSWORKS-2471 fix prepared statement parameter for string PKs

HOPSWORKS-2447 Pip conflicts modal not closing when clicking confirm

HOPSWORKS-2415 reset elastic client in case it uses the wrong certificate

HOPSWORKS-2414 Verify that API token used when starting Jupyter with AutoPushOnShutdown enabled has write access to git repo

HOPSWORKS-2352 Snowflake connector requires pytz <= 2021.0

HOPSWORKS-2303 Change python pip search to use pypi simple index and JSON API

HOPSWORKS-2293 Fix test for kvm

Improvement

HOPSWORKS-2484 Fix SSL/TLS settings for Glassfish network listeners

HOPSWORKS-2460 Create admin end-point for user management

HOPSWORKS-2433 Azure Root CA missing from glassfish trust store.

HOPSWORKS-2432 TLS for Resource Manager UI

HOPSWORKS-2423 Advertise Rondb managment node through consul

HOPSWORKS-2422 Advertise Kafka brokers through Consul

HOPSWORKS-2421 [ndb-chef] Fix config.ini for RonDB

HOPSWORKS-2408 Feature store activity should return validation results

HOPSWORKS-2405 Upgrade Python version we install in recipes

HOPSWORKS-2404 remove metadata designer code

HOPSWORKS-2396 Add AzureML Studio notebook

HOPSWORKS-2368 Do not require API token for working with public GitHub and Gitlab repos using Jupyter

HOPSWORKS-2367 Hopsworks Connector Docs

HOPSWORKS-2366 Replace service with systemctl in recipes

HOPSWORKS-2351 derived fg provenance tests - links/usage

HOPSWORKS-2345 Get Kafka Broker Endpoints through REST Api

HOPSWORKS-2336 It should not be mandatory to provide a version when installing a pip or conda library

HOPSWORKS-2314 Snowflake connector

HOPSWORKS-2247 Replace feature store job tracking with provenance

HOPSWORKS-2139 Use Hudi commit ID as statistics timestamp

HOPSWORKS-2071 Data validation v2

HOPSWORKS-2052 Save preferences for dataset and experiments filter

New Feature

HOPSWORKS-2474 Docker job

HOPSWORKS-2458 Online-FS Service Integration

HOPSWORKS-2399 Add new frontend deployment

HOPSWORKS-2384 Add support for coalesce in training dataset

HOPSWORKS-2195 Attach Jupyter configuration to notebook as Xattr

HOPSWORKS-1978 schematized tags

Sub-task

HOPSWORKS-2193 Add support for request batching

HOPSWORKS-2185 Implement inference controller

HOPSWORKS-2106 Implement storage initializer webhook for KFServing

Task

HOPSWORKS-2476 Bump hops dependencies to 3.2.0.3-RC0

HOPSWORKS-2348 Disable storage and compute quota by default

HOPSWORKS-2342 Add PyTorch Maggy hyperparameter optimization example notebook

HOPSWORKS-2326 Display alert if a more recent version of hops or hsfs exists in python service

HOPSWORKS-2317 Remove featurestore module from hops-util-py and hops-util

HOPSWORKS-2305 Remove security questions

Release 2.1.0

Release notes - Hopsworks - Version 2.1.0

Bug

HOPSWORKS-2359 Always show GPU configuration fields for Jobs and Jupyter
HOPSWORKS-2358 Remove ROCM GPU monitoring code
HOPSWORKS-2346 Identifying if a project is using the base image on ECR does not work
HOPSWORKS-2338 handle empty tfrecord files in TFDataEngine
HOPSWORKS-2333 create_training_dataset method throws Connector not found exception
HOPSWORKS-2311 Template required ec2-init parameters for the unmanaged version
HOPSWORKS-2307 Docker garbage collection should not remove base or python37 images for any hopsworks version
HOPSWORKS-2302 Hide python pip search bar as underlying pip search is now deprecated
HOPSWORKS-2294 Installing a python library from git should enclose the url in quotes
HOPSWORKS-2283 Reading from the online feature store doesn't work on Databricks
HOPSWORKS-2249 provenance elasticsearch - count and aggregations should be separate methods
HOPSWORKS-2248 Python file path is not shown in Job details
HOPSWORKS-2238 Feature Group schema primary key info not set for partition keys
HOPSWORKS-2219 Remove NaN from statistics
HOPSWORKS-2202 Docker registry caches deleted manifests
HOPSWORKS-2199 Glassfish consul domain should be included in Kafka super.users
HOPSWORKS-2198 Cannot get the schema of a shared kafka topic
HOPSWORKS-2187 Add anaconda user to hadoop group only if the group exists. Create hops-system Anaconda env conditionally
HOPSWORKS-2182 Disable unattended-upgrades in Ubuntu
HOPSWORKS-2174 issues attaching xattr to shared files
HOPSWORKS-2164 Add API Key annotation on the commits endpoint
HOPSWORKS-2159 ruby tests fix
HOPSWORKS-2158 Query constructor ignores more than 2 joining conditions
HOPSWORKS-2151 Tag validation doesn't work for key only tags
HOPSWORKS-2145 Flask (sklearn) race condition when reading pid file
HOPSWORKS-2102 Conversion of Jupyter notebooks to Python programs should consider notebook kernel
HOPSWORKS-1984 Exception when creating online feature group without primary key

Task

HOPSWORKS-2348 Disable storage and compute quota by default
HOPSWORKS-2337 Upgrade hudi to 0.7.0 version
HOPSWORKS-2327 Ui fixed for generating training datasets
HOPSWORKS-2324 Display alert if project is using older docker image
HOPSWORKS-2320 Set PARQUET as default for non-hudi offline feature groups
HOPSWORKS-2287 Bump up docker.io version for ubuntu
HOPSWORKS-2275 Remove InodeEncodingStatus Bean
HOPSWORKS-2274 Feature store "NOT FOUND" errors should return 404
HOPSWORKS-2241 Consul security recipe should not try to get Hopsworks IP if not needed
HOPSWORKS-2206 HSFS profile to install with and without Hive dependencies
HOPSWORKS-2196 Remove hudiEnabled flag and sync endpoint from hopsworks
HOPSWORKS-2189 Update spark-avro package for to_avro/from_avro methods
HOPSWORKS-2188 Remove partition key requirements for HUDI feature groups and allow users to set precombine keys
HOPSWORKS-2178 Remove leftover ldconfig in hops::install
HOPSWORKS-2163 Add missing chef changes of HOPSWORKS-2057 in master
HOPSWORKS-2144 Bump hops dependencies to 3.2.0.2-RC0
HOPSWORKS-2138 remove slider and LLAP
HOPSWORKS-2132 Check that base environment does not contain broken requirements when building docker
HOPSWORKS-2131 Add pip check command to Python service
HOPSWORKS-2127 Bump TensorFlow to 2.4 and PyTorch to 1.7
HOPSWORKS-2103 Add documentation for connecting to the Hopsworks Kafka cluster from an external client

Improvement

HOPSWORKS-2341 allow hsfs to get all metadata from feature store within experiment function
HOPSWORKS-2315 [hops-hadoop-chef] Template in Chef application key size
HOPSWORKS-2280 Save statistics configuration also for training datasets
HOPSWORKS-2277 Add VARBINARY(N) to proposed types for online features
HOPSWORKS-2201 Move to delete API for removing images from local docker registry
HOPSWORKS-2200 set fs.azure.skipUserGroupMetadataDuringInitialization to true for ADLSv2
HOPSWORKS-2183 Add feature group, training dataset, storage connector counters to FeaturestoreDTO summary
HOPSWORKS-2180 Disable transactions when Spark writes to the online feature store
HOPSWORKS-2177 extend on-demand feature groups parquet/orc/delta/hudi/csv tables
HOPSWORKS-2176 Add API key annotations to provenance endpoint
HOPSWORKS-2172 Reduce default varchar size for string features
HOPSWORKS-2168 Add keyword support to feature groups and training datasets
HOPSWORKS-2157 Store information about primary and precombine keys for time travel enabled feature groups separately from Hive Metastore
HOPSWORKS-2143 experiments/models list for any user with access (including shared)
HOPSWORKS-2129 Should run conda clean after installing a package
HOPSWORKS-2120 enable full provenance by default
HOPSWORKS-2119 Create an admin rest endpoint for user registration
HOPSWORKS-2101 Add resource to download files using api key
HOPSWORKS-2026 Add "converting notebook" job state
HOPSWORKS-1817 Replace influxdb with prometheus + pushgateway for Spark applications
HOPSWORKS-1591 UI improvements for storage connectors
HOPSWORKS-1193 Make deleting a project more intuitive
HOPSWORKS-288 Allow editing a job's configuration property without clearing the rest

New Feature

HOPSWORKS-2197 Add ADLS storage connector for the feature store
HOPSWORKS-2184 Install Python libraries from Anaconda environment file
HOPSWORKS-2165 Sup...

Release 2.0.0

Release notes - Hopsworks - Version 2.0.0

Task

HOPSWORKS-2287 Bump up docker.io version for ubuntu
HOPSWORKS-2186 Remove installation of slack operator in Airflow cookbook
HOPSWORKS-2161 Option to disable just the Registration UI
HOPSWORKS-2156 Pin all TensorFlow projects in Docker image
HOPSWORKS-2126 Cleanup old hops-util(-py) feature store notebooks
HOPSWORKS-2125 Bump hops dependencies to 3.2.0.1-RC0
HOPSWORKS-2109 Remove base_env from PythonDep
HOPSWORKS-2094 Setup GPU monitoring without enabling it even when it's not a GPU deployment
HOPSWORKS-2090 Bump up docker.io version for Ubuntu
HOPSWORKS-2072 [hops-hadoop-chef] Recipe should check if Hops distro has been build with correct profiles
HOPSWORKS-2043 Upgrade base and airflow conda environment to python 3.7
HOPSWORKS-2042 Unmanaged Hopsworks Image
HOPSWORKS-2041 Bump Hops Hadoop version to 3.2.0.1-SNAPSHOT
HOPSWORKS-2040 Docker registry does not need to be versioned
HOPSWORKS-2039 Bump hops dependencies to 3.2.0.0-RC3
HOPSWORKS-2037 Rename map_fun and wrapper_fun to train_fn
HOPSWORKS-1945 Store AWS access/secret key in the secret store.
HOPSWORKS-1944 Split feature store tests in several categories
HOPSWORKS-1387 Upgrade to NDB 8

Bug

HOPSWORKS-2202 Docker registry caches deleted manifests
HOPSWORKS-2182 Disable unattended-upgrades in Ubuntu
HOPSWORKS-2162 Airflow page does not open
HOPSWORKS-2151 Tag validation doesn't work for key only tags
HOPSWORKS-2150 S3 storage connector should not bother about secrets if iam role is configured
HOPSWORKS-2149 Disable generating and sharing ssh key freely
HOPSWORKS-2147 Add feature group api scope to statistics resource endpoint
HOPSWORKS-2146 Resolve namenode consul name before setting feature group location
HOPSWORKS-2145 Flask (sklearn) race condition when reading pid file
HOPSWORKS-2128 Create api key not adding the right scope
HOPSWORKS-2122 Backend not handling null TimeTravelFormat sent by client
HOPSWORKS-2116 Add --network option to docker build
HOPSWORKS-2095 Uninstalling conda libraries doesn't work in airgapped environments
HOPSWORKS-2092 Overwriting a feature group by dropping/recreating loses default value of appended features
HOPSWORKS-2081 Fix search for registration disabled
HOPSWORKS-2080 Hops services should wait for Hopsworks before they start
HOPSWORKS-2068 Singleton initialization failure
HOPSWORKS-2067 shared featuregroup creation
HOPSWORKS-2065 Restart systemd units after failure
HOPSWORKS-2064 UI fixes for data preview
HOPSWORKS-2062 ProjectException should set prefix usrMsg from the causing exception's messge
HOPSWORKS-2061 Datasets browser order files by name not working while paginating
HOPSWORKS-2058 Conda environments are not removed when deleting a project
HOPSWORKS-2056 Change DNS server for docker containers in Localhost deployments
HOPSWORKS-2054 Increment timeout when launching Jupyter locally
HOPSWORKS-2053 Fix systemd dependencies
HOPSWORKS-2051 Save preferences for jobs/page filter
HOPSWORKS-2048 Jupyter-git missing configuration for git_backend
HOPSWORKS-2047 Non LTS Nvidia Driver version doesn't work on Azure VMs
HOPSWORKS-2038 Python client can't handle disabled online feature store
HOPSWORKS-2015 Users should not be allowed to create Dataset 'Projects'
HOPSWORKS-1988 IPynb converter fails for notebooks which contain special characters in the filename
HOPSWORKS-1918 Spark executor memory not modifiable from UI when starting jupyter
HOPSWORKS-1876 Data validation UI doesn't fill in Job parameters
HOPSWORKS-1861 API keys not all selectable for Github Jupyter notebooks

Improvement

HOPSWORKS-2140 Increase retries for HDFS Mover in hops-hadoop-chef
HOPSWORKS-2130 Refactor on-demand feature groups for HSFS
HOPSWORKS-2124 Remove R installation from spark::yarn
HOPSWORKS-2123 Add option to disable datasource validation when deploying Hopsworks
HOPSWORKS-2113 Add endpoint to list a user's projects
HOPSWORKS-2112 Return the user activation time as part of the userProfileDTO
HOPSWORKS-2093 remove env/pkgs dir symlinks in conda installation
HOPSWORKS-2091 Label/Prediction feature metadata for training datasets
HOPSWORKS-2084 Remove default_storage option from feature groups
HOPSWORKS-2077 add support for reading csv files as tf data object and rename feeder to tf_data
HOPSWORKS-2046 Kafka broker should use zookeeper fqdn instead of IP
HOPSWORKS-2045 artifact usage
HOPSWORKS-2044 Add option to return an error message if a user starts a YARN job but no nodemanagers are available
HOPSWORKS-2025 Delete a job's executions
HOPSWORKS-2010 Upgrade hudi to 0.6.1-SNAPSHOT for hsfs
HOPSWORKS-1986 Add redshift/snowflake examples and libraries
HOPSWORKS-1982 Deequ statistics for Feature Groups/Training Datasets
HOPSWORKS-1878 Add default args for Jobs
HOPSWORKS-1847 online feature store storage connectors should not be stored in the database
HOPSWORKS-1833 Move beam related files from conf directory to lib
HOPSWORKS-1769 enable publishing models to other projects
HOPSWORKS-1374 Add Hudi API to HSFS and commits to the FeatureStore metadata

New Feature

HOPSWORKS-2014 Manage AWS roles per application in Hopsworks
[HOPSWO...

Releases 1.4.0 - 1.4.1

    Release notes - Hopsworks - Version 1.4.0

Improvement

• [HOPSWORKS-1314] - It should be possible to start Jupyter with a Shared Dataset as base directory
• [HOPSWORKS-1655] - Refactor FeatureGroupService for the new clients
• [HOPSWORKS-1657] - Generate per system user X.509 certificates
• [HOPSWORKS-1752] - Changning Deployment Repos for Enterprise Edition
• [HOPSWORKS-1762] - Add support for the Spark-Tfrecord library
• [HOPSWORKS-1786] - xattr refactor - tags and featurestore search
• [HOPSWORKS-1791] - [ePipe] Add support for large XAttrs
• [HOPSWORKS-1804] - Integrate Prometheus with Consul
• [HOPSWORKS-1815] - manage kubernetes through kagent
• [HOPSWORKS-1816] - Use consul for hive-site and metrics.conf configuration
• [HOPSWORKS-1825] - Connect to MySQLd's using 127.0.0.1 instead of private IP
• [HOPSWORKS-1829] - Register Kibana with consul
• [HOPSWORKS-1834] - Add XAttrs configurations to chef
• [HOPSWORKS-1852] - Jupyter server admin page returns 500
• [HOPSWORKS-1854] - Add datanode replacement upon failure parameters
• [HOPSWORKS-1883] - Add JOB audience to feature group delete endpoint
• [HOPSWORKS-1892] - TrainingDataset/FeatureGroup create automatically increments version if no version provided by user
• [HOPSWORKS-1897] - Add example for submitting Spark/PySpark jobs remotely
• [HOPSWORKS-1899] - Add user and project create functions in hops Python library
• [HOPSWORKS-1921] - Show certificate password in the UI instead of sending it by email.
• [HOPSWORKS-1924] - Add Kafka scope to API keys
• [HOPSWORKS-1925] - Upgrade conda to version 2020.07
• [HOPSWORKS-1935] - Show project ID in the project settings
• [HOPSWORKS-1948] - Distribute HSFS in the base environment
• [HOPSWORKS-1954] - Use hopsworks version as a tag for all docker images
• [HOPSWORKS-1972] - Add new api_key_value to featurestore connect method
• [HOPSWORKS-1974] - [DOC] large xattributes - usage and upgrade details
• [HOPSWORKS-1975] - Add support for passing job arguments in the Airflow workflow builder
• [HOPSWORKS-1976] - New cloud installer script
• [HOPSWORKS-1979] - Pydoop in hops-util-py should be able to handle hopsfs scheme
• [HOPSWORKS-1996] - Allow setting additional trusted registry in container-executor
• [HOPSWORKS-1999] - Fix project cleanup
• [HOPSWORKS-2006] - Fall back to safe default if image name not set for project
• [HOPSWORKS-2008] - Bump Maggy version and update hops-examples for TF2.2
• [HOPSWORKS-2033] - UI update of jobs

New Feature

• [HOPSWORKS-1005] - Index Jupyter notebook server logs in elastic
• [HOPSWORKS-1691] - Use docker to run user code.
• [HOPSWORKS-1702] - Add user registration page in Admin UI and add option to disable default sign up page
• [HOPSWORKS-1839] - Jupyter GitLab support
• [HOPSWORKS-1849] - ACL for datasets
• [HOPSWORKS-1896] - Generate API key for admin Role
• [HOPSWORKS-1902] - Add support for managed docker registries
• [HOPSWORKS-1908] - Add flink remote client in hops-examples

Bug

• [HOPSWORKS-1421] - Hopsworks ignores chef parameter to set default max number of user projects
• [HOPSWORKS-1504] - Workflow builder should sanitize job names
• [HOPSWORKS-1568] - Installing Node.js must work for airgapped deployments
• [HOPSWORKS-1623] - Overwrite config.ini for ndb-chef
• [HOPSWORKS-1633] - Sparkmagic links to Spark UI and driver logs are broken
• [HOPSWORKS-1634] - MaggyFacade throws getSingleResult for findByAppId in parallel experiments
• [HOPSWORKS-1671] - increase systemctl timeout for ndbmtd
• [HOPSWORKS-1672] - Featurestore Descriptions containing ' cannot be handled by hive
• [HOPSWORKS-1740] - HopsKafkaAdminClient initialisation fails if kafka is down/unreachable and then never recovers
• [HOPSWORKS-1768] - Restrict the usage of scan operations
• [HOPSWORKS-1782] - Set C_INCLUDE_PATH and CPLUS_INCLUDE_PATH when installing libraries in Anaconda
• [HOPSWORKS-1793] - Increase certificate keysize
• [HOPSWORKS-1794] - Make Hopsworks logo consistent with the branding
• [HOPSWORKS-1795] - Close connections in AirflowDagFacade
• [HOPSWORKS-1796] - hops-util(-py) should use consul domain to reach Hopsworks
• [HOPSWORKS-1797] - Hopsworks should listen on an internal port with certificate signed by HopsCA
• [HOPSWORKS-1799] - Is kubernetes enabled is not evaluated correctly in front-end after variables service refactoring
• [HOPSWORKS-1800] - cache provenance-epipe ...

Release 1.3.0

    Release notes - Hopsworks - Version 1.3.0

Bug

• [HOPSWORKS-1382] - Placeholders for auditing REST API requests
• [HOPSWORKS-1471] - Increase MaxNoOfTriggers and MaxNoOfFiredTriggers to delete large no of FKs
• [HOPSWORKS-1545] - Handle cases where Elasticsearch is down
• [HOPSWORKS-1546] - ruby tests
• [HOPSWORKS-1556] - Kafka controller fixes
• [HOPSWORKS-1557] - Dataset menu hidden as browser is not scrollable
• [HOPSWORKS-1558] - dataset - share with cluster - fails
• [HOPSWORKS-1559] - Truncate conda commands before upgrading to 1.1.0
• [HOPSWORKS-1560] - Fix migration of datasets (HOPSWORKS-295)
• [HOPSWORKS-1561] - Restrict access to elasticsearch data directory to elastic user
• [HOPSWORKS-1563] - Schema of a shared topic is shown in one line
• [HOPSWORKS-1565] - Removing a shared dataset removes it also from the source project
• [HOPSWORKS-1566] - provenance elastic client - index not found exception not handled
• [HOPSWORKS-1568] - Installing Node.js must work for airgapped deployments
• [HOPSWORKS-1569] - Can't deploy with disable password login flag set to true
• [HOPSWORKS-1578] - epipe operation fail on missing index
• [HOPSWORKS-1579] - Align services on the project service nav bar
• [HOPSWORKS-1581] - [hops-hadoop-log] Touch hadoop.log with correct ownership and permissions
• [HOPSWORKS-1583] - Incorrect path for PYSPARK_ARCHIVES_PATH in spark-env.sh
• [HOPSWORKS-1589] - Cannot upload to a shared dataset
• [HOPSWORKS-1590] - Kafka advertised listeners external port conflict among brokers
• [HOPSWORKS-1596] - Incorrect sklearn logs format
• [HOPSWORKS-1599] - Cannot start/stop services from admin UI due to missing CommunicationController property
• [HOPSWORKS-1600] - Services Status enum missing INIT state of kagent
• [HOPSWORKS-1601] - Append projectId to topic name in kafka tour to avoid clashes with other topics
• [HOPSWORKS-1603] - Kagent JWTHelper should retry in case of hopsworks authentication failure
• [HOPSWORKS-1604] - In hops python library, the s3 connector tries to read parquet files as type avro
• [HOPSWORKS-1609] - [ePipe] Fix version conflicts in Elasticsearch
• [HOPSWORKS-1612] - Can't view directories with name url encoded
• [HOPSWORKS-1616] - Hopsworks ear download URL is incorrect when installing with Kubernetes
• [HOPSWORKS-1619] - Can't zip/unzip directories with name containing url encoded chars
• [HOPSWORKS-1631] - [tensorflow-chef] Kernel-devel not installed correctly on some Centos flavors
• [HOPSWORKS-1636] - [hops-hadoop-chef] Configuration key in force renew-service-tokens script is wrong
• [HOPSWORKS-1637] - Papermill creates connection file without kernel_id in the filename which breaks notebook versioning
• [HOPSWORKS-1643] - Filebeat kagent multiline property is not configured for new ELK 7.2
• [HOPSWORKS-1645] - duplicate class in hopswoks-common
• [HOPSWORKS-1646] - Leftover state when updating experiments table
• [HOPSWORKS-1650] - Search does not work when in a service other than Data Sets
• [HOPSWORKS-1653] - Error from Provenance when sharing dataset to target project name including upper case characters
• [HOPSWORKS-1654] - Feature store page stuck in loading view if project name contains upper case letters
• [HOPSWORKS-1663] - ePipe crashes after rerunning featurestore tour job
• [HOPSWORKS-1664] - Shared dataset showing up in move and copy modals without project-origin prefix
• [HOPSWORKS-1669] - Only chief worker should return wrapper return value in distributed training
• [HOPSWORKS-1679] - Pin version of Java used in Berksfile
• [HOPSWORKS-1682] - Downloading a file results in renamed file with the full path prefixed in the filename
• [HOPSWORKS-1684] - [hops-util-py] nvidia-ml-py should be included in install_requires, remove tensorflow-gpu
• [HOPSWORKS-1685] - Livy server should start after ResourceManager
• [HOPSWORKS-1686] - Calling create training dataset with non existing storage connector doesn't fail
• [HOPSWORKS-1690] - nvml_monitor fails when devices does not support device calls
• [HOPSWORKS-1696] - Increase timeout for listing conda environments when creating projects
• [HOPSWORKS-1699] - OOM Exceptions when parsing feature store metadata in hops-util-py
• [HOPSWORKS-1703] - Create training dataset endpoint does not return full entity of the new training dataset
• [HOPSWORKS-1704] - fetching schema and data preview not working in featurestore shared with another project
• [HOPSWORKS-1706] - Use system's nameserver for DNS service discovery
• [HOPSWORKS-1707] - Activating users using User-Admin broken
• [HOPSWORKS-1708] - When sharing a featurestore dataset training dataset should also be shared
• [HOPSWORKS-1710] - Add permission inheritance configuration to hive-site
• [HOP...

Release 1.2.0

    Release notes - Hopsworks - Version 1.2.0

Bug

• [HOPSWORKS-1283] - Zip does not work when there is a space in the file path
• [HOPSWORKS-1379] - Timers should be rescheduled when failed and handle exceptions
• [HOPSWORKS-1382] - Placeholders for auditing REST API requests
• [HOPSWORKS-1507] - [ndb-chef] List only TCP and UDP sockets when starting MGM server
• [HOPSWORKS-1509] - Allow all OU in nodes DN in elasticsearch configuration
• [HOPSWORKS-1511] - pci-tools package not available and torchvision broken for 0.4.1
• [HOPSWORKS-1515] - airflow webserver , scheduler and sqoop should not be enabled when service.enabled is false
• [HOPSWORKS-1534] - Kibana error when recreating the project enviroment
• [HOPSWORKS-1541] - [ePipe] Skip non searchable fields for schemabased exmetadata
• [HOPSWORKS-1543] - [ePipe] Retry in case of conflict when adding datasets and projects
• [HOPSWORKS-1545] - Handle cases where Elasticsearch is down
• [HOPSWORKS-1550] - Do not upgrade pip binary in tensorflow default recipe
• [HOPSWORKS-1553] - provenance cleaner timer
• [HOPSWORKS-1556] - Kafka controller fixes
• [HOPSWORKS-1559] - Truncate conda commands before upgrading to 1.1.0
• [HOPSWORKS-1560] - Fix migration of datasets (HOPSWORKS-295)
• [HOPSWORKS-1561] - Restrict access to elasticsearch data directory to elastic user
• [HOPSWORKS-1563] - Schema of a shared topic is shown in one line
• [HOPSWORKS-1564] - Job args expat tries to set null instead of empty string
• [HOPSWORKS-1565] - Removing a shared dataset removes it also from the source project
• [HOPSWORKS-1578] - epipe operation fail on missing index
• [HOPSWORKS-1589] - Cannot upload to a shared dataset
• [HOPSWORKS-1590] - Kafka advertised listeners external port conflict among brokers
• [HOPSWORKS-1594] - HOPSWORKS-1554 breaks conda search in Hopsworks
• [HOPSWORKS-1599] - Cannot start/stop services from admin UI due to missing CommunicationController property
• [HOPSWORKS-1600] - Services Status enum missing INIT state of kagent
• [HOPSWORKS-1601] - Append projectId to topic name in kafka tour to avoid clashes with other topics
• [HOPSWORKS-1603] - Kagent JWTHelper should retry in case of hopsworks authentication failure
• [HOPSWORKS-1610] - Prometheus alerting-rules files owned by root

Improvement

• [HOPSWORKS-836] - Send message in Hopsworks to a user that has been added to a project.
• [HOPSWORKS-1257] - sudoers script should be owned by root and 700
• [HOPSWORKS-1306] - [ePipe] Add support for HA Elasticsearch
• [HOPSWORKS-1308] - [ePipe] Rewrite and fix the metrics code
• [HOPSWORKS-1481] - Add endpoints to retrieve feature groups and training datasets by name
• [HOPSWORKS-1483] - Refactor Admin API - Hosts
• [HOPSWORKS-1494] - Feature store UI improvements
• [HOPSWORKS-1527] - Check TensorBoard readiness using http in Experiments service and for running applications
• [HOPSWORKS-1532] - No need to mount an S3 bucket for the certificates in Databricks
• [HOPSWORKS-1538] - Automatically setup hive and hopsfs on databricks cluster
• [HOPSWORKS-1544] - Add endpoint to retrieve featurestore by name
• [HOPSWORKS-1554] - Add support for proxy configuration in condarc
• [HOPSWORKS-1562] - Improve databricks docs for the feature store

Task

• [HOPSWORKS-1330] - Remove deprecated LocalFsService (Charon) codebase
• [HOPSWORKS-1354] - Import(upload) template saves files in /Projects/Uploads/<Sample.json>
• [HOPSWORKS-1407] - Change error message when failing to export project-user certificates
• [HOPSWORKS-1522] - Update support link in Hopsworks and docs
• [HOPSWORKS-1525] - [hops-hadoop-chef] Add option to allow Yarn auto discover HW capabilities
• [HOPSWORKS-1528] - Remove unnecessary code to set kibana default index
• [HOPSWORKS-1536] - Right-click rename does not work in Datasets browser
• [HOPSWORKS-1540] - Upgrade PyTorch to 1.4
• [HOPSWORKS-1549] - [kagent-chef] Update Hops CA bundle after host certificate rotation
• [HOPSWORKS-1551] - Dynamically create Airflow DB connection only if Airflow is installed
• [HOPSWORKS-1575] - Replace hops.readthedocs with hopsworks.readthedocs
• [HOPSWORKS-1577] - [ndb-chef] Restrict MySQL connections to localhost for localhost installations
• [HOPSWORKS-1582] - Add a new db user for Kafka
• [HOPSWORKS-1611] - Tensorflow::default fails to install avro-python3 due to upstream changes

Release 1.1.0

    Release notes - Hopsworks - Version 1.1.0

Bug

• [HOPSWORKS-1214] - Airflow HopsworksLaunchOperator does not handle job state App Master Failed
• [HOPSWORKS-1245] - [ePipe] Fixes for xattrs handling to be work with the changes introduced in HOPS-1434 and HOPS-1451
• [HOPSWORKS-1246] - [ePipe] Check for the existence of elastic fields before delete
• [HOPSWORKS-1296] - [ePipe] log replay fails on xattr op
• [HOPSWORKS-1305] - Fix broken Anaconda export yml
• [HOPSWORKS-1318] - Featurestore should throw an error if you try to create the same featuregroup twice
• [HOPSWORKS-1323] - [ndb-chef] Fix bug in NDB cluster-init.sh
• [HOPSWORKS-1325] - Online Feature store tests fail when project name contains uppercase letters
• [HOPSWORKS-1342] - Add parameter for s3 encryption type in core-site.xml
• [HOPSWORKS-1353] - Flink timer should handle case when services are not installed
• [HOPSWORKS-1385] - HA elastic does not have a synchronization point between starting nodes and creating indices
• [HOPSWORKS-1398] - User activation date is irrational
• [HOPSWORKS-1408] - AirflowManager kills Hopsworks deployments if Airflow directory does not exist
• [HOPSWORKS-1412] - Fix create featuregroup with partition keys
• [HOPSWORKS-1413] - AWS S3 does not accept empty Authorization header when downloading from a public bucket
• [HOPSWORKS-1425] - Update stack to handle hopsfs paths starting by hopsfs instead of hdfs
• [HOPSWORKS-1426] - Avoid using 100% containers for yarn containers
• [HOPSWORKS-1428] - hops_airflow fails to install pymssql 3.0.3, pin it to 2.1.4
• [HOPSWORKS-1431] - Better handling of Jupyter/Git failures
• [HOPSWORKS-1432] - Elasticsearch node should always use its private ip
• [HOPSWORKS-1434] - Scale jvm -Xmx for Spark and Jupyter based on allocated memory
• [HOPSWORKS-1442] - Update statistics job shows app file as undefined
• [HOPSWORKS-1443] - [featurestore] Spark returning Infinity values for mean statistics of string type features
• [HOPSWORKS-1451] - Storage connector ignored when creating a training dataset from the UI
• [HOPSWORKS-1456] - [hops-hadoop-chef] Hopsworks security related properties are not templated correctly
• [HOPSWORKS-1457] - Correct systemd scripts
• [HOPSWORKS-1458] - spark should not bring any hops jar with it
• [HOPSWORKS-1461] - The JWT signing key should be decoded first before using it to sign jwt
• [HOPSWORKS-1463] - Hops ndb version not taken in account when downloading dal
• [HOPSWORKS-1464] - fix chef for HOPS-1542
• [HOPSWORKS-1484] - Shared feature stores do not appear in the UI
• [HOPSWORKS-1485] - Sparkmagic upgrades not working
• [HOPSWORKS-1488] - Deep Learning tour project doesn't create Kafka inferenceschema
• [HOPSWORKS-1489] - Increase default Heap size and max shards per node for Elasticsearch
• [HOPSWORKS-1490] - Handle Kibana failures in Hopsworks Kibana client
• [HOPSWORKS-1496] - Elastic::default does not template correctly hosts in elasticsearch.yml
• [HOPSWORKS-1497] - hops-util-py incorrectly sets Kafka config
• [HOPSWORKS-1498] - hops-util-py can't find attributes in KafkaTopicDTO
• [HOPSWORKS-1507] - [ndb-chef] List only TCP and UDP sockets when starting MGM server
• [HOPSWORKS-1508] - [kagent-chef] Update kibana x.509 certificate when host certificate is rotated
• [HOPSWORKS-1510] - Remove the code to delete index patterns since we delete kibana project index
• [HOPSWORKS-1516] - [ePipe] handle the case if XAttrAddAll is received for a deleted inode
• [HOPSWORKS-1518] - [livy-chef] Create root Hops directory
• [HOPSWORKS-1519] - The subject of ELK JWT token should be always lowercase
• [HOPSWORKS-1521] - Dataset browser refresh button not working
• [HOPSWORKS-1523] - Kibana opendistro_security plugin is not idempotent
• [HOPSWORKS-1524] - [hops-hadoop-chef] Hops services startup should be dependent on Glassfish
• [HOPSWORKS-1526] - Can't update job when doing insert into featuregroup
• [HOPSWORKS-1564] - Job args expat tries to set null instead of empty string

Sub-task

• [HOPSWORKS-295] - Datasets
• [HOPSWORKS-1241] - Refactor Kafka API
• [HOPSWORKS-1378] - [featurestore] DDL enable composite key
• [HOPSWORKS-1388] - Remove Text/Blobs from hopsworks database tables with FK

New Feature

• [HOPSWORKS-1109] - Experiments and models v2
• [HOPSWORKS-1334] - [hopsworks-cloud-sdk] SQL query for creating training datasets
• [HOPSWORKS-1338] - [featurestore] Configurable in-bucket path for exporting training datasets to S3
• [HOPSWORKS-1340] - Airflow DAG UI composer
• [HOPSWORKS-1362] - Endpoint to securely get a project user's x.509 credentials
• [

Release 1.0.0

    Release notes - Hopsworks - Version 1.0.0

Bug

• [HOPSWORKS-432] - Clean Kafka ACLs when unsharing a topic or deleting a project.
• [HOPSWORKS-1095] - Localhost (127.0.0.1) cluster definition support
• [HOPSWORKS-1114] - Rework password recovery
• [HOPSWORKS-1146] - Kagent stops reporting if env creation fails with non default repo
• [HOPSWORKS-1152] - Dataset browser not displaying contents of shared datasets with naming conflicts
• [HOPSWORKS-1172] - Upload progress not visible in the beginning of uploading large files
• [HOPSWORKS-1175] - yarnapp/user attribute not propagated correctly to yarn-site.xml
• [HOPSWORKS-1176] - Security risk for default admin password pops up even if the password has been changed
• [HOPSWORKS-1178] - [airflow-chef] Add default timezone for Airflow
• [HOPSWORKS-1180] - Chef upgrade mixes kafka brokers ids
• [HOPSWORKS-1185] - Jupyter not working when Hopsworks runs behind a LB/reverse proxy w/ TLS termination
• [HOPSWORKS-1190] - [featurestore] missing argument when writing petastorm datasets
• [HOPSWORKS-1195] - Bad handling of Job resource missing
• [HOPSWORKS-1197] - Fix broken admin and agent tests
• [HOPSWORKS-1198] - Cleanup hadoop sbin scripts
• [HOPSWORKS-1199] - ErrorCode range mismatch and TestRestCodes not testing all ErrorCodes
• [HOPSWORKS-1203] - [ePipe] Fixes for JSON handling in the XAttr table
• [HOPSWORKS-1207] - Hopsworks broken on Ubuntu 18.04
• [HOPSWORKS-1209] - filebeat_kagent does not start with kagent start-all services script
• [HOPSWORKS-1210] - Jupyter now requires the python kernel.json to be under python{2, 3} dir.
• [HOPSWORKS-1215] - Use OSProcessExecutor for Python library search
• [HOPSWORKS-1216] - tensorflow-chef install recipe CentOS hiccups
• [HOPSWORKS-1218] - Python service not working when searching or installing libraries containing 'http'
• [HOPSWORKS-1225] - add attribute for url to download spark in spark-chef
• [HOPSWORKS-1228] - Remove livy session name from sparkmagic conf file
• [HOPSWORKS-1233] - [ePipe] hivecleaner fail during project removal
• [HOPSWORKS-1234] - Remove sudo command from sudoers scripts
• [HOPSWORKS-1236] - Project cleanup does not work after a failed project creation
• [HOPSWORKS-1246] - [ePipe] Check for the existence of elastic fields before delete
• [HOPSWORKS-1251] - images not rendering in Jupyter notebooks
• [HOPSWORKS-1252] - Airflow HopsworksLaunchOperator waits for wrong Job statuses
• [HOPSWORKS-1253] - [ePipe] Rename operation should read the inodes info from the DB
• [HOPSWORKS-1263] - Project cleanup fails if X.509 to be revoked does not exist
• [HOPSWORKS-1268] - Sanitize python library name/version/reposity/url
• [HOPSWORKS-1271] - Bump HopsHive to use ORC 1.5.5
• [HOPSWORKS-1276] - Anaconda import yml file not working, removal of explicit enabling of python kernel
• [HOPSWORKS-1278] - [ePipe] The tailer should guarantee the order of epochs during recovery and while waiting for events
• [HOPSWORKS-1279] - [ePipe] Check for the existence of the xattr parent field before delete
• [HOPSWORKS-1280] - [hops-util-py] add_module not working in python kernel
• [HOPSWORKS-1282] - Exclude netty from Hudi hdfs dependency
• [HOPSWORKS-1285] - Livy is missing Yarn GENERATING_SECURITY_MATERIAL state
• [HOPSWORKS-1290] - [hops-util-py] numpy_helper and pandas_helper documentation not rendering
• [HOPSWORKS-1297] - Model Serving view not sorted
• [HOPSWORKS-1298] - Cannot remove job associated with featuregroups
• [HOPSWORKS-1300] - [kagent-chef] Change deprecated node.set on install recipe
• [HOPSWORKS-1315] - Hopsworks wrongly sets umask for HDFS client to 000
• [HOPSWORKS-1326] - setup-chef is broken
• [HOPSWORKS-1328] - [airflow-chef] Package libjpeg-devel is no longer available
• [HOPSWORKS-1329] - [hopsworks-cloud-sdk] Fix connecting from Databricks
• [HOPSWORKS-1336] - [kagent] Host private key is not converted to PKCS#1 if it's an upgrade
• [HOPSWORKS-1343] - Airflow Hopsworks connection hook ignores configured connections
• [HOPSWORKS-1350] - [featurestore] default training dataset connector not found
• [HOPSWORKS-1355] - Conda solver is problematic
• [HOPSWORKS-1357] - flyway_schema_history fix for versions < 0.6 doesn't work after 1.0.0
• [HOPSWORKS-1358] - Handle glassfish upgrade from 4.1.2.174 to 4.1.2.181
• [HOPSWORKS-1359] - MySQL server reinitialized during update to 1.0.0
• [HOPSWORKS-1360] - Data validation JS controller does not properly include mainCtrl
• [HOPSWORKS-1365] - When upgrading ignore anaconda base env's lib dir of previous installations
• [HOPSWORKS-1368] - NullPointerException when deleting a project without Onlin...

0.10.0

    Release notes - Hopsworks - Version 0.10.0

Bug

• [HOPSWORKS-197] - Turning on and off tours is a terrible user experience
• [HOPSWORKS-715] - Hopsworks singletons are not single
• [HOPSWORKS-770] - Disable http port and TLS 1.0 on glassfish
• [HOPSWORKS-849] - kagent - conda can get stuck
• [HOPSWORKS-903] - Jupyter cleanup and timer bugfixes
• [HOPSWORKS-904] - Users should be able to specify version of Python library to install when behind PyPi proxy
• [HOPSWORKS-905] - remote_material_references is not cleaned up during project deletion
• [HOPSWORKS-913] - Feature store quota missing from admin UI in hopsworks
• [HOPSWORKS-917] - Non HA admin UI when HA namenodes
• [HOPSWORKS-920] - Job monitor thread gets stuck if job is killed while running
• [HOPSWORKS-927] - TensorBoard could not bind to an unsupported address family
• [HOPSWORKS-928] - [Featurestore] Bug: nested spark schemas are not automatically translatable to Hive
• [HOPSWORKS-929] - [Featurestore] Don't allow hyphen in featuregroup names because Hive do not allow it
• [HOPSWORKS-930] - [Featurestore] bug, in API retrieval code in feature-registry featurestore is undefined
• [HOPSWORKS-932] - Jobs UI date selector should persist user-selected values
• [HOPSWORKS-937] - Adjust Cgroup configuration to account for longer Cgroup deletion and make CPU utilization configurable
• [HOPSWORKS-938] - Adding pyFiles to Jupyter does not add it to python path
• [HOPSWORKS-941] - Fix dataset access request and related tests
• [HOPSWORKS-945] - Zipping and unzipping is broken with Hops TLS
• [HOPSWORKS-948] - [hops-hadoop-chef] Copy hadoop logs utility is not removed from crontab
• [HOPSWORKS-949] - [airflow-chef] Install recipe does not create root hops installation directory
• [HOPSWORKS-951] - sqoop and airflow bugfixes
• [HOPSWORKS-953] - [kagent-chef] kagent falsely assumes service is dead when it restarts
• [HOPSWORKS-955] - [airflow-chef] Use assigned roles in JWT to determine if a user is admin
• [HOPSWORKS-958] - AirflowJWTManager randomly does not renew tokens
• [HOPSWORKS-962] - copy_to_hdfs is broken in hops-util-py
• [HOPSWORKS-965] - Certificate enddate should always be UTC
• [HOPSWORKS-968] - Dataset browser pagination and select all not working
• [HOPSWORKS-970] - upgrade chef ulimit cookbook
• [HOPSWORKS-972] - Missing sasl/sasl.h breaks PyHopsHive installation
• [HOPSWORKS-973] - Dela install should create the hops group before adding members to it
• [HOPSWORKS-974] - Blank padded day in rspec breaks tests
• [HOPSWORKS-975] - [airflow-chef] Change restart policy of airflow scheduler
• [HOPSWORKS-976] - Airflow file manager does not refresh secret directory when navigate to another project
• [HOPSWORKS-977] - Pin tornado version to 5.1.1
• [HOPSWORKS-978] - dela::install should create /srv/hops/ before creating sub directories
• [HOPSWORKS-979] - Cannot change default 'admin' password of [email protected]
• [HOPSWORKS-980] - HOPSWORKS-950 breaks hops-util-py in python 2.7 environments
• [HOPSWORKS-983] - [ePipe] Wrong type for long values in query condition handling
• [HOPSWORKS-987] - User admin to slow and user search returning duplicated results
• [HOPSWORKS-988] - Launching Jupyter server gradually gets slower and times out
• [HOPSWORKS-990] - uploading big files from web ui can cause 'invalidated token'
• [HOPSWORKS-992] - [kagent-chef] Invalid command in kagent's service status script
• [HOPSWORKS-993] - [Feturestore] tf record schema infer for float arrays is not working correctly
• [HOPSWORKS-994] - TensorBoard in experiments service fails to start if tensorflow-gpu is in the project environment
• [HOPSWORKS-996] - Jobs monitor threads may saturate the system
• [HOPSWORKS-1000] - Parameter servers in ParameterServerStrategy do not return Spark task when all workers are finished
• [HOPSWORKS-1003] - Jupyter convert notebook should inject SparkSession and handle spaces in notebook name
• [HOPSWORKS-1004] - YarnJobMonitor does not have a full list of Yarn running states
• [HOPSWORKS-1007] - Install version specific kernel packages, not generic ones
• [HOPSWORKS-1009] - Non-websocket requests in Jupyter notebook are blocked due to cross-origin policy
• [HOPSWORKS-1011] - hops-metadata-dal-impl-ndb schema update renames hdfs_inode_attributes to hdfs_directory_with_quota_feature
• [HOPSWORKS-1012] - Metadata designer does not work
• [HOPSWORKS-1013] - Descrease websocket_ping_interval to avoid proxied websocket connections being dropped
• [HOPSWORKS-1014] - ui bugs
• [HOPSWORKS-1017] - Date in field "feature store created" is NaN
• [HOPSWORKS-1022] - NOT_START is not a final state for Log Aggregation
• [HOPSWORKS-1030] - [hops-util-py] copy_to_local bugfixes and removal of localize
• [

0.9.1

    Release notes - Hopsworks - Version 0.9.1

Bug

• [HOPSWORKS-197] - Turning on and off tours is a terrible user experience
• [HOPSWORKS-903] - Jupyter cleanup and timer bugfixes
• [HOPSWORKS-913] - Feature store quota missing from admin UI in hopsworks
• [HOPSWORKS-920] - Job monitor thread gets stuck if job is killed while running
• [HOPSWORKS-927] - TensorBoard could not bind to an unsupported address family
• [HOPSWORKS-928] - [Featurestore] Bug: nested spark schemas are not automatically translatable to Hive
• [HOPSWORKS-929] - [Featurestore] Don't allow hyphen in featuregroup names because Hive do not allow it
• [HOPSWORKS-930] - [Featurestore] bug, in API retrieval code in feature-registry featurestore is undefined
• [HOPSWORKS-932] - Jobs UI date selector should persist user-selected values
• [HOPSWORKS-938] - Adding pyFiles to Jupyter does not add it to python path
• [HOPSWORKS-941] - Fix dataset access request and related tests
• [HOPSWORKS-945] - Zipping and unzipping is broken with Hops TLS
• [HOPSWORKS-949] - [airflow-chef] Install recipe does not create root hops installation directory
• [HOPSWORKS-951] - sqoop and airflow bugfixes
• [HOPSWORKS-953] - [kagent-chef] kagent falsely assumes service is dead when it restarts
• [HOPSWORKS-955] - [airflow-chef] Use assigned roles in JWT to determine if a user is admin
• [HOPSWORKS-958] - AirflowJWTManager randomly does not renew tokens
• [HOPSWORKS-968] - Dataset browser pagination and select all not working
• [HOPSWORKS-974] - Blank padded day in rspec breaks tests
• [HOPSWORKS-975] - [airflow-chef] Change restart policy of airflow scheduler
• [HOPSWORKS-976] - Airflow file manager does not refresh secret directory when navigate to another project
• [HOPSWORKS-977] - Pin tornado version to 5.1.1
• [HOPSWORKS-978] - dela::install should create /srv/hops/ before creating sub directories
• [HOPSWORKS-979] - Cannot change default 'admin' password of [email protected]
• [HOPSWORKS-987] - User admin to slow and user search returning duplicated results
• [HOPSWORKS-988] - Launching Jupyter server gradually gets slower and times out
• [HOPSWORKS-990] - uploading big files from web ui can cause 'invalidated token'
• [HOPSWORKS-992] - [kagent-chef] Invalid command in kagent's service status script
• [HOPSWORKS-993] - [Feturestore] tf record schema infer for float arrays is not working correctly
• [HOPSWORKS-996] - Jobs monitor threads may saturate the system
• [HOPSWORKS-1004] - YarnJobMonitor does not have a full list of Yarn running states
• [HOPSWORKS-1007] - Install version specific kernel packages, not generic ones
• [HOPSWORKS-1012] - Metadata designer does not work
• [HOPSWORKS-1017] - Date in field "feature store created" is NaN
• [HOPSWORKS-1022] - NOT_START is not a final state for Log Aggregation

Task

• [HOPSWORKS-959] - Remove apache2 dependency from cookbooks
• [HOPSWORKS-984] - Remove commons-fileupload dependency from kmon and admin modules