[BUG] Registering MinIO (S3) snapshot repository fails with "Connect timed out"

[pjuri]

Describe the bug

I’m running OpenSearch as part of Graylog Helm installation under Kubernetes. I’m trying to register a snapshot endpoint with MinIO. I’m following this document: https://opensearch.org/docs/latest/tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore/

When I try to register the repository with curl (using the REST API), I get "Connect timed out" error. Using tcpdump I can see that no connection to provided IP address is attempted. When I manually test the connection to MinIO with curl, it works. (I.e. it’s not a network issue.)

If I remove s3.client.default.endpoint setting, I can see OpenSearch connecting to Amazon servers. (Which is not what I want.)

I suspect this might be just a misconfiguration, but no matter what I try, I get the same results.

Related component

Plugins

To Reproduce

[opensearch@opensearch-cluster-master-0 ~]$ opensearch-keystore create
An opensearch keystore already exists. Overwrite? [y/N]y
Created opensearch keystore in /usr/share/opensearch/config/opensearch.keystore
[opensearch@opensearch-cluster-master-0 ~]$ opensearch-keystore add s3.client.default.access_key
Enter value for s3.client.default.access_key:
[opensearch@opensearch-cluster-master-0 ~]$ opensearch-keystore add s3.client.default.secret_key
Enter value for s3.client.default.secret_key:
[opensearch@opensearch-cluster-master-0 ~]$ grep s3.client.default config/opensearch.yml
s3.client.default.protocol: "http"
s3.client.default.endpoint: "http://1.2.3.4:9000/"
s3.client.default.path_style_access: "true"

Did steps above on all 3 cluster members.

[opensearch@opensearch-cluster-master-0 ~]$ curl -X POST "http://localhost:9200/_nodes/reload_secure_settings"
{"_nodes":{"total":3,"successful":3,"failed":0},"cluster_name":"opensearch-cluster","nodes":{"Ug2a4ZiqS_6sNDvKlFRNbg":{"name":"opensearch-cluster-master-2"},"zi7xQcAsT0WyPEXLozMEJQ":{"name":"opensearch-cluster-master-0"},"R6I3MgjqRrS85OjyIWHCaw":{"name":"opensearch-cluster-master-1"}}}[opensearch@opensearch-cluster-master-0 ~]$
[opensearch@opensearch-cluster-master-0 ~]$ curl -X PUT "http://localhost:9200/_snapshot/minio-repo?pretty" -H 'Content-Type: application/json' -d '

{
"type": "s3",
"settings": {
"bucket": "opensearch",
"base_path": "opensearch/snapshot/"

}
}'
{
"error" : {
"root_cause" : [
{
"type" : "repository_verification_exception",
"reason" : "[minio-repo] path [opensearch/snapshot/] is not accessible on cluster-manager node"
}
],
"type" : "repository_verification_exception",
"reason" : "[minio-repo] path [opensearch/snapshot/] is not accessible on cluster-manager node",
"caused_by" : {
"type" : "i_o_exception",
"reason" : "Unable to upload object [opensearch/snapshot//tests-nZNGJ5szRh-Pd5gX3q44dA/master.dat] using a single upload",
"caused_by" : {
"type" : "sdk_client_exception",
"reason" : "sdk_client_exception: Failed to connect to service endpoint: ",
"caused_by" : {
"type" : "i_o_exception",
"reason" : "Connect timed out"
}
}
}
},
"status" : 500
}

tcpdump shows no traffic to MinIO

Test if the Minio endpoint is reachable:

[opensearch@opensearch-cluster-master-0 ~]$ curl http://1.2.3.4:9000/

AccessDeniedAccess Denied./minio17FE44A7FEAD5E72dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8[opensearch@opensearch-cluster-master-0 ~]$

tcpdump shows connection with MinIO was established

Expected behavior

Snapshot endpoint should be successfully registered, allowing me to make snapshots and recoveries.

Additional Details

Plugins
plugins:
enabled: true
installList:
- repository-s3

Host/Environment (please complete the following information):

• OS: Ubuntu Server
• Version: 22.04, kernel 5.15.0-102-generic

Additional context
Kubernetes: v1.28.14
Containerd: 1.7.2-0ubuntu1~22.04.1
Docker image: opensearchproject/opensearch:2.4.0
Helm chart: graylog-2.3.10 - uses https://artifacthub.io/packages/helm/opensearch-project-helm-charts/opensearch

[pjuri]

Here is the list of installed plugins from _cat/plugins: (Only from first node)
opensearch-cluster-master-0 opensearch-alerting 2.4.0.0
opensearch-cluster-master-0 opensearch-anomaly-detection 2.4.0.0
opensearch-cluster-master-0 opensearch-asynchronous-search 2.4.0.0
opensearch-cluster-master-0 opensearch-cross-cluster-replication 2.4.0.0
opensearch-cluster-master-0 opensearch-geospatial 2.4.0.0
opensearch-cluster-master-0 opensearch-index-management 2.4.0.0
opensearch-cluster-master-0 opensearch-job-scheduler 2.4.0.0
opensearch-cluster-master-0 opensearch-knn 2.4.0.0
opensearch-cluster-master-0 opensearch-ml 2.4.0.0
opensearch-cluster-master-0 opensearch-neural-search 2.4.0.0
opensearch-cluster-master-0 opensearch-notifications 2.4.0.0
opensearch-cluster-master-0 opensearch-notifications-core 2.4.0.0
opensearch-cluster-master-0 opensearch-observability 2.4.0.0
opensearch-cluster-master-0 opensearch-performance-analyzer 2.4.0.0
opensearch-cluster-master-0 opensearch-reports-scheduler 2.4.0.0
opensearch-cluster-master-0 opensearch-security 2.4.0.0
opensearch-cluster-master-0 opensearch-security-analytics 2.4.0.0
opensearch-cluster-master-0 opensearch-sql 2.4.0.0
opensearch-cluster-master-0 repository-s3 2.4.0