AWS OpenSearch 2.3 authentication issue with Graylog 5.0.2

[lkwjohn]

Having authentication issue with AWS OpenSearch authentication with Graylog 5.0.2 within ECS Fargate.
Previous setup without authentication works fine.

Expected Behavior

1. In the config file, add elasticsearch_hosts = https://username:[email protected]:443
2. Start Graylog and it should be able to authenticate with AWS OpenSearch using the basic auth

Current Behavior

1. In the config file, add elasticsearch_hosts = https://username:[email protected]:443 where the password container Upper case, Lower case, Numeric and special char of either a _ - or !
2. Start Graylog
3. Unauthorized to connect to OpenSearch org.graylog2.storage.versionprobe.VersionProbe - Unable to retrieve version from Elasticsearch node vpc-parentsgateway-zxdhaqbggz44a52lzk26amm2wu.ap-southeast-1.es.amazonaws.com:-1: unknown error - an exception occurred while deserializing error response: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'Unauthorized': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false') at [Source: (okhttp3.ResponseBody$BomAwareReader); line: 1, column: 13] | at [Source: (okhttp3.ResponseBody$BomAwareReader); line: 1, column: 13] -- | --

Steps to Reproduce (for bugs)

1. Add authentication for the OpenSearch
2. Add elasticsearch_hosts = https://username:[email protected]:443 to graylog.config

Context

I done a few testing.

Curl Test within where Graylog is running

curl https://username:[email protected]:443 within where the Graylog is running, was able to connect to the OpenSearch with the response { "name" : "xxxxx", "cluster_name" : "xxx:x", "cluster_uuid" : "xxx", "version" : { "distribution" : "opensearch", "number" : "2.3.0", "build_type" : "tar", "build_hash" : "unknown", "build_date" : "2023-04-20T07:23:19.274646Z", "build_snapshot" : false, "lucene_version" : "9.3.0", "minimum_wire_compatibility_version" : "7.10.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "The OpenSearch Project: https://opensearch.org/" }

Test on VersionProbe.java snippet

I have tested the VersionProbe.java portion, and it works fine, suspecting the issue could lies where it took the elasticsearch_hosts url from the config (config parser) to the VersionProbe.java. Below is the snippet of what i tested of the VersionProbe.java.

public static void main(String[] args) throws MalformedURLException {
        System.out.println("Hello World2!");

        
        try {
            String url = new String("https://username:[email protected]:443");
            URI host = new URI(url);
        

            OkHttpClient okHttpClient = new OkHttpClient();
            
            ObjectMapper objectMapper = new ObjectMapper();

            Retrofit retrofit = new Retrofit.Builder()
                        .baseUrl(host.toURL())
                        .addConverterFactory(JacksonConverterFactory.create(objectMapper))
                        .client(addAuthenticationIfPresent(host, okHttpClient))
                        .build();
            final RootRoute root = retrofit.create(RootRoute.class);

            final Converter<ResponseBody, ErrorResponse> errorResponseConverter = retrofit.responseBodyConverter(ErrorResponse.class, new Annotation[0]);
            final Consumer<ResponseBody> errorLogger = (responseBody) -> {
                try {
                    final ErrorResponse errorResponse = errorResponseConverter.convert(responseBody);
                    System.out.println("Unable to retrieve version from Elasticsearch node");
                    // LOG.error("Unable to retrieve version from Elasticsearch node {}:{}: {}", host.getHost(), host.getPort(), errorResponse);
                } catch (IOException e) {
                    System.out.println("Unable to retrieve version from Elasticsearch node 2");
                    System.out.println(e);
                    // LOG.error("Unable to retrieve version from Elasticsearch node {}:{}: unknown error - an exception occurred while deserializing error response: {}", host.getHost(), host.getPort(), e);
                }
            };

            System.out.println(rootResponse(root, errorLogger)
                .map(RootResponse::version));
        } catch (URISyntaxException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
    }

The response was Optional[VersionResponse{number=2.3.0, distribution=opensearch}]

• Graylog Version: 5.0.2
• Java Version: openjdk-17.0.5.0.8-2
• Elasticsearch Version: 2.3
• MongoDB Version: 6.0.4
• Operating System: Redjat ubi:9.1.0-1817
• Browser version: NA

[janheise]

Intermediate result: I checked against the included OpenSearch 2.5 in our DataNode and a user who's password uses the mentioned special chars and had no problems.

[lkwjohn]

May I check the Graylog version you used was 5.0.2? Are there anyway you can recommend me to debug this on my side?

[janheise]

This was using the current dev version I have on my machine. But the code involved should not have any changes since 5.0 (at least after a first glance). But I'll check with a 5.0 version next. The first test was just to make sure that I enabled the security plugin correctly. Can you give me some more info about your setup? OpenSearch is your own installation? Or the AWS service? Also, it might be helpful, if you could share the startup logs up to the point where the error occurs - this could help track it down, too.

[lkwjohn]

The OpenSearch 2.3 I used was a AWS service, I have enabled the fine grained access control (fgac) and created a master user, the policy of the fgac, I allowed all methods. Https for the OpenSerach was enabled.

[janheise]

@lkwjohn thx for the info.
This is probably not the solution that you're looking for: I asked my colleagues that work with AWS instances and they disallow those chars in passwords because of a similar issue. Can you change the password?
I am not closing this issue but it will take more time to deep dive into it and see why it's happening.

[janheise]

Also, for clarification because I might have misunderstood your second sentence from the top: you were using 5.0.2 without auth before enabling it (and refer to this as "worked before") or did you upgrade from an older graylog version (which worked) to 5.0.2?

[lkwjohn]

I was using 5.0.2 graylog with AWS openseach 2.3 without auth. After that, I did not upgrade the graylog when I enable the AWS opensearch from non auth to auth.

As for the char to avoid in the password, are there others char I should avoid using?

[janheise]

My colleagues use the following in their exclusion policy: '"@?^&*()~?;:[]{}.,\/|<>%#!$_

[williamtrelawny]

it seems spaces are excluded characters too. It'd be really nice to be able to use more than alphanumerals in our Opensearch user passwords, for all the common security reasons 👍

Double-quoting or single-quoting the elasticsearch_hosts value doesn't seem to work either. E.g. elasticsearch_hosts = 'https://user:pw with [email protected]:9200'

[janheise]

@williamtrelawny It's not that easy (actually, it is) - because in that area of an URL/URI per HTTP-Specification only a few special characters are allowed.

[janheise]

I'm closing this - the issue has not come up for others in the meantime and I was not able to reproduce it.
The solution is just to make sure that you use only the supported special characters (see above).