Locks under high load using RDF4JProtocolSession

[domkun]

Hi there,

I load tested an application that uses GraphDB as triple store. The application creates many transactions, mostly small and fast. After a short time of running the load test tha application locks up. Analysis showed, that the threads are waiting for a connection out of the pool, see screenshot.

It seems that even the Ping Scheduler is blocked.

When the connection pool is configured to be bigger using -Dhttp.maxConnections=100 the load test ran successfully.

Also, if using a plain SPARQL connection to the GraphDB the load test is running fine with no locks. I tried to trace down the issue but could not find the problem. Maybe it is something in the SharedHttpClientSessionManager.ServiceUnavailableRetryHandler where the max connections are checked.

rdf4j/core/http/client/src/main/java/org/eclipse/rdf4j/http/client/SharedHttpClientSessionManager.java

Lines 132 to 141 in 2a998c3

	// worst case, the connection pool is filled to the max and all of them idled out on the server already
	// we then need to clean up the pool and finally retry with a fresh connection. Hence, we need at most
	// pooledConnections+1 retries.
	// the pool size setting used here is taken from `HttpClientBuilder` when `useSystemProperties()` is used
	int pooledConnections = Integer.parseInt(System.getProperty("http.maxConnections", "5"));
	if (executionCount > (pooledConnections + 1)) {
	return false;
	}
	HttpClientContext clientContext = HttpClientContext.adapt(context);

Did anyone ever experience something like this?

[hmottestad]

Could you provide a small maven project that I can use to test this myself?

[hmottestad]

[hmottestad]

I ran the JMeter test plan and I think I have reproduced your issue. Looks like everything is waiting on the httpClient.

[domkun]

This is what I also observed. I think it has something to do with many small transactions that get started very fast. I just played around with some transactions and the following code also lead to everything waiting on the httpClient. I still cannot grab the root of this.

    @Scheduled(
            every = "2s"
    )
    @Blocking
    public void scheduled() {
        for (int i = 0; i < 500; i++) {
            try (RepositoryConnection con = repo.getConnection()) {
                con.begin();
                RepositoryResult<Statement> statements = con.getStatements(null,
                        Values.iri("http://xmlns.com/foaf/0.1/firstName"), null
                );
                con.commit();
            }
        }
    }

[hmottestad]

Your tip on setting http.maxConnections seemed to help for me.

-Dhttp.maxConnections=1000

Makes me believe that it's just an issue with running out of connections. We should probably bump the minimum within RDF4J to something a bit more reasonable.

[domkun]

Indeed it is a resource problem.

I am just wondering if there should be a possibility to configure the timeouts of the underlying HTTP client. I did a quick test with the following code, setting the connection request timeout, the connect timeout and the socket timeout to 1 second. Then the test runs without locking. Of course there are failing transactions due to the timeouts that now happen but at least the application keeps being responsive.

HTTPRepository repository = new HTTPRepository(repositoryUrl.get());
CloseableHttpClient httpClient = HttpClientBuilder.create()
    .setConnectionManager(connectionManager)
    .evictExpiredConnections()
//  .setRetryHandler(retryHandlerStale)
//  .setServiceUnavailableRetryStrategy(serviceUnavailableRetryHandler)
    .useSystemProperties()
    .setDefaultRequestConfig(
        RequestConfig.custom()
            .setCookieSpec(CookieSpecs.STANDARD)
            .setConnectTimeout(1000)
            .setSocketTimeout(1000)
            .setConnectionRequestTimeout(1000)
            .build()
    )
    .build();
SharedHttpClientSessionManager sessionManager = new SharedHttpClientSessionManager();
sessionManager.setHttpClient(httpClient);
repository.setHttpClientSessionManager(sessionManager);

At the moment I am not able to set the RetryHandler and the ServiceUnavailableRetryStrategy from the outside as these are private inner classes, so I left them out.

Second test was adding and configuring the connection manager of the http client to be a PoolingHttpClientConnectionManager, as if I understand correctly the defaults of the HTTP client are max. 5 concurrent requests per route and max 25 concurrent requests in total. So running the test with the following code made it run without any locks or timeouts.

HTTPRepository repository = new HTTPRepository(repositoryUrl.get());
PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager();
connectionManager.setMaxTotal(200);
connectionManager.setDefaultMaxPerRoute(200);
CloseableHttpClient httpClient = HttpClientBuilder.create()
    .setConnectionManager(connectionManager)
    .evictExpiredConnections()
    .useSystemProperties()
    .setDefaultRequestConfig(
        RequestConfig.custom()
            .setCookieSpec(CookieSpecs.STANDARD)
            .setConnectTimeout(1000)
            .setSocketTimeout(1000)
            .setConnectionRequestTimeout(1000)
            .build()
    )
    .build();
SharedHttpClientSessionManager sessionManager = new SharedHttpClientSessionManager();
sessionManager.setHttpClient(httpClient);
repository.setHttpClientSessionManager(sessionManager);

[fkleedorfer]

If you are using the GrapDB community edition, there is a limit of 2 parallel transactions on the server. Might that be your bottleneck?

[domkun]

We also tested against the GraphDB Enterprise. In fact this test led to further investigations.

Also increasing the max http connections of the underlying Apache HttpClient solved the problem. And using the community edition of GraphDB but connecting through the plain SPARQL endpoints also worked without a problem.

As in the thread dump all threads have been blocked in the process of fetching a connection from the pool I thought I start a discussion about the behaviour. I will try to make a small reproducer on monday If I find some time.

[fkleedorfer]

Very interested in your http client tweak, looking forward. Am I assuming correctly that this tweak fixes the problem with GraphDB Community as well?

[domkun]

Yes, setting http.maxConnections system property did resolve the issue when using the Community Edition.