Publisher cannot get ssl cert from workload api, HTTPS API returning SSL_ERROR_SYSCALL error

[emulic]

Describe the bug
When calling publisher HTTP API via HTTPS like i.e.
curl https://****:9705/swagger/v2/openapi.json
or
curl -X POST https://*****:9705/v2/configuration/diagnostics
following error occurs:
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to *****:9705

Following is logged in the log of the publisher module:
[16:04:17.2977] dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[39] Connection id "0HMV0DH6G5AMO" accepted. [16:04:17.2978] dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[1] Connection id "0HMV0DH6G5AMO" started. [16:04:17.3094] dbug: Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware[1] => ConnectionId:0HMV0DH6G5AMO Failed to authenticate HTTPS connection. System.Security.Authentication.AuthenticationException: The server mode SSL must use a certificate with the associated private key. at System.Net.Security.SslStream.AcquireServerCredentials(Byte[]& thumbPrint) at System.Net.Security.SslStream.GenerateToken(ReadOnlySpan1 inputBuffer, Byte[]& output)
at System.Net.Security.SslStream.NextMessage(ReadOnlySpan1 incomingBuffer) at System.Net.Security.SslStream.ProcessBlob(Int32 frameSize) at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](CancellationToken cancellationToken) at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken) at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware.OnConnectionAsync(ConnectionContext context) [16:04:17.3106] dbug: Microsoft.AspNetCore.Server.Kestrel.Connections[2] Connection id "0HMV0DH6G5AMO" stopped. [16:04:17.3109] dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[7] Connection id "0HMV0DH6G5AMO" sending FIN because: "The Socket transport's send loop completed gracefully."

Calling the API methods via HTTP is working as expected.

I followed descriptions #calling-the-api-over-http and #built-in-http-api-server but used different host ports then described there.

To Reproduce
Steps to reproduce the behavior:

1. Configure publisher to map and expose the ports 80 and 443 as following:
   , "PortBindings": { "80/tcp": [ { "HostPort": "9704" } ], "443/tcp": [ { "HostPort": "9705" } ] } }, "ExposedPorts": { "443/tcp": {}, "80/tcp": {} }
2. From the EdgeDevice call some of the API methods via HTTPS i.e. (replace **** with your host name):
   curl https://****:9705/swagger/v2/openapi.json
3. Take note about error occurred
   curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to *****:9705

Expected behavior
The called API method should return a response just like calling via HTTP is doing.

Desktop (please complete the following information):

• OS: Ubuntu 20.4
• Publisher v 2.9.2.1

Additional context
Add any other context about the problem here.

[marcschier]

The issue happens only when running in Iot edge. The server certificate produced by the workload API is wrong somehow (looks like the private key is missing). I have added code to check that a private key exists in it, and if not fall back to generating a self signed certificate (like when running outside iot edge). Could you test the preview build published under tag '2.9.3-preview3' and if you still see issues, send me the log of OPC Publisher starting?

[emulic]

I'll need some time to get back to this topic and find time for testing but I'll do that.
Can we expect in some future version to get this solved in the way as it was originally intended, without using self signed certificates and using existing certificate chain rolled out by the customer PKI infrastructure instead?

[marcschier]

We have a feature (#2081) that I have opened that is tracking this for the application certificate. We could expand this to also provision the ssl certificate. I cannot talk to timing though.

[emulic]

Actually this is not what I asked. The original idea to reuse already enrolled certificate of the host is actually perfect.
Whatever certificate is there, enrolled by using EST, SCEP, manually or however, can simply be used. No need to depend on some specific certificate enrollment solution or product. I mean, a certificate must be there to allow EdgeDevice to IoT Hub communication, device to device communication in a nested hierarchy or communication to OPC UA endpoints. Why to not reuse that certificate by the REST API of the publisher? This issue with the missing private key shouldn't be the reason to drift away from that great idea. Finally, edgeHub is using the same certificate in the similar way if I properly understand the documentation. If it works in edgeHub, why shouldn't it work in the publisher module?

[marcschier]

enrolled

Agreed. I re-open to track this down with IoT Edge team.

[marcschier]

Issue lies in an incompatibility with .net.

Not supported: The 'unix' scheme is not supported.. Unable to use workload API to obtain

[marcschier]

Fix is in main and will be in 2.9.4-preview1