You are viewing limited content. For full access, please sign in.

Question

Question

Self-Hosted LF API Server fails when connecting to repository over TLS

asked on January 6, 2023

We're testing the self-hosted LF API Server, and it works fine when the connection to the Repository Server does NOT use SSL.  However when SSL is enabled for the connection (from API to LFS), we receive the following response: 

{
  "type": "accessDenied",
  "title": "Error: No Laserfiche Repository connection could be made.",
  "status": 401,
  "instance": "/v1/Repositories/MCCi-Test/Token",
  "operationId": "95a8eb5e57194b8b9babcaedcf2c01a8",
  "errorSource": "Api Server",
  "errorCode": 201,
  "traceId": "00-75f5b72d1734abb38240b1ea6f1f6a96-b05954a36463887e-00"
}


Digging into the LFRepositoryAPI operation log, we see it fails due to certificate validation: 

[95a8eb5e57194b8b9babcaedcf2c01a8]:LFRepositoryAPI started at 2023-01-06T15:40:46.3047858Z.
[INFO+0ms(3)]Correlated operation: 80000058-0001-f800-b63f-84710c7967bb
[INFO+0ms(3)][POST] https://mccitestpro.mccicloud.io/lfrepositoryapi/v1/Repositories/MCCi-Test/Token
[INFO+0ms(3)]Fail to get Authorization header from context
[INFO+0ms(3)]Fail to get Authorization header from context
[INFO+152ms(3)]Failed to log in with the provided credentials.
[ERR+175ms(3)]One or more errors occurred. (One or more errors occurred. (The SSL connection could not be established, see inner exception.))
THE SSL CONNECTION COULD NOT BE ESTABLISHED, SEE INNER EXCEPTION.
        THE REMOTE CERTIFICATE WAS REJECTED BY THE PROVIDED REMOTECERTIFICATEVALIDATIONCALLBACK.
    -----TRACES----
    System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
     ---> System.Security.Authentication.AuthenticationException: The remote certificate was rejected by the provided RemoteCertificateValidationCallback.
       at System.Net.Security.SslStream.SendAuthResetSignal(ProtocolToken message, ExceptionDispatchInfo exception)
       at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptions sslAuthenticationOptions)
       at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
       at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
       --- End of inner exception stack trace ---
       at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
       at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
       at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
       at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request)
       at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
       at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
       at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
       at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
       at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
       at Laserfiche.HttpClient.TimeoutHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
       at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
    ------
[INFO+178ms(3)]TokenController GetAccessToken failed for MCCi-Test
[ERR+195ms(3)]Error: No Laserfiche Repository connection could be made.
ERROR: NO LASERFICHE REPOSITORY CONNECTION COULD BE MADE.
    -----TRACES----
    Laserfiche.Repository.LFSConnector.Exceptions.AccessDeniedException: Error: No Laserfiche Repository connection could be made.
       at Laserfiche.Repository.LFSConnector.Connectors.RepositoryConnector.CreateConnection_V1(String repositoryId, CreateConnectionRequest request, Boolean createCookie)
       at Laserfiche.Sites.SiteApiRepository.V1.Controllers.TokenController.<>c__DisplayClass3_0.<<GetAccessToken>b__0>d.MoveNext() in E:\A\1261\s\src\V1\Controllers\TokenControllerV1.cs:line 85
    --- End of stack trace from previous location ---
       at Laserfiche.Sites.SiteApiRepository.Polly.PollyExecutor.<>c__DisplayClass2_0`1.<<ExecuteAsync>b__0>d.MoveNext() in E:\A\1261\s\src\Polly\PollyExecutor.cs:line 31
    --- End of stack trace from previous location ---
       at Polly.CircuitBreaker.AsyncCircuitBreakerPolicy.<>c__DisplayClass8_0`1.<<ImplementationAsync>b__0>d.MoveNext()
    --- End of stack trace from previous location ---
       at Polly.CircuitBreaker.AsyncCircuitBreakerEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext, ExceptionPredicates shouldHandleExceptionPredicates, ResultPredicates`1 shouldHandleResultPredicates, ICircuitController`1 breakerController)
       at Polly.CircuitBreaker.AsyncCircuitBreakerPolicy.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
       at Polly.AsyncPolicy.ExecuteAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
       at Laserfiche.Sites.SiteApiRepository.Polly.PollyExecutor.ExecuteAsync[T](Func`1 action, ISparkContext sparkContext) in E:\A\1261\s\src\Polly\PollyExecutor.cs:line 28
       at Laserfiche.Sites.SiteApiRepository.Controllers.ODataBaseController.<>c__DisplayClass5_0`1.<<TryCatchAsync>b__0>d.MoveNext() in E:\A\1261\s\src\Controllers\ODataBaseController.cs:line 51
    --- End of stack trace from previous location ---
       at Laserfiche.Sites.SiteApiRepository.Util.ControllerUtil.TryCatchAsync[T](ControllerBase controller, ISparkContext sparkContext, Func`1 p, Action`1 resultLogAction, Func`2 customResponseHandling) in E:\A\1261\s\src\Util\ControllerUtil.cs:line 61
    ------
    -----DATA-----
        OperationId = 95a8eb5e57194b8b9babcaedcf2c01a8
[INFO+213ms(3)]HTTP Reponse: 401.
[INFO+213ms(3)]Completed at 2023-01-06T15:40:46.5182802Z MilliSpeedOp.
[LFRepositoryAPI]Error: No Laserfiche Repository connection could be made.[95a8eb5e57194b8b9babcaedcf2c01a8][2!0](213ms)>[Warning][-1][80000058-0001-f800-b63f-84710c7967bb][system]

 

We're enabling/disabling the use of SSL/TLS to the repo server by adjusting the following parameter in the appsettings.json:

  •     "EnableLaserficheServerSSL":  true,

 

The environment: 

  • WebServer 10.20.11.4 (mcmccitestLF01.mccicloud.io): Web Client, Weblink, LF API Server
  • AppServer 10.20.11.36 (mcmccitestLF02.mccicloud.io): LF Server (with repository “mcci-test”)

 

For this connection, the SSL client would be 10.20.11.4 and the SSL server would be: 10.20.11.36. The Certificate Validation would be performed by the LFAPI on 10.20.11.4. 

Interestingly enough, we also have Web Client and Weblink on the WebServer, both of those configured to connect using SSL/TLS to the same repository on the AppServer. Both connect successfully without any certificate validation errors. Additionally, connecting from browser on 10.20.11.4 to 10.20.11.36 using HTTPS, the browser indicates the same certificate is passing validation. 

We've also manually validated the certificate via the standard validation criteria: 

  1. hostname used for connection (mcmccitestLF02.mccicloud.io) matches the common name on the certificate (*.mccicloud.io). 
  2. current date is within the certificates valid from and valid to dates
  3. sha2 signed
  4. the SSL client (10.20.11.4) can reach the CRL list site and successfully retrieve the CRL. 
  5. the certificate is not on the CRL list. 
  6. the certificate chain of trust is intact and the root certificate is in the Trusted Root Certificates cert store on the the SSL client system (10.20.11.4) (this is a godaddy issued certificate). intermediate certificate is available on the SSL server system. 

 

Monitoring the communication with wireshark provides no additional insight... the SSL handshake proceeds normally, and then upon completion the SSL Client (10.20.11.4) sends a communication to terminate the session. This is seen as the Encrypted Alert packet in the capture. Presumably, it does this because at that stage, while the SSL session has been negotiated, this is where the SSL client (LFAPI) has performed the cert validation, and has deemed it "not valid", so the session is terminated.  Below is that capture of the LFAPI connecting to the LFS Repo. 

And the following is the Web Client on the same WebServer connecting to the same LFS Repo. Notice, it completes exactly the same as above, except there's no "encrypted alert" and the conversion progresses successfully beyond the end of the handshake. 

The only thing we found via wireshark is when the LFAPI does the CLIENT HELLO portion of the SSL handshake, it erroneously puts the REPO name in the SSI value. 

Here is the CLIENT HELLO (as captured by wireshark) with a correct value when the connection is initiated by the Web Client: 

 

And here is the CLIENT HELLO (as captured by wireshark) with the incorrect value when the connection is initiated by the LFAPI: 

We're sending this in to Support today as a bug.  

Based on our particular bindings, this doesn't create a problem with making the connection, IIS serving the correct site and certificate (as their is only 1 site, and 1 https binding and it has no SNI enabled).  However, it may still point to the underlying issue... Either:

a) the LF API is using a different certificate validation process that's finding some validation problem that Web Client and Weblink's connections do not find. 

OR

b) this incorrect hostname value showing in the SNI parameter is indicative of the wrong hostname ALSO being used in the name match evaluation of the certificate validation process. 

I'd be surprised no one else ran into this, unless everyone isn't using SSL on the connection from the API to LFS.  Has anyone successfully exercised the self-hosted API with the "EnableLaserficheServerSSL" setting to true?

Based on the swagger page, we're Build# : 1.0.0.41. 

 

2 0

Answer

APPROVED ANSWER
replied on February 27, 2023

This issue should be resolved in version 1.1 of the Self-Hosted Laserfiche API Server. The download link to install version 1.1 can be found here, https://developer.laserfiche.com/api/server/index.html, under the "Installation steps" section. 

 

@Dustin Foster this is the same version (1.1.0.3) sent through the support ticket.

2 0

Replies

replied on January 6, 2023

It doesn't look like your certificate has a subject alternative names extension. 
Self-Hosted LF API Server (.NET Core) hostname validator only matches wildcards within a DNS name label of the Subject Alternative Names extension. RFC 4513 provides details about this pattern.

0 0
replied on January 7, 2023 Show version history

The certificate does have valid "DNS Name" labels within the Subject Alternative Name of the certificate:

0 0
replied on January 9, 2023

Subject Alternative Name needs to include the FQDN of your Laserfiche server (mcmccitestLF02.mccicloud.io)

0 0
replied on January 9, 2023

I discussed with Paolo and we're going to try to reproduce this internally. We inspected the code and the issue potentially stems from a custom Host header that the SNI Server Name is using and/or a fallback certificate verification path that may not properly handle wildcard SAN DNS Name values.

1 0
replied on January 10, 2023

@████████ thanks. I was troubled by Paulo's response because that's the entire purpose of a wildcard certificate is to secure multiple hostnames within a given domain namespace without explicitly specifying the FQDNs. 

0 0
replied on January 10, 2023 Show version history

I did switch to using a self-signed SSL Certificate, as opposed to the wildcard, to test, and the same issue occurred. I verified that the self-signed certificate had the FQDN value within the SAN extension, and the trust WAS setup for the self-signed certificate. After setting up the trust, I verified that I achieved a successful connection via Web Client as well as a browser to the IIS Splash Page (presenting the self-signed certificate).

Whether or not the certificate in-play has the proper SAN configured, the issue seems to stem from the LF Repository API sending the wrong value in the SNI header.

0 0
replied on January 10, 2023 Show version history

@████████- Paolo brought that up because there is a real outstanding bug where the API Server has a custom certificate name validation path that does not currently include wildcard handling. However, that appears to be a fallback code path that only gets triggered if there's an error from the normal/native .NET 6 HttpClient check. If you're running into this, it's unlikely to be the primary problem (which I think is the SNI one).

@████████ - Could you please try generating another cert with SAN values of both the server FQDN and "Mcci-Test"? I believe that would allow it to pass SNI validation and thus test the theory.

If you have IIS installed on the server with LFS, please check the 443 binding to see if "Require SNI" is enabled. It's unlikely that setting ultimately matters here but as one of the few explicit SNI-related configs it's worth checking.

 

0 0
replied on January 10, 2023

@████████ - I can confirm that "Require SNI" is NOT enabled for the IIS Binding; it was not enabled with the wildcard, nor the self-signed binding ​(the wildcard binding was replaced with the self-signed binding for testing; it wasn't an additional binding).

I did try to enable "Require SNI" and specify "MCCi-Test" as the host name, but that was unsuccessful. 

I will see if I can get a certificate generated with "MCCi-Test" included in the SAN, and test that way. I'll update here with the results!

 

0 0
replied on January 10, 2023

Thanks! Could you also check the value of the Host header in the HTTP call API Server is making? It seems as of .NET Core 2.1 that "By default, the SNI will be the DNS name of the Uri used in the request. If a custom, "Host:" request header is added, then the SNI will use that name in the TLS/SSL handshake."

I couldn't find anything obvious in the code that's setting a custom SNI Server Name value so I suspect it's inheriting from the Host header, which there is code to set.

0 0
replied on January 10, 2023

@████████ - I generated a self-signed certificate with the custom SAN, which included the server FQDN and "MCCi-Test", setup the trust (placed a copy in the Web Server's Trusted Root), then switched the App Server bindings over to using this new "Custom SAN Certificate".

I can confirm that with this new certificate, having "MCCi-Test" in the SAN extension, I AM able to achieve a successful connection to the repository!!

I do not see a "host" value specified in the header, but I will upload a HAR Trace to the LF Support Case I have open, for your review.

 

With this, we've confirmed that having the DNS Name that it wants in the certificate SAN results in success, but now to determine where/why it's specifying the incorrect SNI/DNS for the host.

2 0
APPROVED ANSWER
replied on February 27, 2023

This issue should be resolved in version 1.1 of the Self-Hosted Laserfiche API Server. The download link to install version 1.1 can be found here, https://developer.laserfiche.com/api/server/index.html, under the "Installation steps" section. 

 

@Dustin Foster this is the same version (1.1.0.3) sent through the support ticket.

2 0
replied on February 27, 2023

@████████ - I can confirm that the issue of incorrect SNI is resolved in this new version, and I'm able to make successful SSL/TLS connection to the repository with my wildcard certificate. Thanks for the assist!!

3 0
You are not allowed to follow up in this post.

Sign in to reply to this post.