Hi Chad,

While the configuration UI has them next to each other, there are two entirely separate communications here.

The Directory Server STS URL is used for the sole purpose of specifying where to send end users authentication redirects. Specifically, it sets the issuer value in the wsFederation node of .\Forms\Web.config:

<wsFederation persistentCookiesOnPassiveRedirects="true" passiveRedirectEnabled="false" realm="https:/forms.example.com/Forms/" reply="https:/forms.example.com/Forms/" issuer="https://lfds.example.com/LFDSSTS/" homeRealm="urn:laserfiche:lfdsdb:LFDS" requireHttps="true" />

You can actually repurpose that value to effectively disable logins for a Forms Portal as I describe here. The important part is that it's purely about End Users->LFDSSTS and doesn't affect the backend Forms->LFDS communication in any way. The latter is what your error message is about.

For LFDS and Forms 10.4.2 and earlier

The Forms->LFDS backend communication uses WCF over HTTP messages sent to LFDS's 5048/5049 service ports. WCF can work with multiple transport protocols, and uses something called Message Security which is transport-agnostic. To quote Microsoft: "Message security uses the WS-Security specification to secure messages. The WS-Security specification describes enhancements to SOAP messaging to ensure confidentiality, integrity, and authentication at the SOAP message level (instead of the transport level [like HTTPS])." None of these calls happen through a browser so mixed-content doesn't come into play at all.

Within an Active Directory domain, Message Security essentially uses Windows Authentication to sign and validate messages. When one of the communicating parties is outside the domain (e.g. Forms DMZ), this AD authentication mechanism no longer works. 

What to do then? Use Certificate signing/authentication for the messages instead. This is exactly what the LFDS "Alternate Binding" is for. It provides an LFDS endpoint configured for certificate-based message security instead of AD-based security.

Configure LFDS to enable the Alternative Service using the XmlEndpointUtility, then configure Forms to use the Alternate Service with its own EndpointUtility (Under .\Forms\Forms\bin).

This changes the Forms Web.config endpoints to:

    <!-- WCF client configuration -->
    <client>
      <endpoint address="net.tcp://localhost:8168/lfrouting" binding="netTcpBinding" bindingConfiguration="timeoutBinding" contract="Laserfiche.Forms.Routing.IRoutingEngineService" name="" />
      ...
      <endpoint address="net.tcp://localhost:8738/lflicensing" binding="netTcpBinding" bindingConfiguration="timeoutBinding" contract="FormsModel.SharedContracts.ILicensingService" name="" />
      <endpoint address="http://lfds.example.com:5048/LicenseManager/service2" binding="ws2007HttpBinding" bindingConfiguration="WS2007HttpBinding_ILicenseManager" contract="LicenseManagerService.ILicenseManager" name="AltLicenseManagerService" behaviorConfiguration="AltServiceBehavior" />
      <endpoint address="http://lfds.example.com:5048/LicenseManager/service2" binding="ws2007HttpBinding" bindingConfiguration="WS2007HttpBinding_ILicenseManager" contract="LicenseManagerService.ILicenseManager2" name="AltLicenseManagerService2" behaviorConfiguration="AltServiceBehavior" />
      <endpoint address="http://lfds.example.com:5048/LicenseManager/sts2" binding="ws2007HttpBinding" bindingConfiguration="WS2007HttpBinding_ILFSecurityTokenService" contract="LicenseManagerSTS.ILFSecurityTokenService" name="AltLicenseManagerSTS" behaviorConfiguration="AltServiceBehavior" />
    </client>
    <behaviors>
      <endpointBehaviors>
        <behavior name="AltServiceBehavior">
          <clientCredentials>
            <clientCertificate storeLocation="LocalMachine" x509FindType="FindByThumbprint" findValue="788A44ED79A66Q5CE1D276A113B86836752FDF44" />
          </clientCredentials>
        </behavior>
      </endpointBehaviors>
    </behaviors>
  </system.serviceModel>

For LFDS and Forms 10.4.3+

LFDS and STS 10.4.3 switched from WCF to HTTP(S). Note that not all other Laserfiche applications have yet, so the Alternate Service is still sometimes applicable. See HTTPS and WCF Configuration Information for Laserfiche Directory Server 10.4.3 and Certificate Requirements for Laserfiche Directory Server. You can still use the Alternate Service instead if desired.

I briefly tested the following configuration to have Forms use HTTPS for backend communication with LFDS and it seemed to work:

1. Ensure LFDS 10.4.3 is configured with an HTTPS binding on port 5049 through its XmlEndpointUtility, with the certificate requirements linked above met.
2. Open Forms Web.config and update the three LFDS LicenseManager endpoints to use HTTPS and port 5049 like so:

   <endpoint address="https://lfds.example.com:5049/LicenseManager/service" binding="ws2007HttpBinding" bindingConfiguration="WS2007HttpBinding_ILicenseManager" contract="LicenseManagerService.ILicenseManager" name="LicenseManagerService" />
   <endpoint address="https://lfds.example.com:5049/LicenseManager/service" binding="ws2007HttpBinding" bindingConfiguration="WS2007HttpBinding_ILicenseManager" contract="LicenseManagerService.ILicenseManager2" name="LicenseManagerService2" />
   <endpoint address="https://lfds.example.com:5049/LicenseManager/sts" binding="ws2007HttpBinding" bindingConfiguration="WS2007HttpBinding_ILFSecurityTokenService" contract="LicenseManagerSTS.ILFSecurityTokenService" name="LicenseManagerSTS" />
   

3. Make the exact same changes to the .\Forms\bin\RoutingEngineServiceHost.exe.config file.
    
4. Restart the Forms Routing Service and see if FormsConfig shows a successful connection to LFDS.