There are all sorts of DNS and privilege-escalation-to-DNS vulnerabilities

I understand the usefulness of an authority, but when I see people disabling encryption because they can't find an authority I wonder why that would ever be a good decision.

The odds of someone gaining DNS control on a server is much lower than an ARP attack from the network making encryption incredibly important for confidential data. It's like the lock breaks on your balcony door and while your waiting for it to get fixed you just say screw it I won't lock the front door either.

Here is the case number

Case# 240381

I said in the ticket here

We don't know why since forms.domain.com does resolve to the correct address and successfully connects and syncs to LFS.

I feel changing the alias we use to resolve a server's address feels so unrelated to the initial problem, which is a KERBEROS problem and to do with how users passwords are resolved. We even made sure the custom alias resolves to the correct address on both DCs just as a shot in the dark, even though nothing states this is a requirement. (this all makes more sense when you see the ticket)

There is one thing with hosting public forms on a second server that confuses me and you might know how this works.

I can not understand how the communication works when sharing a single routing service between 2 forms servers. You have to open like 10 ports just to do this and there is no TLS option here.

Say we have an LFS/Forms/Routing server and a Public Forms (Web App Only) server sharing the routing service (and sharing a DB): Does the public server ever even make a connection to the repository to begin with? Based on our Wireshark monitoring it does not appear that it does, it seems to send a single packet to the routing service just to trigger a lookup of what to do from the DB. If this is the case, that means there is no traffic containing any confidential data between servers and/or between networks (in the case of a DMZ).

In order:
Re:

I understand the usefulness of an authority, but when I see people disabling encryption because they can't find an authority I wonder why that would ever be a good decision.

As noted earlier, deal with this by addressing the inadequate PKI Certificate Authority infrastructure, not by disabling encryption (in scenarios where encryption is called for). 

making encryption incredibly important for confidential data. It's like the lock breaks on your balcony door and while your waiting for it to get fixed you just say screw it I won't lock the front door either.

Agreed that encryption is incredibly important for confidential data. The encryption and authentication (certificate validation) components of TLS, while technically separate operations, are complementary and together provide a security posture significantly greater than the sum of their parts.

Performing encryption without authentication is like locking the door with a lock that will accept any key. The security value of such a lock is extraordinarily diminished vs one that only opens to authorized keys.

I agree that when it comes to cybersecurity, something is usually better than nothing, but for this particular matter having a functioning internal CA is such baseline table stakes for an IT org with AD in 2024, even for a smaller one like this customer, that it should be addressed from that side. I'll again refer to @████████'s excellent video tutorial Installing and Configuring Active Directory Certificate Services (youtube.com) as well as this one from a Microsoft MVP Active Directory Certificate Services Install & Config in just 20mins (youtube.com). 

I feel changing the alias we use to resolve a server's address feels so unrelated to the initial problem, which is a KERBEROS problem and to do with how users passwords are resolved.

Ah, target server names are extremely relevant to Kerberos. Target names for servers or user accounts are absolutely core to how Kerberos functions. A DNS alias alone will break Kerberos because the required SPNs for the aliased name (among other things) will not be present by default. Without these, Kerberos fails because the target (recipient) cannot decrypt the Kerberos ticket (among other errors). Where Windows Authentication appears to work with a DNS alias (where additional SPNs have not been set), it's usually because the Negotiate protocol is falling back to NTLM after a Kerberos failure, and Microsoft recently deprecated NTLM.

Instead of a simple DNS alias, set an AD Computer Name Alias, which will automatically handle the Kerberos SPN configurations in addition to the DNS record. See: Using Computer Name Aliases in place of DNS CNAME Records - Microsoft Community Hub

That article also helps explain why names are important for Kerberos. Give it a full read.

Case# 240381

The main thing I noticed in this case is that Forms is configured to use Laserfiche Server authentication. The specific stack trace error you reported was:

Stack Trace:
Caught exception: Laserfiche.Forms.CommonUtils.Exceptions.LFFormsException
Message: The username or password is incorrect. [LFF11-WrongUsernameOrPassword]
at Laserfiche.Forms.CommonUtils.LfFormsNamedUserSession.LfLogIn(String userName, String password, String tenantName, Boolean tryDomainAccount)
at Laserfiche.Forms.CommonUtils.LfFormsNamedUserSession.LogIn(String userName, String password, String tenantName)
at Laserfiche.Forms.CommonUtils.FormsSessionManager.CreateConnection(IEntityContext appContext, String userName, String password, String tenantName, String sessionToKill, Int32 oauth_id)
at Laserfiche.Forms.CommonUtils.LaserficheMembership.ValidateUser(IEntityContext appContext, LogOnModel account, String& authenticationCode)
at E_Forms.Controllers.AccountController.LogIn(LogOnModel model, String returnUrl)

I checked the code for the "LfFormsNamedUserSession.LfLogIn(String userName, String password, String tenantName, Boolean tryDomainAccount)" function and found the following:

• If the username contains the "\" character (like "domain\username")
  • Forms checks that both the domain and username components are present. If there are not, it throws the Exception "FormsErrorCode.WrongUsernameOrPassword"
  • If a domain prefix is not provided (like "\user" I think), it performs a "IsDomainUserValid" function pre-check in the Forms database to see if the domain user exists there. If the pre-check fails, it then checks AD (which is slower, so we do it second). If the AD check fails, it throws the Exception "FormsErrorCode.WrongUsernameOrPassword".
• If those initial checks pass, it will try to open a valid session to the repository using the provided credentials. If there is a (repository?) AccessDeniedException for any reason other than Laserfiche Server error code 9025: Error Reading Repository Information, or Repository Misconfigured Forms will also return "FormsErrorCode.WrongUsernameOrPassword"

Those are the three places where the "WrongUsernameOrPassword" Forms exception is mentioned in the relevant code for Forms domain user login when configured for "Laserfiche Server Authentication".

One thing that jumped out at me in the code is that Forms does not appear to accept UPN formatted domain usernames (user@domain.com), only the "user\domain" format. I'm not 100% on that, but I don't see any obvious handling for it. I know other Laserfiche applications, including Directory Server and Windows Client will accept domain credentials with UPN formatted usernames.

I don't suppose you were trying a username format other than "domain\user"?

If not, the issue quite likely came from the "try to open a repository session" step where the Windows impersonation bit failed for Kerberos config reasons. I can't immediately think of why else switching the Laserfiche Server address from the DNS alias to the machine name would have made it work. That just screams Kerberos issue and could likely be addressed with a Computer Name Alias, assuming both machines are on the same domain.

Finally, I would advise switching to LFDS for auth if at all possible. Much cleaner. 

------------------------------

I can not understand how the communication works when sharing a single routing service between 2 forms servers. You have to open like 10 ports just to do this and there is no TLS option here.

Forms has two components, the Forms IIS web application and the Forms Routing Service Windows service. When thinking about how these two components interact, I find it helpful to view the Forms IIS app as the Forms client, and the Windows service as the "Forms server".

The Forms IIS web app communicates with the Forms Routing Service using Windows Communication Foundation's (WCF) TCP (NetTcpBinding) transport method. The NetTcpBinding uses the default "Transport" security mode uses Windows Authentication to authenticate the client and server, and TLS for the encryption itself. The TLS implementation is handled by the Windows operating system and AD in some way that does not require the usual binding of certificates to ports. This default Windows/AD-based security mode requires that the client and server are on the same (or trusted) AD domain. See:

• Transport Security with Windows Authentication - WCF | Microsoft Learn
• How to: Use Transport Security and Message Credentials - WCF | Microsoft Learn

The Forms Routing Service implements a collection of sub-services (lfrouting, lfinstance, lflookup, etc.) that each have their own WCF endpoint. Each of those endpoints has its own designated port and path. The Forms IIS web app implements a WCF client that communicates with the various Forms Routing Service WCF endpoints. User form submissions, for example, are sent to the lfrouting endpoint. The granularity of those endpoints (and all their ports) is the foundation for advanced configurations like a Forms Routing Service Cluster.

When you edit the Forms IIS Web.config or RoutingEngineServiceHost.exe.config files as part of a Forms DMZ or load balanced setup, you're modifying the WCF endpoint and (sometimes) transport security configurations.

The transport security mode is the "TLS option". It's enabled by default. If you set "<security mode="None" />", you are disabling both service auth and TLS encryption. For DMZ scenarios where a Forms IIS instance is not hosted on a trusted domain, you have the option to use certificate authentication instead of Windows Authentication. The new Forms Multi-Server Configuration Utility helps facilitate a certificate auth configuration so you can still secure Forms IIS to Routing Service traffic in DMZ scenarios. For general info, see:

Working with Certificates - WCF | Microsoft Learn

And to quote that article, to again reinforce that the authentication and encryption functions of certificates and TLS should not be considered separable:

The primary function of a certificate is to authenticate the identity of the owner of the certificate to others.

...

Disable chain trust

When creating a new service, you may be using a certificate that is not issued by a trusted root certificate, or the issuing certificate itself may not be in the Trusted Root Certification Authorities store. For development purposes only, you can temporarily disable the mechanism that checks the chain of trust for a certificate.

Re:

Say we have an LFS/Forms/Routing server and a Public Forms (Web App Only) server sharing the routing service (and sharing a DB): Does the public server ever even make a connection to the repository to begin with?

The separate Forms IIS web application instance does not communicate with Laserfiche Server as far as I can tell. The Routing Service handles the calls to Laserfiche Server for Save to Repository tasks and is almost certainly the component running the user authentication calls that try creating a repository session. If the stack trace above came from an event log on the server hosting the Routing Service, that's definitely the case.

Based on our Wireshark monitoring it does not appear that it does, it seems to send a single packet to the routing service just to trigger a lookup of what to do from the DB. If this is the case, that means there is no traffic containing any confidential data between servers and/or between networks (in the case of a DMZ).

The Forms IIS web app not connecting to the repository does not mean there is no traffic containing any confidential data going between servers/networks. The Forms IIS app has all sorts of sensitive communications with the Forms Routing Service and potentially SQL (for lookups). At minimum, user-inputted login credentials (except with Kerberos), auth tokens (for backend validation), and all form submission data is sent from the separate Forms IIS app to the Routing Service over the network. Many attacks today start with an adversary capturing credentials or access tokens from network traffic.

The only scenarios today where I'd generally even consider not having the Forms IIS web app to Routing Service traffic encrypted is:

• A DMZ setup with the public/secondary Forms IIS instance
  • Only serving unauthenticated/anonymous public Forms, so no login info or auth tokens, which I'd explicitly disable by blocking the login page to be safe.
  • With those public forms either having no lookups, or no lookups that could possibly return any sensitive info (e.g., a lookup of counties by state would be fine).

Instead of a simple DNS alias, set an AD Computer Name Alias, which will automatically handle the Kerberos SPN configurations in addition to the DNS record. See: Using Computer Name Aliases in place of DNS CNAME Records - Microsoft Community Hub

That article also helps explain why names are important for Kerberos. Give it a full read.

This is all new information to me. I am confused why localhost works if this is the case. This one field where you enter the repository server name is so much more than it appears, I thought it was just a way for Forms to find the LFS service in case it is on another machine. This helps elaborate on what happened though, at the point of closing the case I was completely in the dark on why users were being told their pass was wrong.

Given this information I am surprised "localhost" works out of the box

We were using all 3 possibilities. username (PQDN), domain\username (FQDN), and the very rare alternative user@domain.extension. We can simply switch to localhost and it will tell me my password is correct regardless of what syntax I use for my username and when we switch to forms.domain.com it says my password is wrong.

The fact that it says your password is wrong is misleading since that is not the case. Also their IT has an audit trail on kerberos traffic, it shows no kerberos request is made at all in the situation where it is telling us our password is wrong.

Finally, I would advise switching to LFDS for auth if at all possible. Much cleaner. 

This is whole other can of worms for us.

The separate Forms IIS web application instance does not communicate with Laserfiche Server as far as I can tell. The Routing Service handles the calls to Laserfiche Server for Save to Repository tasks and is almost certainly the component running the user authentication calls that try creating a repository session. If the stack trace above came from an event log on the server hosting the Routing Service, that's definitely the case.

This is what I was thinking was the case, but was looking to confirm.

At minimum, user-inputted login credentials (except with Kerberos)

 That is not good! The option to make forms public does not show in the interface when using the primary forms server, so they must manually login as a local admin user on the public server just to make a form public. This means a master password is unencrypted in a network transmission.

Update:

Found the credentials in plain text on the network

I can even see my domain credentials if I attempt to login to the public server by accident.

What are we missing to encrypt this? Can you link me to this multi-server config utility?

 The new Forms Multi-Server Configuration Utility 

There is a configuration utility? The last few that were configured were done with notepad.

This is the documentation we reference for sharing a routing service, seems it might be out of date

https://support.laserfiche.com/resources/4156/hosting-laserfiche-forms-11-in-a-perimeter-network-dmz

Re:

This is all new information to me. I am confused why localhost works if this is the case. This one field where you enter the repository server name is so much more than it appears, I thought it was just a way for Forms to find the LFS service in case it is on another machine. This helps elaborate on what happened though, at the point of closing the case I was completely in the dark on why users were being told their pass was wrong.

Given this information I am surprised "localhost" works out of the box

We were using all 3 possibilities. username (PQDN), domain\username (FQDN), and the very rare alternative user@domain.extension. We can simply switch to localhost and it will tell me my password is correct regardless of what syntax I use for my username and when we switch to forms.domain.com it says my password is wrong.

"localhost" and IP addresses* are not valid Kerberos SPN targets, so Windows has special handling for them where it automatically uses NTLM instead. You're not encountering Kerberos-related issues with "localhost" because it's falling back to the insecure NTLM protocol. 

*Microsoft recently added support for manually setting Kerberos SPNs for IP addresses to help with legacy compatibility in environments where (a) IP addresses must be used for app configurations that perform WinAuth and (b) NTLM is fully disabled. This does not apply to Laserfiche. Don't do this.

The "password is incorrect" error message can be misleading if you take it too literally. Don't get hung up on literal interpretation of the words. Understand that it's a defined exception class in Forms which a variety of auth-related errors can fall under. In this case, the real error is "the target server is unable to decrypt the Kerberos ticket with your credentials, and therefore cannot validate that they are correct". Any time you see a Laserfiche-defined error code or exception class like "[LFF11-WrongUsernameOrPassword]", it's an exception we defined that one or more other underlying errors fall under, and the error description necessarily can't detail every conceivable lower-level exception that might trigger it.

I approach troubleshooting such error codes like so:

1. Double-check what it says at face value, for example by trying both user\domain and user@domain.com formats and typing out the known correct password in Notepad before copy/pasting it in. And if that doesn't work:
2. Interpret it extremely broadly as indicating there's an issue in the general area, like "There's an issue somewhere in the auth chain resulting in an auth failure".

Re:

This is whole other can of worms for us.

I'd be interested in knowing why.

Re:

That is not good! The option to make forms public does not show in the interface when using the primary forms server, so they must manually login as a local admin user on the public server just to make a form public. This means a master password is unencrypted in a network transmission.

The technical function of the Forms Portal license flag is to enable on a Forms instance:

1. Serving public forms
2. Marking Forms as public

Having a Forms Portal license for each Forms server (Primary and DMZ) allows you to mark forms as public from the internal/primary instance. That's how Forms Portal licenses have worked for over a decade, from before my time.

Found the credentials in plain text on the network
...
What are we missing to encrypt this?

WCF transport encryption with certificate auth, which you can set up with the new Forms Multi-Server Configuration Utility.

Re:

There is a configuration utility? The last few that were configured were done with notepad. 

Yes, it was released with Forms 11 Update 5. You can find documentation on using the utility here: Laserfiche Forms Multi-Server Configuration Utility - Education Resources

And the utility itself it in the \Program Files\Laserfiche\Forms\ directory after installation.

Please be aware that it's an initial release and may have some rough edges or cases it doesn't handle as well. The main goal for the first release was to make configuring Forms DMZ/workgroup setups with WCF certificate encryption easier and less error-prone with the utility than by hand, for the most common scenarios. We'd appreciate if you document and share any issues you run into with it so we can improve future versions.