Hi Matt,

First, a basic but important question: How exactly do you know that traffic is being blocked? Where is that information coming from?

The lflicensing service endpoint is likely involved in application license checks and potentially sending requests to LFDS for user synchronization.

I am not aware of any well-known vulnerabilities associated with services that run on TCP port 8738. More specifically, it is neither a "well-known" port for any protocol (e.g., HTTPS on 443) nor a standard registered port for any other application I could find.

They said it's blocked that way within their environment too, not just ours, so they suspect there is a known vulnerability on that port that is causing it to be blocked by something other than our firewall.

This is highly unlikely. I am reasonably experienced in searching for security vulnerabilities. There are zero hits in the NIST National Vulnerability Database of CVEs for the search term "port 8738", nor do did I get any hits anywhere else. The only public mentions I can find about a service using that port are about Laserfiche.

In any event, here are your next steps:

1. From the DMZ server, run PowerShell command "Test-NetConnection $internalFormsServerHostname -Port 8738"
   1. That command will attempt to open a basic TCP connection on the port. If it succeeds, you know you don't have a TCP/IP-level (network/Windows) firewall block.
   2. If it fails, you almost certainly have a local or network firewall blocking the traffic.
   3. Note that the outbound (source) port is potentially an high-range ephemeral/dynamic port, not also 8738. You need to make sure DMZ server outbound traffic from those ports is allowed. A different ephemeral/dynamic port may be used each time. You can verify if this range is used via Wireshark capture on the DMZ machine (set a destination port filter for 8738 to exclude everything else).
       
2. Check if Windows Firewall is enabled on the DMZ and internal machines.
   1. If enabled, check if there is an Outbound rule on the DMZ machine that would let the TCP traffic destined for 8738 out, and an Inbound rule on the internal machine that would let the TCP 8738 traffic in. Make sure to check the firewall rule scopes (e.g., that a rule permitting the traffic isn't restricted to Domain traffic, etc.)
   2. If not enabled, Windows Firewall is obviously blocking anything.
       
3. If the Test-NetConnection TCP test succeeds but the Forms traffic is still getting blocked, there's likely something doing application-layer inspection. First, confirm there is no network security appliance doing traffic inspection.
   1. If there is, check its logs to see if it's blocking anything. If it is, investigate what rule is getting tripped, which will likely lead to the "why". As any rule being tripped on this legitimate application traffic is either a false positive or unnecessarily overzealous, put in an appropriate exclusion. For an example of "unnecessarily overzealous", I once saw a Web Application Firewall (WAF) all /LFDSSTS auth redirects. When we looked at what rule tripped, it was "URL parameter contains more than eight encoded characters", from the URL's relayState param containing the encoded original Web Client/Forms URL. This wasn't a security threat but still ran into a default rule that considered that pattern "possibly suspicious".
   2. Really, really confirm that there is no network security appliance in the path. I've seen way too many cases where after days of troubleshooting, someone finally realized that a network/security team was routing all the DMZ/internal traffic through a transparent security appliance and didn't tell anyone. 
   3. Also confirm there is no outbound proxy configured for the DMZ machine that could affect such TCP traffic.
   4. If there is neither a network security appliance nor outbound proxy, it could be endpoint protection (Sophos), though I think that's unlikely. If you're at this step, check the Sophos logs (or engage whichever team can do so). Such a block action should be recorded. If it's not, ask the security team to put it into whatever "Log Everything" mode is, then reproduce the issue and check the logs again. 
       
4. If at this point Test-NetConnection is still failing, you have a basic "why can't I establish a TCP connection between Server A and Server B" and someone in your IT organization who is responsible for "network security and routing" needs to get involved with troubleshooting, because it's not an application layer issue.

Hope that's useful guidance on the networking and security side.