Is there an option to use STARTTLS in workflow STMP communications?
Workflow Email Configurator - STARTTLS available as an option?
Workflow 10.4.3 uses TLS 1.2 by default. Workflow 10.4.2 and below don't unless you set the .NET Framework and WinHTTP TLS reg keys. That would also be a relevant difference between Forms (which is working) and Workflow (which is not).
Upgrade Workflow to 10.4.3 and see if that fixes the problem.
For SMTP Inbound mail flow, after deprecation of TLS 1.0 and 1.1, we will accept only TLS 1.2 connection. However, we will continue accepting SMTP Connection which is unencrypted without any TLS. Although we do not recommend email transmission without any encryption.