Hi Jesse,
From the screenshot that dialog is coming from the browser and is prompting directly for Windows credentials for pass-through authentication. However, if you have LFDS authentication set up, then Forms does not want these credentials directly; instead it requires a token from LFDSSTS derived from these credentials, so that's why trying to specify creds here will fail.
In IIS, can you check for the Forms application, in the Authentication feature, that Windows authentication is disabled? If LFDS authentication is configured for Forms, only Anonymous authentication should be enabled in IIS (in the web.config file, authentication mode should be "None"), and a custom process is used to enforce the LFDSSTS method.
This seems to be the most likely explanation for your behavior; when the session expires, the site thinks it should use Windows authentication directly, so prompts you. When cancelling out, it will "use" (very loosely speaking) the anonymous authentication to understand enough to kick you back to LFDSSTS signin, where you can provide credentials as normal.
Hope this helps!