Ok, so the live log is data in the Server Service itself. Which explains why it doesn't have to match the trace log and why subscriber can be offline for it to work.

Just restarting the Workflow Service can resolve the issue sometimes.

There is no recent errors often when this happens. It seems the situation where restarting the workflow service is the solution is when at some point in the past, workflow lost a connection to it's database.

Even if the database is working fine hours or days later, the subscriber was maybe told to be suspended and never updated to be active again, but I don't see a status of the subscriber anywhere that displays this.

I am not even sure if Workflow is really telling the subscriber to be suspended, but I can't think of anything else since it's not a clear configuration disconnection, but always a restart or re-install of workflow that fixes it.

In this case there was a 5 minute outage for communications to the SQL server at 1am, but 12 hours later there was still no events and no additional errors since 1am.

Workflow Server does not tell the Subscriber to suspend.

What does the repository tab show in the Subscriber Trace?

Well everything looks good now since I took these screenshots before restarting the service to resolve the issue but next time I will look here. What would I look for though, I expect I will see the last event time will be out of date but I can already see that from the last event in the Events tab.

Maybe here I would see the Queue building up?

It depends. If there's a problem connecting to the repository, it will have a warning icon at the top in the "Processing loads" graph. Clicking it will tell you what the error was.

The rest of it might tell you if the problem is actually with the Subscriber or something else. You would see things like when it last got a list of updated rules from the Workflow Server, when it got its last event from the repository, whether there's a backlog of messages to process.

We have determined the cause, and there is something concerning here. IT had started taking down the SQL server for some security updates. We found that if it is down for more than 30 seconds, all changes in the repository just get dumped into a queue until the Workflow Service is restarted. At that point those changes will start being processed.

The problem is this queue is only so large, if the SQL server was down for 30 seconds at some point in the past, and since then too many changes have been made, then only some of the changes will be processed.

They have been doing this security audit thing every week at 1am, so we have seen this happen many times. Each time it starts with a message in the communication log that the connection to the SQL server timed out or that the license was not able to be established with Directory Server because Directory Server could not see the database. Then there is nothing in the trace log until the Workflow Service is restarted.