Sure:

This is a requisition and purchasing workflow, so the entry flows from the initiator to their department director for approval.  We have a list field 'DirectorApproval' with options for Approved, Denied, Pending, and Not Required.  I have a wait condition the routing activity for the Dept Director that waits for DirectorApproval to equal Approved or Denied. 

Every once in a while the Director will mark the field Approved, and Workflow apparently recognizes the condition is met - according to the Conditions tab in the Workflow details - but does not move on from the Routing Activity to the next activity.  Like its still waiting, but the condition clearly shows it has been met.

Because this is a production environment and the issue has been ongoing for some time, management requests for me to terminate and restart these as soon as I discover them.

I manually terminated and restarted the workflow to keep them operating. 

That in itself should be telling:  I made the workflow so that it checks the various approval fields before routing to the user for approval.  This way if a field is already Approved, Denied, or Not Required, Workflow bypasses routing to them.  I am able to restart these workflows without altering any fields and on subsequent workflow instances, the field is recognized as being Approved, Denied, or Not Required and is routed accordingly

I am keen to see the outcome of this thread. I also have a customer with exactly the same issue that comes up time and again and is persistent across WF upgrades. Since it is a very busy system, up to 15 000 running/waiting instances at any given time and between 15000 and 20000 new instances triggered per day, this issue tends to come up often, albeit at random.

Usually only spotted once users complain that files are not moving after updating them.

I am actually troubleshooting a few issues in an open case at the moment and I can only come up with some wild theories at this point as to why WF may be behaving this way.

I'll be keeping an eye on this thread and will be sure to share if I discover anything concrete. ;-)

Also, for everyone's information, I have opened a case with Support and will report any significant findings.

Thanks for everyone's input and interest in this!

Support did a remote session to our Workflow server.  After poking around a good bit, they recommended we upgrade to latest version.  We were on 9.1.0 and have upgraded to 9.1.1.486.  Upgrade was just done, so I will keep this thread updated on if the issue recurs.

Thinking I have made significant progress towards discovering the cause - and possibly an actual resolution to this issue that has plagued me for so long, I reported my above findings immediately to Support. 

Disappointingly, I have yet to receive a meaningful response. 

I had similar problem before, WF hang in waiting state.

I think you were using “route entry to” and it was waiting, and you updated the WF and then you updated the running instance.

Check your WF admin console Error Log: Current Error Log, and look for this error: Object reference not set to an instance of an object.

if the error is there, then take a look at this page:

https://support.laserfiche.com/ForumsFrames.aspx?Link=viewtopic.php%3ft%3d19945%26amp

These are the steps to recreate the problem:

1- Publish a WF with Route Entry to Folder with waiting status for example <tag> is set and to Send Email every 5 minutes for 60 email - the idea is to have "Route Entry to Folder" to be in waiting state while sending emails.

2- Run it on entry

Definition: Version 1
Designer : Version 1 - Waiting
 

Diagnostic: Version 1 - False - Next Timer: (Date/Time + 5 minutes)

2- Wait for the first email to be sent so you can 5 minutes to do next step

3- Publish: Overwrite: Yes - Update running WF: Yes

Definition: Version 2
Designer : Version 1 - Waiting
Diagnostic: Version 2 - True - Next Timer: (Date/Time + 5 minutes)

4- Wait for the next email, then notice values changed

Definition: Version 2
Designer : Version 2 - Waiting
Diagnostic: Version 1 - False - Next Timer: (Date/Time + 5 minutes)

5- When the Next Timer the WF will stuck on running with these values

Definition: Version 2
Designer : Version 2 - Running
Diagnostic: Version 1 - False - Next Timer: Empty

6- Error in Activity log but last modified date remain the same

7- Set the tag to true to end the wait and you will see an activity (wait for entry change) highlighted in blue in the instance details, but the conditions tab shows the conditions for it have evaluated successfully.

 

Ali,

That is quite interesting.  What had you done to work around this problem in your environment?

Bonnie & Sheldon:  Thank you very much for validating that I'm not crazy or making a careless mistake somewhere!  As a technical-, analytical-, scientific-minded guy who believes that the computer universe is governed strictly by cause & effect rules, this is the kind of problem that keeps me up at night.  The more of us we can find, the more data we'll have to establish a pattern or some kind of link. 

John Geist:  Thanks for your input; I have not tried separating out the wait condition, but I can see how that would at least give better tracking info, and maybe even give me better results.  That may be the next thing I try.

Currently, I do have a 'Check' workflow, running on a 15 minute schedule from 7:00a - 6:00p, that goes through each folder where entries wait.  For each entry, if the appropriate wait condition is found to be satisfied, it sends me an email.  not ideal, but at least I don't have to wait to hear it 'through the grape vine'.

I've noticed 2 problems with this 'Check' workflow - the first of which, I feel I can address; the second, I'm not so sure.

1.  if I don't address a given entry within 15 minutes of reporting, the check workflow will trigger another email on the next check.  I'm going to use a 'Stuck' tag in attempt to prevent these repeats.

2.  If the Check WF happens to catch an entry where the wait condition has been met, but workflow hasn't had reasonable time to act, a false positive will be reported.  I was hoping to somehow check when the wait condition was met and compare that time to current time.  A difference of more than five minutes surely indicates the workflow is stuck, I figure. 

EDIT: Disregard what I wrote previously. (now deleted)

What do you mean by the wait condition has been met but the entries have not moved? Can you give us a better idea of the actual process that is supposed to occur

This workflow is showing as terminated in your screenshot, so it's past that condition step or it terminated in that activity.

My two cents....I have found that separating the wait condition out into its own activity seems to be more consistent and give better visibility for tracking, such putting that right before the routing activity. Have you tried that by chance?

Folks,

I think I'm on to something:

I noted yesterday that some instances show a status of Running in the Search Results pane, while the majority have Waiting status.  Those running instances seem to stay running for longer than any activity should run. So, I went to digging.

Those instances that were sitting with Running status showed an email activity that had been running for more than a few minutes - days in most cases.  See, my WFs are setup so that if they are not acted on by designated users, an email is sent to a manager everyday.  Every day for 90 days.  Eventually that repeating email child activity - its parent activity is a Route to User activity - hangs and runs indefinitely.  So when the user does change their approval field, the wait condition is met, but the instance is still stuck on running that email activity.  See below, a normal instance's activities tab and a 'hung' instance's activities tab.

Normal...

Hanging....

We are experiencing the same problems as being described. We have however 2 variations to the this problem.

1. Where the Entry Condition is met and workflow recognizes that the condition is met when looking at the conditions tab, but still the workflow does not progress. the only solution is to manually complete or terminate the Workflow and restart it on the document. On this we were also advised to upgrade which we did a couple of weeks ago.

2. where the Entry Condition is met and workflow does not seem to recognize the change in the entry has taken place. In this instance, you could find the event in the subscriber, but the even does not seem to be promoted to the workflow service. In some cases we found you can do one of the following for workflow to become aware of the change:

  a. Change the condition being evaluated back to a "false" state, save the document and then change it back to a "true" state. the second time around on some entries workflow will then become aware of the condition being met and carry on.

  b. In some instances we found if the Entry is manually moved to a different folder, then Workflow becomes aware of the Change that initially should have made the waiting condition become satisfied.

  c. some instances nothing neither of the 2 approaches mentioned above worked and the only way to get workflow back on track is to complete or terminate the workflow and start it again on the document.

The second variation is still happening on 9.1.1.365 and just this morning we had 22 entries effected by this.

Thanks very much for the post Miruna.  Your explanation, along with Support's is very helpful.  I feel you all are definitely on to the root of the problem.

I have made the above changes you all recommended and will monitor for more occurrences.

Thanks again!