Question

Form kicked off 300+ workflow instances in 12 minutes

Workflow Forms

Updated March 21, 2019

asked on July 25, 2017

I have a form that once submitted, calls a workflow that basically creates a folder in the repository and returns a weblink URL to the forms business process. Since we are returning data to the forms process the "Wait for the workflow to finish before proceeding?" is set to yes. The form and workflow have been tested and work successfully.

At some point when access rights were being updated within the repository where this folder is to be created, the workflow user (along with everyone) had the permission to create folders removed. This obviously terminated the workflow with an 'Access Denied' error when the form invoked the workflow and attempted to create a folder. The form process went into a suspended state yet the workflow kept being called continuously every few seconds until it was manually stopped. While the form was in a suspended state I could reopen the 'View Error Log' tab and see the list of errors growing.

There were around 350 terminated instances of the workflow within 12 minutes. Fortunately our monitors alerted us to the suspended instance and we were able to stop it quickly.

Is this expected behavior, a workflow being called continuously when it terminates while the forms process is suspended? Is there an option I don't know about or other way to prevent something like this from occurring again?

We have corrected the repository access rights and things are working again but I hate to think of something like this occurring after normal working hours!

Thanks,
Andrew

0 0

Answer

SELECTED ANSWER

replied on July 26, 2017

Hi Andrew,

If you are using Forms 10.2.1 and you don't want Workflow service task to be retried automatically when it gets suspended, you can change option in the [cf_options] table in Forms database to disable it.

Find the entry with optionname AutoRetryExcludeWorkFlow, and change its optionvalue to true. Then restart routing service.

Please note that this option would affect all processes in your Forms server.

3 0

replied on March 21, 2019

Hi Rui,

I just want to point out that we made your recommended change on a client's server because they were experiencing a runaway workflow kicked off by Forms (repeating one instance per second) and after doing so Forms completely stopped tracking instances. Any instances that were submitted between when we made the change and when we reverted it have no history whatsoever in Forms or SQL. It is as if they never happened, and LF Support has no idea what could have caused it. Just a word of warning to anyone who might try this solution.

0 0

Replies

replied on July 25, 2017

Unfortunately this is expected behavior.

I just open a similar case.

I call a workflow by forms and wait, the issue is that i send info from the form to an update word and turns out that because got at least one break line the workflow crash and forms start to run every seconds. In this case my database grow to much and the log goes to like 100 gb in a day.

2 0

replied on July 26, 2017

Thanks Rui and everyone else that commented

We decided to update the AutoRetryExcludeWorkFlow option to be on the safe side. That we rarely get workflow terminations the single try option seems much more sane than continuously calling a terminating workflow forever. I can imagine scenarios where that could be quite damaging.

I tested this ( we are on version 10.2.1.157 ) by recreating the conditions I initially posted. There was one attempt to call the workflow which terminated and the forms process suspended. We were notified, fixed the issue in the repository and then used the retry option on the suspended instance and the forms process completed without issue. Perfect!

Andrew

2 0

replied on July 25, 2017

I agree - it's awkward that it keeps calling the workflow, I've had similar issues.

As a workaround, and it's not ideal, but you could wrap part (or all) of your Workflow in a try-catch statement - even if all the catch section does is send a warning email to someone - at least that way if there is an error, the workflow won't terminate and end up getting called again. Of course, that would depend on how well your forms process handles the workflow coming back showing complete without populating values back into your form - that may cause other bigger problems for you...

1 0

replied on December 11, 2018

Just following up on this old thread. Have there been any updates to Forms that resolve this issue? Meaning, can Forms processes be configured individually to not keep retrying the Workflow Service Task when it terminates? Or allow a termination to notify Forms so that it can put an instance into a suspended state? I've configured the AutoRetryExcludeWorkFlow to true in the cf_options table, but it would be nice to be able to do it on a process basis instead of the switch affecting all processes.

1 0

replied on July 25, 2017 • Show version history

Hi Andrew

Support claims that the continuously call to workflow might get fix with another version of Forms.

I, at least for my issue. Im going to run some test and let you know. The one its failing to me is 10.2.1.157

You cal also try that.

Update 07/26/2017

The update still presents the issue. Now i got the 10.2.1.205 so dont try that.

0 0

replied on September 28, 2017

Hi,

Could we request a there is an option within the Forms Process Diagram that we can configure so that Forms will either retry a Terminated workflow or not?

0 0

replied on September 28, 2017

Hi Ronald,

Thanks for the feature request. Do you mean that you would like there to be an option to control retry on individual Workflow service task? So it's not just an option for whole site, but rather an option on task level?

1 0

replied on September 29, 2017

Hi Rui,

Yeah that's exactly what I was looking for, so that you can control it for each workflow service task.

2 0

You are not allowed to follow up in this post.

Question

Question

Form kicked off 300+ workflow instances in 12 minutes

Answer

Replies

Sign in to reply to this post.