You are viewing limited content. For full access, please sign in.

Question

Posted to Laserfiche Cloud

Question

Workflow Reliability: What Can Cause a Laserfiche Cloud Workflow Instance to Stop Progressing Without Failing? (Simple Workflow Ran for 3 Hours, No Error)

asked on June 6

Hello,
We recently had a Laserfiche Cloud workflow instance remain in a running state for approximately 3 hours without completing or generating an error. Under normal conditions, this workflow completes in approximately 10 seconds.


The workflow itself is pretty simple. Its primary purpose is to generate a custom Process ID for a given Business Process by reading a value from a Data Management table and incrementing a counter on the appropriate row. The table contains fewer than 30 rows and the workflow performs only a read and an update on one row.


After seeing that the instance kept running for a few hours, I immediately "re-tried" the workflow using the same process context and inputs. The new instance completed successfully in its normal 10 second timeframe without any workflow changes.


Because the replacement instance succeeded immediately, I'm trying to understand what types of conditions could cause an individual workflow instance to stop progressing without failing outright.
My questions are:

  • What can cause a Laserfiche Cloud workflow instance to remain running for hours without generating an error?
  • When canceling a stuck instance and starting a new instance succeeds immediately, does that typically indicate a platform issue, a resource lock, a service dependency issue, or something else?

As a mitigation, I have added an escalation activity that will notify me if future instances exceed the expected runtime. I am also considering allowing this workflow to run concurrently to not block the business process from running since the workflow itself is not mission-critical.


This workflow supports a time-sensitive business process where users expect near-immediate completion. Even short delays are noticeable, and in this case the 3-hour delay caused operational disruption because downstream activities were blocked while the workflow remained in a running state.

I am interested not only in understanding possible root causes, but also in learning best practices for monitoring, resiliency, and prevention of similar incidents in Laserfiche Cloud.


Thanks!

stuck_wf.png
stuck_wf.png (133.79 KB)
0 0

Replies

replied on June 8

This was likely caused by one of our servers running out of resources on 6/4. We're looking at potential solutions for mitigation.

As a side note:

  • Workflow is not synchronous, so near real-time execution is not guaranteed. Depending on load on the account, workflow instances may queue up
  • If this ID is generated by reading a a value from a table and adding 1, the values are not guaranteed to be unique. 2 (or more) workflow instances may read the table at the same time, which would result in them generating the same ID value. 
0 0
replied on June 8

Hi Miruna,

Is there a way to garuntee sequential number generation within LF? I presume a remote REST service could be create and and designed to protect against collisions. Other than that, reading from the LF table, writing the metadata, then checking the table for duplicates could also help prevent collisions. Loop and repreat until success. A bit messy though.

0 0
replied on June 9

Remote web service would do it. Calling into an external SQL to get the value from a sequence will also work as SQL will handle concurrency. 

1 0
replied on June 12
replied on June 12

Miruna,

it works a treat!

0 0
replied on June 8

Thank you for your response! Knowing that this was likely related to a server resource issue is very helpful.

From an operational standpoint, how are incidents like this typically communicated to customers? For example, if a server resource issue causes workflow delays or stalled instances, is there a status page, service health notification, incident report, or other communication channel that administrators should monitor?

Since our workflows support time-sensitive business processes, understanding how and when platform-related issues are communicated would help us distinguish between workflow-specific troubleshooting and broader service disruptions.

 

Thanks again!

0 0
replied on June 9

Issues are posted to https://status.laserfiche.com. While an issue is ongoing, a message also appears on the login screen.

For this one, the server was rolled within minutes of exhibiting resource issues, so it was treated as maintenance and not posted because other parts of the cluster were ensuring the service was not impact. The impact only became clear with further investigation. We are looking at how we can further improve visibility. 

0 0
You are not allowed to follow up in this post.

Sign in to reply to this post.