You are viewing limited content. For full access, please sign in.

Question

Question

Errors Processing Workflow from Public Submission

Forms Workflow Version 11
Updated November 6, 2023
asked on October 30, 2023

We have a public Forms environment running in a DMZ. 

Forms Version 11.0.2212.30987

We have it configured so the Forms instance in the DMZ is the same system that is running internally as well (same database, and using the routing service on our internal server).

We haven't run a lot of stuff through the public environment yet, but we have a new one that we launched last week, including an email campaign inviting submissions.  Since that launch, I've noticed a bunch of these submissions getting suspended (not all of them, but about 25%-30% of them).

The process only has three parts - the Message Start Event, a Workflow Service Task, and then the End Event.  The error is happening on the Workflow Service Task.  This workflow copies the details of the submission into a different process for internal processing and review.  So it's a pretty simple process - but the hiccup is specifically calling the Workflow.

The event logs around the time of the error looks like this on the DMZ:

An unexpected error has occurred. [LFF502-UnexpectedError]

Details:
URL: 
Error: UnexpectedError
Date: 10/30/2023 9:40:42 AM (Mountain Standard Time)
HTTP Status Code: 500
Business Process ID: 0

Stack Trace:
Caught exception: Laserfiche.Forms.CommonUtils.Exceptions.LFFormsException
Message: An unexpected error has occurred. [LFF502-UnexpectedError]


Inner exception: System.Data.Entity.Infrastructure.DbUpdateConcurrencyException
Message: Store update, insert, or delete statement affected an unexpected number of rows (0). Entities may have been modified or deleted since entities were loaded. See http://go.microsoft.com/fwlink/?LinkId=472540 for information on understanding and handling optimistic concurrency exceptions.
   at System.Data.Entity.Internal.InternalContext.SaveChanges()
   at Laserfiche.Forms.Routing.SubmissionRecovery.CheckSubmissionsGetNonProcessedInternal(RoutingEngineAppContext context)

Inner exception: System.Data.Entity.Core.OptimisticConcurrencyException
Message: Store update, insert, or delete statement affected an unexpected number of rows (0). Entities may have been modified or deleted since entities were loaded. See http://go.microsoft.com/fwlink/?LinkId=472540 for information on understanding and handling optimistic concurrency exceptions.
   at System.Data.Entity.Core.Mapping.Update.Internal.UpdateTranslator.ValidateRowsAffected(Int64 rowsAffected, UpdateCommand source)
   at System.Data.Entity.Core.Mapping.Update.Internal.UpdateTranslator.Update()
   at System.Data.Entity.Core.Objects.ObjectContext.ExecuteInTransaction[T](Func`1 func, IDbExecutionStrategy executionStrategy, Boolean startLocalTransaction, Boolean releaseConnectionOnSuccess)
   at System.Data.Entity.Core.Objects.ObjectContext.SaveChangesToStore(SaveOptions options, IDbExecutionStrategy executionStrategy, Boolean startLocalTransaction)
   at System.Data.Entity.Infrastructure.DbExecutionStrategy.Execute[TResult](Func`1 operation)
   at System.Data.Entity.Core.Objects.ObjectContext.SaveChangesInternal(SaveOptions options, Boolean executeInExistingTransaction)
   at System.Data.Entity.Internal.InternalContext.SaveChanges()

And the event logs look like this on the internal server:

An unexpected error has occurred. [LFF502-UnexpectedError]

Details:
URL: 
Error: UnexpectedError
Date: 10/30/2023 9:40:43 AM (Mountain Standard Time)
HTTP Status Code: 500
Business Process ID: 305
Instance ID: 319124
Business Process Name: Holiday Skip-a-Pay - AMUCU

Stack Trace:
Caught exception: Laserfiche.Forms.CommonUtils.Exceptions.LFFormsException
Message: An unexpected error has occurred. [LFF502-UnexpectedError]


Inner exception: System.Net.WebException
Message: The remote server returned an error: (401) Unauthorized.
   at System.Net.HttpWebRequest.GetResponse()
   at Laserfiche.Forms.Routing.WrapHttpWebRequest.GetResponse()
   at Laserfiche.Forms.Routing.LFWorkflowService.Execute(Int32 instanceId, IRoutingContext routingContext, RoutingInstanceStatus OriginalStatus)
   at Laserfiche.Forms.Routing.ServiceTask.Execute(Int32 instanceId, IRoutingContext routingContext)

The error message I see within the Forms client looks like this:

An unexpected error has occurred. [LFF502-UnexpectedError]

Details:
URL: 
Error: UnexpectedError
Date: 10/30/2023 9:40:43 AM (Mountain Standard Time)
HTTP Status Code: 500
Business Process ID: 305
Instance ID: 319124
Business Process Name: Holiday Skip-a-Pay - AMUCU

Stack Trace:
Caught exception: Laserfiche.Forms.CommonUtils.Exceptions.LFFormsException
Message: An unexpected error has occurred. [LFF502-UnexpectedError]


Inner exception: System.Net.WebException
Message: The remote server returned an error: (401) Unauthorized.
   at System.Net.HttpWebRequest.GetResponse()
   at Laserfiche.Forms.Routing.WrapHttpWebRequest.GetResponse()
   at Laserfiche.Forms.Routing.LFWorkflowService.Execute(Int32 instanceId, IRoutingContext routingContext, RoutingInstanceStatus OriginalStatus)
   at Laserfiche.Forms.Routing.ServiceTask.Execute(Int32 instanceId, IRoutingContext routingContext)

These have all processed correctly once I retried the suspended tasks - but I haven't had any luck discovering why some of these are getting suspended while most of them are not.

Has anyone seen a similar issue and/or have suggestions on how to address this issue?

Thank you.

0 0

Answer

SELECTED ANSWER
replied on October 31, 2023 • Show version history

@████████- thank you for responding.

I don't have a cluster running - just the one routing engine running on the internal server.

But your comment has pointed me to a couple issues with the routing, and that could possibly be the cause of the other errors I'm experiencing.

  1. At some point, the Routing service on the DMZ server has started back up.  I had it configured to be disabled, and to not start automatically, but it started at some point.  So I turned that back off.
  2. I've realized that our vendor, when opening up the access between the DMZ server and the internal server, opened the connection to port 8172 for the routing service on the internal server (per Laserfiche's whitepaper), but they didn't open up the alternate port 8176 (per my instructions to them).  We're using port 8176 because we also use Web Deploy (in fact, I think I'm the first person who reported conflicts on that port with Web Deploy, that led to Laserfiche adding functionality to switch to the alternate port when conflicted). 
    I'm irritated both that they didn't follow my instructions, and that I didn't notice until a year later (but in my defense, the majority of the forms are processing without issue, it's just a small percent of them that are suspending, and this is the first high-volume process we've put in place). 
    I'm going to have to get our vendor to make some changes, and that's going to take time. 
    Hopefully, that resolves the issue.  🤞

 

EDIT TO ADD: Some combination of one of these two issues being resolved appears to be the resolution.  I haven't had anything suspended for this reason since these items were resolved.  Although I can't definitely say the issue is resolved when it was happening inconsistently before - I do have a couple dozen examples of the form submitting without the prior issues - so I'm feeling pretty good about it.  I'm going to mark this comment as the answer, for anyone reviewing this post in the future, but I have to say thank you so much to Yuxuan Wang, because I wouldn't have been looking in this direction without their suggestion, so thank you!

3 0

Replies

replied on October 30, 2023

Hi Matthew,

 

From DbUpdateConcurrencyException, do you have a routing service cluster configured on your internal server(s)? where multiple routing engines are processing submissions and workflow callback. 

1 0
SELECTED ANSWER
replied on October 31, 2023 • Show version history

@████████- thank you for responding.

I don't have a cluster running - just the one routing engine running on the internal server.

But your comment has pointed me to a couple issues with the routing, and that could possibly be the cause of the other errors I'm experiencing.

  1. At some point, the Routing service on the DMZ server has started back up.  I had it configured to be disabled, and to not start automatically, but it started at some point.  So I turned that back off.
  2. I've realized that our vendor, when opening up the access between the DMZ server and the internal server, opened the connection to port 8172 for the routing service on the internal server (per Laserfiche's whitepaper), but they didn't open up the alternate port 8176 (per my instructions to them).  We're using port 8176 because we also use Web Deploy (in fact, I think I'm the first person who reported conflicts on that port with Web Deploy, that led to Laserfiche adding functionality to switch to the alternate port when conflicted). 
    I'm irritated both that they didn't follow my instructions, and that I didn't notice until a year later (but in my defense, the majority of the forms are processing without issue, it's just a small percent of them that are suspending, and this is the first high-volume process we've put in place). 
    I'm going to have to get our vendor to make some changes, and that's going to take time. 
    Hopefully, that resolves the issue.  🤞

 

EDIT TO ADD: Some combination of one of these two issues being resolved appears to be the resolution.  I haven't had anything suspended for this reason since these items were resolved.  Although I can't definitely say the issue is resolved when it was happening inconsistently before - I do have a couple dozen examples of the form submitting without the prior issues - so I'm feeling pretty good about it.  I'm going to mark this comment as the answer, for anyone reviewing this post in the future, but I have to say thank you so much to Yuxuan Wang, because I wouldn't have been looking in this direction without their suggestion, so thank you!

3 0
replied on November 1, 2023

Thanks Matthew, according to Laserfiche's whitepaper, the routing engine in DMZ should indeed be disabled form service. And hope your findings will resolve the issue !🤞

0 0
replied on November 2, 2023

I'm keeping my fingers crossed.

I'm still waiting for the vendor to help with opening the ports.

But I haven't had the issue in the 3 days since I re-disabled the routing engine service on the DMZ server.  That isn't a guarantee that resolved the issue, because we've had a lot fewer submission in those three days, and the issue is so intermittent, but it's a positive sign.  🙂

1 0
replied on October 30, 2023

I'm sure you thought of this, bu tis this box checked in the properties of the workflow?

 

0 0
replied on October 30, 2023

Thank you @████████

That's not it.  That is marked, but I don't think it would make a difference, as it isn't even getting into the Workflow actually starting.

It's just so weird, since it isn't consistently happening with every submission, only a small sampling of them.  And I haven't found any pattern to which ones experience the issue and which ones do not.

0 0
You are not allowed to follow up in this post.

Sign in to reply to this post.

Details
Related Posts
Loading ...