You are viewing limited content. For full access, please sign in.

Question

Question

Laserfiche Forms encountered an internal error LFF500-InternalError when clickin on Reports tab

Forms Version 11
Updated May 20, 2024
asked on May 10, 2024 • Show version history

This only affects a few of our users. Other users are fine. 

All other tabs in Forms work fine. User can load Inbox, Start Process, Design, Manage, and Monitor with no problems. 

Clicking on the Reports tab will cause the page to hang for a few minutes and then throw the errors in the screen shot. 

I've tried all the basic things you're supposed to. Restart, clear cache, reset browser, different browser, updates are automatic and handled by network team. 

The error in Event Viewer logs and from F12 on the user's computer suggest a component of Forms can't be loaded. I'm as certain as I can be there are no network or server capacity or bandwidth issues. 

Has anyone else seen this? 

0 0

Replies

replied on May 13, 2024

Hi Glen,

HTTP 500 errors should have a backend event log message describing the issue. The HTTP 500 status code by itself just means "server side error". As a server-side error, it is very unlikely this has anything to do with Mac or Safari.

You mentioned that you checked event logs:

The error in Event Viewer logs and from F12 on the user's computer suggest a component of Forms can't be loaded.

The DevTools output about "failed to load resource" is because the backend server had an error and responded with an HTTP 500.

Can you share the full text of any corresponding Event Viewer logs? Check both the Forms "Admin" and "Operational" event logs. Different types of error messages are logged to the different channels and sometimes you need to put them together to get the full picture.

Please also share the full Forms version (e.g., 11 Update 5) or build number (click username -> About in Forms) of the instance the error occurs on.

1 0
replied on May 13, 2024

We are at the most recent patch of Forms 11. 

I could not find any Events in the server's Event Viewer logs that happened at the same time as the user was getting the error. I'll look again to be sure, because yes a 500 error should absolutely be logged on the server. I did not check IIS logs though. 

It's an odd situation. No Windows user has this issue, and I've had Mac users from two other departments use the reports tab with no problems. 

My gut instinct tells me it's a security issue, like network policy or firewall app screening or Phobos or something else like that is messing up the Reports component for that small handful of users. I was hoping someone else in the community had run into this exact issue and knew the cause and how to fix it. 

0 0
replied on May 13, 2024

Here's the error from our Event Viewer on the server, or rather AN error that I think is the one. Note text bolded by me. I think this is partial confirmation of my theory that something on the client computers is blocking access. 

************

An unexpected error has occurred. [LFF502-UnexpectedError]

Details:
URL: /Forms/handlers/downloadattachment.ashx?req=DownLoad&aid=250666&authkey=c370df54-91f3-4a86-a699-0ac8feb0cffc
Error: UnexpectedError
Date: 5/13/2024 12:09:34 PM (Pacific Standard Time)
HTTP Status Code: 500
Business Process ID: 0
User: ************
IP: ************
Browser: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.1 Safari/605.1.15
Business Process Name: 

Stack Trace:
Caught exception: Laserfiche.Forms.CommonUtils.Exceptions.LFFormsException
Message: An unexpected error has occurred. [LFF502-UnexpectedError]


Inner exception: System.Web.HttpException
Message: The remote host closed the connection. The error code is 0x800703E3.
   at System.Web.Hosting.IIS7WorkerRequest.RaiseCommunicationError(Int32 result, Boolean throwOnDisconnect)
   at System.Web.Hosting.IIS7WorkerRequest.ExplicitFlush()
   at System.Web.HttpResponse.Flush(Boolean finalFlush, Boolean async)
   at System.Web.HttpWriter.WriteFromStream(Byte[] data, Int32 offset, Int32 size)
   at System.IO.Stream.InternalCopyTo(Stream destination, Int32 bufferSize)
   at E_Forms.Handlers.DownLoadAttachment.ProcessRequest(HttpContext context)
 

replied on May 13, 2024

Here's the error that shows up in the Event Viewer log. There should not be timeouts happening with connections between the Forms and SQL server. They have far more capacity than they need for what they're doing, the servers are conected directly to eat other on a very fast backbone, and these are only happening when a few specific Mac users are trying to open the Reports tab.

***************

Entity Framework returned the following error when trying to communicate with SQL Server: Maximum number of retries (5) exceeded while executing database operations with 'SqlAzureRetryExecutionStrategy'. See inner exception for the most recent failure.. [LFF18-EntityFrameworkInaccessible]

Details:
URL: /Forms/webapi/v1/businessprocess?bp_fields=id,name&idAndNameOnly=true&isReportingAccess=true&limit=-1&sort=name_asc&thenSortBy=name_asc&totalCount=show
Error: EntityFrameworkInaccessible
Date: 5/13/2024 1:46:37 PM (Pacific Standard Time)
HTTP Status Code: 500
Business Process ID: 0
User: ***********
IP: ****************
Browser: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15
Business Process Name: 

Stack Trace:
Caught exception: Laserfiche.Forms.CommonUtils.Exceptions.LFFormsException
Message: Entity Framework returned the following error when trying to communicate with SQL Server: Maximum number of retries (5) exceeded while executing database operations with 'SqlAzureRetryExecutionStrategy'. See inner exception for the most recent failure.. [LFF18-EntityFrameworkInaccessible]


Inner exception: System.Data.Entity.Infrastructure.RetryLimitExceededException
Message: Maximum number of retries (5) exceeded while executing database operations with 'SqlAzureRetryExecutionStrategy'. See inner exception for the most recent failure.
   at System.Data.Entity.Infrastructure.DbExecutionStrategy.Execute[TResult](Func`1 operation)
   at System.Data.Entity.Core.Objects.ObjectQuery`1.GetResults(Nullable`1 forMergeOption)
   at System.Data.Entity.Core.Objects.ObjectQuery`1.<System.Collections.Generic.IEnumerable<T>.GetEnumerator>b__0()
   at System.Data.Entity.Internal.LazyEnumerator`1.MoveNext()
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at E_Forms.WebApi.Version1.QueryServices.BusinessProcessServices.Get(IWebApiContext webApiContext, Int32 recordQueriedCount, Int32 recordSkippedCount, String sort, String bpCategory, String filterByName, Int32 filterByVolume, String lastQueryTime, String bp_list, String attributes_groups, String thenBy, Boolean idAndNameOnly, Boolean isReportingAccess)
   at lambda_method(Closure , Object , Object[] )
   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.<>c__DisplayClass6_2.<GetExecutor>b__2(Object instance, Object[] methodParameters)
   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ExecuteAsync(HttpControllerContext controllerContext, IDictionary`2 arguments, CancellationToken cancellationToken)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Controllers.ApiControllerActionInvoker.<InvokeActionAsyncCore>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Controllers.ExceptionFilterResult.<ExecuteAsync>d__6.MoveNext()

Inner exception: System.Data.Entity.Core.EntityCommandExecutionException
Message: An error occurred while executing the command definition. See the inner exception for details.
   at System.Data.Entity.Core.EntityClient.Internal.EntityCommandDefinition.ExecuteStoreCommands(EntityCommand entityCommand, CommandBehavior behavior)
   at System.Data.Entity.Core.Objects.Internal.ObjectQueryExecutionPlan.Execute[TResultType](ObjectContext context, ObjectParameterCollection parameterValues)
   at System.Data.Entity.Core.Objects.ObjectContext.ExecuteInTransaction[T](Func`1 func, IDbExecutionStrategy executionStrategy, Boolean startLocalTransaction, Boolean releaseConnectionOnSuccess)
   at System.Data.Entity.Core.Objects.ObjectQuery`1.<>c__DisplayClass7.<GetResults>b__5()
   at System.Data.Entity.Infrastructure.DbExecutionStrategy.Execute[TResult](Func`1 operation)

Inner exception: System.Data.SqlClient.SqlException
Message: Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlDataReader.TrySetMetaData(_SqlMetaDataSet metaData, Boolean moreInfo)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
   at System.Data.SqlClient.SqlDataReader.get_MetaData()
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
   at System.Data.Entity.Infrastructure.Interception.InternalDispatcher`1.Dispatch[TTarget,TInterceptionContext,TResult](TTarget target, Func`3 operation, TInterceptionContext interceptionContext, Action`3 executing, Action`3 executed)
   at System.Data.Entity.Infrastructure.Interception.DbCommandDispatcher.Reader(DbCommand command, DbCommandInterceptionContext interceptionContext)
   at System.Data.Entity.Core.EntityClient.Internal.EntityCommandDefinition.ExecuteStoreCommands(EntityCommand entityCommand, CommandBehavior behavior)

Inner exception: System.ComponentModel.Win32Exception
Message: The wait operation timed out


 

0 0
View 4 previous replies
replied on May 13, 2024 • Show version history

Quick note, it's helpful if you make subsequent replies to the comments (even your own) rather than the post, so Answers threads them in order. It can quickly get difficult to follow with a bunch of long top-level comments.

So the underlying error is:

Entity Framework returned the following error when trying to communicate with SQL Server: Maximum number of retries (5) exceeded while executing database operations with 'SqlAzureRetryExecutionStrategy'. 

The real error is here:

Inner exception: System.Data.SqlClient.SqlException
Message: Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.

That's a communication path between the Forms Routing Service and SQL. I see it's referencing "SQL Azure" for a retry strategy. Is this connecting to Azure SQL Database or Azure SQL Managed Instance?

Laserfiche Forms is using Microsoft's own System.Data.SqlClient library to connect to SQL. The error message is coming from there and Laserfiche is surfacing it. 

"Execution Timeout Expired" means Forms sent a SQL query and didn't receive a response within the execution timeout, which is 30s by default in .NET. I'm not sure if Forms has a custom execution timeout. If the execution timeout is 30 and the max retries is 5, that's consistent with the description of the "page hanging for a few minutes" (while it retries) and then throwing the errors in the screen shot" (after the last retry fails). 

Execution timeouts are categorically not network related as far as I'm aware. I'm sure there's some scenario where it's possible, but given everything else that's working we can safely rule networking out.

A SQL instance can have plenty of resources and still not finish processing an individual query within the execution timeout window for a variety of reasons. 

Troubleshooting this sort of issue starts by identifying the specific SQL queries that are failing to complete within the execution timeout. 

I would recommend both:

  • Opening a support case
  • Using SQL monitoring tools to try to identify the queries failing to complete in time. Looking for ones made against the Forms database that take >30s is a good place to start. See: 
    • Monitor and performance tuning in Azure SQL Database and Azure SQL Managed Instance
    • SQL Server Profiler (ignore the warning about it being deprecated, Profiler is great for this and the "replacement" called Extended Events is awful to work with).
      • Microsoft says: "SQL Server Profiler is used for activities such as:
        • Stepping through problem queries to find the cause of the problem.

        • Finding and diagnosing slow-running queries."

      • Which is exactly what we want to do.

      • Note: Does not work with Azure SQL Database
  • Support will need this info anyway

 

Finally, you said "We are at the most recent patch of Forms 11."

Is that:

1 0
replied on May 14, 2024

We have an on-site SQL server cluster of one physical machine and two VMs. My ability to administer that SQL server is limited.

The strange thing is this issue only seems to afflict specific users, and it affects them consistently. Other users have never seen this issue. There is one unique thing about the users who have this problem, they're very limited in the number of reports and processes they have access to. 

I will temporarily give those users Admin rights and see if the timeouts are being caused by their access to reports being filtered by rights. 

I was unaware of the Hotfix. We are up to Update 5. 

0 0
replied on May 14, 2024 • Show version history

I have determined the cause of the issue. 

We have hundreds of processes and hundreds of reports, and when a user is restricted to manager rights to several of those processes, and rights to only a dozen of the reports, the Reports tab crashes. 

When I gave the user full rights to manage all processes and view all reports the Reports tab worked fine. 

We have several reports that are 'ALL PROCESSES' reports that are for giving a monthly overview for all processes. They perform very VERY slowly, and I wonder if maybe those are the cause because Forms reports needs to check user rights to all processes in those reports.

1 0
replied on May 14, 2024

Glad you sorted it out. That tracks. When you said it consistently affected specific users, I was suspicious it might be related to access rights. Checking access rights is a generally "expensive" operation whenever there are a large number of objects to individually check (reports in this case) along with the need to recursively check and calculate inherited permissions from groups on each one. 

Same reason the Bypass Browse repository privilege can yield significant performance enhancements in some cases:

  • Bypass Browse: The ability to see the existence of all entries in the repository, regardless of whether the user has the Browse right for those entries or not. This can enhance performance, as Laserfiche does not need to calculate rights for each entry in each folder. Does not allow users to see documents if they are tagged with a security tag the user does not have or to see the contents of a folder if they do not have the Read right on the folder.
0 0
replied on May 16, 2024

The error is still occurring even after we deleted the problematic reports and restarted the Forms Routing service. I'll open a support ticket with Laserfiche Consulting. 

1 0
replied on May 16, 2024

Sounds good. If you can, do see if you can get a head start and identify the offending queries with SQL Profiler. If you have the queries, Support can usually pass those to Dev and get a quick response on exactly what the issue is.

1 0
replied on May 20, 2024 • Show version history

I have determined the cause and fixed the issue. 

Snapshots of scheduled reports getting too large for the database query to finish when the user was not an administrator on Forms was the cause. 

The only fix was to delete years worth of snapshots and shrink the database until it started working again. 

FEATURE REQUEST:  Set a limit on the number of snapshots when creating a schedule report either or both by X number of snapshots or by time period (past three months). 

EDIT TO ADD: This bloated the size of our Forms application database by 10GB. 

2 0
You are not allowed to follow up in this post.

Sign in to reply to this post.

Details
Related Posts
Loading ...