Question

Workflow service terminating because of insufficient disk space

Workflow

Updated June 21, 2022

asked on June 21, 2022 • Show version history

Good day
We have a critical system down situation wherein our Workflow Service isn't starting; it'll start for a few seconds and then stop. The event log gives this error:

Machine: *******

Severity: Error

Application Domain: Laserfiche.Workflow.Service.exe

Process Id: 5720

Process Name: C:\Program Files\Laserfiche\Laserfiche Workflow 10.2\Laserfiche.Workflow.Service.exe

Win32 Thread Id: 7992 - 

Thread Name: 

Message: <Exception handlingInstanceId="6d4b012b-c8df-4e9f-afca-97690ae11fe7">
  <Description>An exception of type 'System.IO.IOException' occurred and was caught.</Description>
  <DateTime>2022-06-21 10:38:53Z</DateTime>
  <ExceptionType>System.IO.IOException, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>
  <Message>There is not enough space on the disk.
</Message>
  <Source>mscorlib</Source>
  <HelpLink />
  <Property name="TargetSite">&lt;undefined value&gt;</Property>
  <Property name="HResult">-2147024784</Property>
  <StackTrace>   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.WriteCore(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.FileStream.FlushWrite(Boolean calledFromFinalizer)
   at System.IO.FileStream.Dispose(Boolean disposing)
   at System.IO.FileStream.Finalize()</StackTrace>
  <additionalInfo>
    <info name="MachineName" value="********" />
    <info name="TimeStamp" value="6/21/2022 2:38:53 PM" />
    <info name="FullName" value="Microsoft.Practices.EnterpriseLibrary.ExceptionHandling, Version=5.0.414.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
    <info name="AppDomainName" value="Laserfiche.Workflow.Service.exe" />
    <info name="ThreadIdentity" value="" />
    <info name="WindowsIdentity" value="**********" />
  </additionalInfo>
</Exception>

The problem is none of the Database or Storage locations seem to be short of space. We've checked:

Diskspace on Workflow / Forms server (there's 15 GB free)
Diskspace on Laserfiche Server system drive (C:) and Volume (E:)
Volumes on Laserfiche server (they are set to auto - roll) once they reach the max size (20 GB)
We are able to Upload files to Laserfiche directly
The Workflow and an External Tables database is on two separate databases; both of which have sufficient space. We had our DBA monitor the database as we started the Workflow service to confirm that it isn't running short of temp space.
Removed Subscriber Logs (C:\Programdata\Laserfiche\WF\ServerData) as described here: https://answers.laserfiche.com/questions/136177/Workflow-ServerData-folder-size-is-very-large

Where else could we look for storage space that may be running out / limited? the only clue seems to be the instanceID (presumably of the workflow). Can we disable / terminate the one instance from starting?

Our Laserfiche versions:
Laserfiche Server: 10.3.1 build 236
Laserfiche Workflow: 10.2
Laserfiche Forms: Professional 10.3.1.690

GFLLFWEb2.png (134.63 KB)

| Download

0 0

Answer

SELECTED ANSWER

replied on June 21, 2022

Can you post the entire error message to see if there's an indication of what Workflow was trying to write to?

Also, your list is missing the Workflow volume. Has that been checked for space? (you can see its location from the Workflow Configuration Manager utility)

0 0

replied on June 21, 2022

Thanks Miruna

I've edited my original post to include the entire error message. The Workflow volume is the C:\Programdata\Laserfiche\WF\ServerData location, right?
If so, the C: has sufficient space (screenshot attached). Is there any other place this can limited?

0 0

Replies

replied on June 21, 2022

Have you checked the properties of the SQL Database for Workflow to ensure it doesn't have a fixed size limit or other similar restriction. Even if the machine has drive space available, the database itself could have a limit that has been reached.

3 0

replied on June 21, 2022

Thanks Matthew

Ruled this out: The databases have Auto Growth enabled.

1 0

replied on June 21, 2022

Thanks for the responses. We were able to locate the disk that ran out of space. One of the Workflows had an SDK component that was writing to a Shared drive of a Server, which ran out of space.
It's likely because it was an SDK but the error wasn't trapped and caused the entire Workflow service to stop instead of just the Workflow terminating.
Did it with help from our VAR, starting the service 3 seconds at a time, doing a search for existing workflows, finding the oldest running workflow and finally narrowing down to the activity that it was waiting on (the SDK activity never failed - it would show it is waiting)

2 0

replied on June 21, 2022

Yes, that's a likely cause. If the script can't write to disk and propagates its error out in such a way that it crashes the Workflow Server, it would start running again once you restart the service. So it would show as waiting while its being queued up for execution again.

2 0

replied on June 21, 2022

The stack trace makes it look like you're not flushing the stream to disk in your code, e.g. with a using block. Doing it that way would have routed the exception through the normal exception handling, and the problem would probably have stayed localized to your task. Instead, the stream was closed and the exception triggered by the object finalizer. It's normal configuration for an exception thrown from a finalizer to terminate the process.

The best practice is to close up your resources as soon as you are done with them. This also prevents problems where the service is shut down or terminated without running finalizers, you could easily end up with data loss.

2 0

You are not allowed to follow up in this post.

Question

Question

Workflow service terminating because of insufficient disk space

Answer

Replies

Sign in to reply to this post.