Workflow Memory Management - Workflow can use so much memory that it shuts down it's own API

asked on February 14, 2019

We received a notice in the application log that stated this:

The service '/Workflow/api' cannot be activated due to an exception during compilation. The exception message is: Memory gates checking failed because the free memory (730218496 bytes) is less than 5% of total memory.

Workflow was using massive amounts of memory to cache information and improve performance, so much memory that it shut down it's own API to maximize it's performance. But shutting down the API is not worth improving the performance slightly, especially when there has not been any performance issues yet.

We found that the Workflow Service was the cause of the high memory usage. Workflow was using over 4GB of RAM when in reality the current requested data could fit within a 200MB range.

There was literally 4GB of unnecessary memory storage found by restarting the service and clearing the excess caching.

I am fine with caching in memory, but can we limit this so that it does not shut down it's own API just to squeeze a little more data in? I looked through the workflow administration settings and documentation but could not find any memory settings.

1 0

replied on February 15, 2019

Actually, that's not Workflow shutting down the API, that's IIS/.Net and it's based on the memory usage on the machine as a whole, not the Workflow process itself. Without knowing the specs of the machine, it's hard to tell if Workflow using 4GB is the culprit or if there were any other processes also putting memory presure on the machine.

4GB is a bit high, but not unusual for Workflow. 200MB is low, but normal for a service that just started and hasn't done any work.

It's hard to tell what is normal given that Workflow could also have custom scripts that may not be properly releasing their resources. .Net also employs its own garbage collector to make decisions on when memory should be released.

We don't currently have any open reports of memory leaks in Workflow 10.2.1 or 10.4. If you're on 10.0 or lower, I would recommend upgrading to at least 10.2.1. If you're still concerned about memory usage after that, Tech Support has tools for dumping the memory on the Workflow Server so we can see what it holds.

0 0

View 1 previous reply

replied on February 15, 2019

Is there any way to limit the maximum memory though, so that it remains within what is available to the system and keeps the API online?

There are other services that use memory, but workflow was what filled it up by far with the 4GB request. If we allow it to pull unlimited memory the API will always shut down.

0 0

replied on February 20, 2020

As per our discussion at Empower, this is the parameter you can add to the web service's web config to override .Net's default.

In the web.config for the WF Web Service (by default located under <Program Files>\Laserfiche\Laserfiche Workflow <version>\Web\Laserfiche.Workflow.Web), look for the serviceHostingEnvironment element.

It should look something like this:

<serviceHostingEnvironment multipleSiteBindingsEnabled="true" aspNetCompatibilityEnabled="true"/>

Add

minFreeMemoryPercentageToActivateService="Integer"

and set the value to 0.

It should end up looking something like this:

<serviceHostingEnvironment multipleSiteBindingsEnabled="true" aspNetCompatibilityEnabled="true" inFreeMemoryPercentageToActivateService="0"/>

Save the file and recycle the app pool used by the web service. It should no longer block the API calls when the server's memory usage is high.

I would still ask that you open a support case so we can check if there is an actual memory leak in the server.

0 0

replied on February 20, 2020

Hi Miruna

Thanks for this. I was planning to open another support case but it appears since upgrading the server to WF 10.4.1 they are remaining well under 1GB of usage. I even restarted the service to see the difference in usage and it was within a 100MB after restart compared with the 4GB difference previously.

I don't know of any changes in 10.4.1 that would fix this since we were never able to prove the problem, but so far I am having trouble reproducing. When we were heavily researching this previously 10.2 was the latest version available.

0 0

replied on February 20, 2020

That's great to hear. We have done some fixes for memory management for 10.2.1.

0 0

Question

Question

Workflow Memory Management - Workflow can use so much memory that it shuts down it's own API

Replies

Sign in to reply to this post.