Forcing Garbage Collection in Workflow

asked on January 7, 2014

I have a customer who's Workflow server is very busy. They execute in excess of 15000 new workflow instances per day. At any given time, there are up to 25000 instances running/waiting.

I have noticed that the memory usage of the Workflow Service consistently grows and does not seem to drop by much during quiet times (weekends). Memory usage will start at about 150MB on the Monday and grow to as much as 3.5GB by Friday. If one looks at usage over the weekend, it does not seem to drop by much. A simple reboot of the workflow server will "reset" memory usage down to lower levels again and this cycle repeats weekly.

I have run tests on a Workflow Server where I manually call the .Net Garbage Collector and compare this to when I run the same testing workflows without calling the Garbage Collector. The results seem to be very conclusive for me and memory usage seems a lot more consistent and a lot lower.

My questions is this, will I be creating new problems for myself if I manually call the .Net Garbage Collector in the production environment? I am concerned that I am possibly going to affect the long term functionality of the WF system by doing this. I know WF has it's own internal Garbage Collection process, but clearly, in this environment, this process seems to be lacking a bit.

Thanks

Sheldon

2 0

replied on January 8, 2014

Hi Miruna,

Workflow 9.0.2.245 and yes, custom activities are being run. I used a custom script activity to manually call the Garbage Collector to see the outcome.

I ran tests on my test server and here's the interesting bit. I published a blank workflow (no activities). I then ran 4000 instances of this blank workflow and the memory usage was high, it peaked at over 430MB. Then with the custom script activity calling the Garbage Collector included in an updated version of the workflow with another 4000 instances, it peaked at only 150MB.

I have seen that Workflow eventually reduces the usage of memory, i.e. cleans up, over a period of time. It seems to be somewhat random, but after about 10 minutes of idle time, the WF memory usage comes down to about half, which is around 220MB. Take note, that this is after 10 minutes. It seems that it never gets to its original starting size, which in this case was about 80MB on my test server. I don't expect it to go back to 80MB, but at least if the system is idle, I want to know why it's using this extra memory.

That brings me to my next point, if you are running WF in a very busy environment, as is the case with my customer, then WF does not get much of a chance to stay idle and run cleanup operations. If cleanup operations run regardless of the server being idle or not (which I believe it does), then why is my manual Garbage Collection showing more promising results in memory usage as opposed to letting WF manage this itself?

My assumption is that leaving WF to cleanup for itself results in alot of seemingly "Wasted" memory hanging around.

Any deeper understanding of this behaviour would definitely help.

I will open a case under my account, where I am free to perform as much testing as I need, since I believe I have been able to replicate the behaviour in my own environment.

Please advise on how to install and run Windbg, since I am not familiar with this program and it is not installed on my Windows Server system.

Thanks

Sheldon

0 0

View 3 previous replies

replied on January 9, 2014

The instructions are in the file attached to my previous post.

0 0

replied on January 9, 2014

Hi Miruna,

Now you've lost me. I did read the attachment previously, however I am not familiar with the program Windbg and do not know where/how to get it and execute it.

The first line of the instruction reads: Attach WinDbg x64 or x86 (based on architecture of workflow server) File->Attach to a Process

...and this does not make much sense to me.

Thanks

Sheldon

0 0

replied on January 9, 2014

Oh, sorry about that.

So once you get Windows Debugging Tools installed, they'll show up as "Debugging Tools for Windows" in your Start menu. Launch the link called "WinDbg". If you're running with UAC, make sure you launch it as an administrator.

From the File menu, you should have an "Attach to Process" option. When you click it, it will show a list of the processes running on your WFServer machine. You're looking for "Laserfiche.Workflow.Service.exe". That should open a window that looks a lot like this:

If you don't see the Command pane, open it from the View menu. Then copy and paste the commands from the file in my first post and paste them into the little editbox at the bottom of the command pane and press Enter. At this point, it should say "*BUSY*" while it's making the log file and change back to looking somewhat like a cursor when it's done.

So now you should have the log file. Go to the Debug menu and select Detach Debuggee, then close WinDbg.

0 0

replied on August 11, 2014

Hi both, was there any follow up on this ? what was the outcome ? should we be considering adding garbage collection to our workflows containing custom script activities ?

0 0

replied on August 11, 2014

No, please don't. You should make sure, however, that you properly dispose of all your resources.

Workflow garbage collects every 1 minute. There was nothing in the dump file other than a very active Workflow server.

Are you seeing high memory usage on your Workflow Server?

0 0

replied on August 11, 2014

Hi Chris,

To answer your question, there is still no final conclusion reached for me on the behaviour that I have observed. However, since then, WF 9.1 has been deployed and there seems to be about a 10% improvement in memory usage and it is seemingly more stable for my situation at my client. I am still not convinced that a WF server using nearly 3GB of RAM accumulated over a 2-3 week period is still reasonable though. It gets ample opportunity over the weekends to perform garbage collection, but it seems that it still does not do so as efficiently as I would like. There is a constant trend of increased memory usage, and never any pattern of reduced memory usage if one has to look at the performance counters.

I also don't have any answer for you on why WF uses so much RAM to run a blank workflow.

Perhaps whatever WF is utilizing in memory is not being flagged for cleanup? I wouldn't mind opening this up for further discussion again to help get a better understanding of WF behaviour, so that we can plan required WF resources more efficiently.

Regards,

Sheldon

0 0

Question

Question

Forcing Garbage Collection in Workflow

Answer

Replies

Sign in to reply to this post.