There isn't a way to limit the memory used.  Audit Trail shouldn't require much memory at idle, but some operations can require significant amounts of memory: e.g. importing large log files, exporting large reports.  Those are relatively quick operations and the memory shouldn't be required for long.  However, .Net applications are typically not eager to return memory to the OS and so the effects can be visible to your monitoring tools for a while.  When investigating memory usage, make sure you are distinguishing between the different types.  I could imagine a large value for "Commit Size" and a much smaller one for "Private Working Set", which I don't think is cause for concern.  This is a good starting point for understanding the differences.

We really need a way to set a maximum here, I can't just ask IT to provide some unknown amount. Today trying to configure a report for last 60 days ate up 12GB of memory maxing out the system and crashing the audit trail service all together.

This means I can not get my 60 day report. However if it used an isolated working set it could still accomplish the task without crashing itself, it might just take a little longer.

You can put a little data in memory and then dump it when your done with it, no reason to eat up 12GB.