I've never done this before and there's no built-in option. I'd really like to see this as a feature as well because it would really help with one of my biggest business processes.
It would be a bit tricky, but is possible you could do it with javascript.
Each file upload input has an accompanying div with the "fileUploadSize" class which in turn has an attribute called "fsize" and another called "res"
fsize stores the file size numeric value
res stores a value indicating you whether the number is in KB, MB, etc. (STR_SIZE_KB, STR_SIZE_MB, etc.)
You could add a change event handler to try and detect whenever the files change, then add up the totals to check against your maximum.
You would need to convert the different sizes to get an accurate total, but it should be possible in theory.
Are you using a single upload field, a collection, or a table? Depending on how your form is set up, it could be more difficult to get the right event handler configuration.