I don't think you would be able to do this the way you have explained using the Modern designer. Classic designer, sure, but not the Modern Designer.
It looks like the submissionHandler on the LFForm object doesn't have any way to report which button was clicked, at least not that I can find. And you can't access the buttons directly, because the Javascript runs in a sandboxed iFrame so it's effectively running on a different webpage.
You can definitely run some code on the submissionHandler, and that triggers before it even tries to validate the other form fields. So you could have it prompt to complete the indicated field before submission, like this:
LFForm.onFormSubmission(function(){
if (LFForm.getFieldValues({fieldId:1 }) == "") {
return {error: "Reason for Rejection is missing. It must be populated before submission."};
}
}, {handlerName: "submissionHandler"});
However, that seems like an overly complicated way to require a field, you could just mark it as required. It still is going to do it for any Submit/Approve/Reject/SaveDraft submission instead of just the Reject submission.
The Javascript functionality in the Modern Designer is so limited, that it's really hard to make it work for anything outside of the most basic functionalities - so if custom Javascript is really needed for your process, you are best still sticking to the Classic Designer.
One idea that will work with the Modern Designer is to use the field rules. And this is really where the Modern Designer shines, because the field rule options are more powerful. You could make it so that the Reject button isn't even shown unless your field is populated. Unfortunately, I don't think this can be tied to the standard comment box, but could be tied to a field you add.
The field rule on the form would look something like this:
I also used this CSS to hide the built-in comments box, since I'm using a multi-line field for "Reason for Rejection".
#comment-container
{
display:none;
}
The form looks like this.
When the field is not populated, the Reject button is hidden.
When the field is populated, then the Reject button is shown.
You could even add field rules on other fields that are normally required, but are not needed on a rejection - when the Reason for Rejection is populated. Of course you can hide the fields. But with the new options, you can now leave them shown but disable or change their required status. Like this rule, that takes a field flagged as Required and disables it and switches it to optional if the rejection reason is populated.
