You won't be able to "hijack" the click event of your submission buttons in the form designer, and you won't be able to close the current window. You can open a link in a new tab though. 

1. Create a button on your form with a custom html element
2. define the button with "onclick='rejectFormAction()'" event handler
3. In the JS window define a function "window.rejectFormAction"

<button type="button" onclick="rejectFormAction()" class="my-custom-button">
  Reject
</button>

window.rejectFormAction = () => {
    // Change the state of the form or add an overlay
    // Overlay could say "you can now close this page" for example
   window.open('https://mylink.com', '_blank')
}