If the executable that you are running displays a window or is expecting any user input, it will not terminate - you will need to kill it from the task manager.
If that is not the case: are you able to run the exe from a command window successfully? If you are, then do some experimentation - start with a simple example: exec xp_cmdshell 'dir c:\' and see if that works for you.
quote:Is it possible to open exe program and stop the query execution automatically.
This is not possible. xp_cmdshell operates synchronously, so control is not returned to the T-SQL session until the command is completed.
When you use xp_cmdshell, commands are run in the background. You would not see the calculator window on your screen. You can see calc.exe in task manager (be sure to check the box that says "Show processes from all users" in the task manager).
The bottom line is that xp_cmdshell should NOT be used to run programs that require user input to proceed or require user input to terminate.