BAC 7.x servers are running multiple processes in different JVM, so you first need to understand for which process you want to change the Java Heap settings.
The settings are in HPBAC\bin\<relevant process name> _vm_params.ini files. Any changes in these files require BAC service restart.
A word of caution though. Changing BAC processes memory settings on your own (if not instructed by HP Support) may result in unpredictable results and unstable system. So don't do it unless you fully understand what you are going to do.
If you later run Connect to DB wizard again, it may overwrite your changes, because BAC detects the amount of physical memory, and decides about the deployment and the amount of memory for each process based on the info in the HPBAC\conf\deployment.xml file.
In BAC 7.x each process has it own configuration file <Process-Name>_vm_params.ini (Located under <ProductDir\bin>) , which is created during config server based on the deployment.xml. (Located under <ProductDir\conf>)
Based on which BAC process is reporting the error, one or more of the following .ini files may have to be edited. Please contact the HP Support for further assistance.