ad third application server, on one physical box you can run several instances of HP OV SD application servers (see "Multiple Server Instances" in Administrator's_Guide.pdf, page 175)
ad reaching 1 million tickets, *this is IMHO root cause* of your performace issues, start think over Archiving (see under Tools/System/General/Archiving settings); note that there are also some limitations (especially according to relations) as you will probably see
From my experience: with minimum BD and UI-rules, without Sheduled tasks you have to make 1 app server to 50 active users. With many complex rules and sheduled tasks implemented you have to make 1 app for 30 or less active users.
You have 170 active users - try to make 4 app servers minimum. The good idea mentioned above is to make few instances of app servers per one physical server. I think your server will handle 4-5 app server instances with -Xms200M -Xmx700M for example.