To successfully monitor a system under load, both the approach to monitoring performance and the monitoring itself must be relevant to the context of the performance project. Therefore the first step in monitoring should be defining performance objectives. These refer to data that is collected through the process of performance testing and that is expected to have value in determining or improving the quality of the product. However, these objectives are not necessarily quantitative nor directly related to other stated performance criteria. These objectives usually include all or some of the following characteristics:
➤ Contractual. Performance objectives are usually formally defined between the business customer and the testing entity as: ➤ mandatory. Criteria that are absolutely non-negotiable due to legal obligations, service level agreements (SLA) or fixed business needs. ➤ negotiable. Criteria that are desired for product release but may be modified under certain circumstances. These are typically, but not necessarily, end-user focused.
➤ Precision. The wording in which quantitative aspects of performance objectives are written: ➤ exact. Criteria should be reached exactly as written in the objectives, for example, "50% CPU utilization." ➤ approximate. Criteria falls within certain range or has only one limit, for example, "Memory usage per process not to cross over 50MB", "Response time of at least 90% of transaction X should be equal or less than 3 sec." ➤ Boundaries. Performance objectives frequently define certain values in regard to the application under test: ➤ target. This is the desired value for a resource under a particular set of conditions, usually specified in terms of response times, throughput and resource utilization levels. ➤ threshold. This represents the maximum acceptable value for resources, usually specified in terms of response times, throughput (transactions per second), and resource utilization levels.
Performance objectives and their service attributes are derived from business requirements. Monitored metrics, captured by measuring, show the progress toward or away from performance objectives.
This post is part of the Performance Monitoring Best Practices series - you may see all of the posts under PerfMonitoring tag.