Load testing

Load testing evaluates how the system performs under different levels of traffic. It helps ensure that the system remains stable during high-traffic events such as sales campaigns or seasonal peaks. When you change any part of your commerce solution, such as integrations or extensions, performance tests show how those changes affect response times. Early testing helps you detect issues sooner and reduce later rework costs. We recommend running load tests regularly.

Test phases

The typical phases of a load test are:

1. Definition phase:
   • Formulate scenarios by mapping user journeys.
   • Determine levels of traffic and load profiles.
   • Define primary objectives, including key performance indicators (KPIs).
2. Test script creation phase:
   • Design test scripts using the defined scenarios.
   • Run small-scale tests to ensure the quality of the scripts.
3. Load test performance phase:
   • Carry out different types of load tests based on the established scenarios.
   • Create a final report summarizing the results of the tests.
4. Cleanup phase:
   • Remove data generated by the tests once they have been completed.

Load test scenarios should focus on testing crucial workflows that mimic typical user behavior, with an emphasis on impactful and realistic scenarios for the business. For example, simulate a checkout journey: browsing, adding items to a cart, and placing an order.

Tests should examine the system's behavior under regular load conditions, validate its performance during high-traffic periods, and assess its performance under sustained load over an extended period.

Use the following table as a template to document expected hourly activity:

Types of load tests

Baseline load tests

Conduct baseline load tests to simulate routine traffic scenarios and to validate major architectural changes regularly. Plan for future growth based on the results from load test analyses. Keep historic artifacts (load driver reports featuring performance data, resource usage, and observations) for comparison over time.

Aligned with industry best practices, load tests must run for up to 30 minutes, including a 15-minute period for ramping up. In this ramp-up phase, traffic gradually increases, signaling all components involved to scale up. Scaling doesn’t take place instantly, and ramping up leads to better test results.

Peak load tests

Conduct peak load tests using expected peak traffic in readiness for a campaign or seasonal spikes. The duration of this test must be up to a maximum duration of 20 minutes, including a 15-minute ramp-up period.

Choose the right environment

Obligations and responsibilities

• In the context of load testing, specific data such as Product catalog details may be required. Do not incorporate creating this data directly into your load tests. Generate the necessary data incrementally in advance to avoid undue stress on the system. Consider keeping supporting data for consecutive tests.
• Generate traffic from nearby regions to minimize network overhead.
• Start your load testing conservatively. Perform initial testing at lower loads to validate responses and confirm the functionality of your load test scripts.
• Gradually increase load levels during testing. Directly starting at maximum load is unrealistic behavior and not permitted.
• Include periods of think time in performance tests to mimic human behavior.
• Monitor resource utilization (such as CPU, memory, and network usage) on the load driver machines that simulate the traffic.
• Remove and permanently delete data generated by the tests within one week from when the tests have been completed. Automated cleanup is recommended.

Load allowances

Fair-use allowance

The following tests, aligned with industry standards, are considered fair use and are included as part of the commercetools offering by default:

• Automated load tests of development changes on the non-production Project (for example, regression CI/CD).
• Baseline load tests: Up to four production-level baseline load tests per year across any endpoints (such as Carts, Orders, and Products). Each baseline test cannot exceed the expected baseline load.
• One peak load test per year, with traffic levels not exceeding the expected peak load.
• Tests must run only on non-production Projects.
• Approval is required from your commercetools representative by opening a Support ticket 3 business days in advance of any load test. commercetools may allocate an alternative time window for customer testing if multiple customers claim the same time window for load testing.
• Any single test is capped at 10% of the total number of production orders purchased per year, or an API usage commensurate with such Order volume.
• Aggregate annual test order volume across all load tests must not exceed 20% of the total number of production orders purchased per year, or an API usage commensurate with such Order volume.

Remember: all data generated by the tests must be fully removed and deleted within 1 week once the tests have been completed.

Load Testing Premium Add-On

Exclusion periods

Load tests are not allowed in the following periods of the year:

• Approximately between two weeks before and a week after Black Friday.
• Approximately between a week before and a week after December 25.

Exclusion dates will be provided by commercetools in the Peak Season Readiness materials.