Performance testing for continuous volume testing, tuning, scaling, migration, and risk assessment
“The team managed different challenges as rollout/ migration projects, campaigns (Single Day, Black Friday) or regular performance validation to our complete satisfaction. The high motivation, indefatigable engagement and goal-oriented principle of operation are one of the standing out qualities of Accesa’s team. We are looking forward to working together with the team the next years.” – Nam Nguyen, MediaMarktSaturn Technology Senior Test Strategy Consultant
Client background and business context
Client: largest European electronics retailer
Client’s figures: €22 billion in revenue (2016)
Project: performance testing implementation
Capacity limitation: the client’s solution platform needed initial gauging in real-world scenario conditions. This would allow to better assess risks under a normal daily load or during peak sales periods. Based on the client’s existing measurements, the application was able to handle 400 transactions per second (TPS). As the company grew and diversified its needs, this number of transactions represented a potential risk.
Undefined performance test strategy: performance testing revolved around a rarely conclusive end-to-end (E2E) approach, preventing early issue detection, by overlooking components, features, and endpoint-level isolated assessment.
Limited test platform coverage: limited test scripts customisations at country or brand level. Limitations were also present in terms of request distribution, usage profile and load-sourcing patterns (mobile, web, app).
Deprecated test-data script sourcing that were put in place for previous testing sessions.
Setting up the performance testing strategy was our prio 0 aspect. While keeping close contact with the client’s development, infrastructure, and middle-ware team, we’ve started implementing and documenting a long-term, sustainable approach for performance testing and expanding coverage, in a prioritised manner.
In doing so, we’ve provided our client with an early detection mechanism, accurate prediction strategies, invaluable load-model emulation, and scalable cross-brand and cross-country test coverage.
All these were vital assets in ultimately fulfilling the end-customers’ desire for a customer-experience oriented shopping experience.
Apache JMeter and HP LoadRunner to replicate user load on the SUT
BlazeMeter and Octoperf 3rd party cloud provider
Multi-zone load distribution using AWS, Scaleway, DigitalOcean
Grafana , DynaTrace, Munin, AppDynamics, NewRelic, Kibana as monitoring tools (APM)
Jenkins, Taurus for CI / CD integration of Performance Validation in the pipeline
JAVA, C, Groovy, BeanShell as scripting languages
Containerised load generating infrastructure deployed on client premise
One of the greatest achievements for this project was that we collaborated with our strategic partner’s developers to make the shift towards performance testing.
We did it by changing the mindset and by looking at performance testing as the most important and final validation gate between the features and the end-user. This approach reshaped and amplified their overall inherent skills to a mass-user-centred perspective.
Some of the benefits achieved through our work and close collaboration with the development teams were:
Capability Buffered Solution: during our collaboration we’ve helped visualise pain-points, component strain, scaling factors, and balancing strategy issues. After this endeavour, we were able to deliver to the company comprehensive reports of issues to be one-by-one addressed and fixed. Our contribution enabled the platform to go from 400 TPS (in 2014) to 9000 TPS (measured during 2018’s Black Friday preparation).
Full test coverage: test scenarios and reporting mechanisms were put in place. These assess 100% emulated user-behaviour, with integrated real user monitoring (RUM) and application performance monitoring (APM) solutions, allowing the company to act accordingly.
Extended customisation: multi-environment, multi-brand, multi-country with on-the-fly scaling to any new resource accommodated by the solution platform is now a reality. We also implemented a custom utility framework that aggregates test-data from multiple sources, making it seamlessly available for all test scenarios.
OpenSource: at the heart of our test scenario strategy lays the ubiquity of test-script deployment. That’s why we’re focusing on OpenSource–driven solutions. We delivered full compatibility with small-scale debugging approaches and large-scale live assessment via in-house or commercial SaaS solutions.