[Part 1] Instant coffee, Performance Testing, and the evolution of the Black Friday queues

On 26th November 2015, while many people in the US gathered with their families at home to celebrate Thanksgiving, more people than ever had their minds to the many great online offers. Most probably, last year’s wishbone was about making a wish on getting the best Black Friday deals.

“Don’t buy that now, wait until Black Friday.” – How many times did you hear that? Millions of people say and do that every year. 2016 makes no exception. So, we’ll ask: Will your e-commerce website be ready for what is set to be the busiest online shopping day ever?

Time flies like an arrow and the 2016 holiday season is about to start in a few weeks. But there’s still some time to prepare. Along with your omni-channel strategy, holiday promotional tactics, ad budgets, inventory, refund and return policy, it is essential to make sure your customers’ experiences are seamless when they explore Black Friday offers on your website or buy the magical Christmas gifts for their dear ones.

One practice that helps you ensure superior customer experiences in this context is performance testing, a non-functional testing technique that is performed to investigate, determine or validate application attributes in terms of speed, stability and scalability under a various load. These aspects are measured in order to meet precise performance objectives for the application under test.

In this two-part article, we’ll go into detail about the need for performance testing, why you should consider the business value of performance testing, and how to prepare your e-commerce website for Black Friday and the entire holiday season. Lastly, we’ll look at the future of performance testing.

But first things first.

The road to Black Friday and the holiday season: What was different in 2015?

The United States

Statistics from Adobe revealed that in 2015 online sales during Thanksgiving and Black Friday weekend surpassed 2014 with 18% in the US. Despite the fact that some experts estimated Cyber Monday’s importance as a shopping event would be decreasing, in 2015, sales on Cyber Mondayhave exceeded $3 billion, showing 16% growth from the previous year.

Quiet streets, overloaded e-commerce websites

But the most notable aspect remains this: for the first time ever, 2015 was the year when popularity of online beat offline holiday shopping and more people did their Black Friday shopping in front of their laptop instead of waiting in front of the store. More than 103 million people shopped online over the Black Friday 2015 weekend, whereas less than 102 million people made a purchase in traditional stores.

Slowly but surely, online shopping has been winning the hearts of the consumers over the years. And the trend stays strong.

Mobile sales also continued to increase in 2015, but they have not yet leveled with desktop. Mobile is mainly used for browsing and exploring the best deals while desktop is used for purchases.


Black Friday is global. Although in Europe, it is still perceived as an American event, there are some clues on how the phenomenon grows in the UK, Germany and France.

In the UK, as in the US, the data suggests more interest in online sales. 2015 Black Friday was the UK’s biggest day for online retail ever when online sales passed £1bn on a single day for the first time.

Have you ever thought how much it would cost you if your e-commerce website were down for an hour during peak hour on Black Friday?

Regarding the traffic, Black Friday boosted online traffic by 60% than on a normal day in 2015 and at least 15 retailers’ websitescouldn’t manage the traffic or were very close to their limits of what they could handle. Other websites didn’t “survive” the load with a very expensive impact. Last year, John Lewis experienced record levels and during the peak hour its website was down for 1 hour. At 5 orders per second, 18.000 orders must have been an expensive outage.

What has changed since 2014?

From the site outages point of view, 2015 was better than 2014, when Black Friday was marked by famous “Website is down” statuses. However, the major change stays in the new pattern of commerce: people move from queues to online and that drives retailers to prepare their e-commerce websites to respond quickly enough to the intended users, handle the planned user load and beyond and remain stable under expected and unexpected load.

The road to Black Friday and the holiday season: 3 predictions for 2016 and beyond

1. People will choose to shop more online, giving up to queuing up in front of the store. 2015 Black Friday was the tipping point.

As Adobe Digital Insights 2016 Holiday Shopping predictions for the US and Europe also show, consumers have become more comfortable spending money online.

2. Black Friday might become “Black Week”, so it is more than ever necessary that retailers prepare for a 5 or 7-day peak period.

3. As consumers want a seamless experience, investing in performance testing is essential. A delayed online experience or a broken experience will make them find a better one a click away.

Performance Testing – Why should you consider its business value?

We all expect more from our online experiences. We expect websites to be user-friendly and to look professional, we expect them to be interactive and to offer us fresh experiences, we want them to be secure, and we need them to be fast. Read the last sentence again. Yes, we don’t want the technology to be fast, we need it to be fast. Our general perception of time varies depending on many factors such as location, emotions and other external stimuli. These factors also apply to our online experiences: the average web users perceive the amount of time it takes a page to load as being 15% slower than the actual load time, and later, when recalling the experience, they remember it took about 35% longer than it actually did (Psychology of Performance, 2010).

Moreover, users’ expectations regarding website speed are quite demanding; in 2006 the average online users expected pages to load in 4 seconds or less, today they expect load times of 2 seconds or less for a seamless experience; some users even expect the website to be like instant coffee: they have no patience for the ritual of dripping the coffee down the system and drawing a foamy heart. Not even espresso with its special brief brew time wouldn’t be acceptable. Websites that load in a flash (0.1 seconds) give the users the satisfying feeling of instantaneous response. For the business, according to Aberdeen Group, a 1-second delay in page load time equals 11% fewer page views, a 16% decrease in customer satisfaction, and 7% loss in conversions.

However, any customer-centric company knows how important is its overall customer experience, and that customer experience is positive when the website is also up and running, not only fast. After all, why launch new deals every hour if your users cannot push the buy button?

As a business in the e-commerce industry, you need to prepare and ensure you have the right systems in place so that your online customers are not disappointed; error pages, slow to load websites, a lack of product information will all lead to poor customer experiences and missed sales. Therefore, when it comes to understanding and improving an application, beyond speed, it becomes essential to test its responsiveness in terms of stability – so you would know how well it can handle a particular load, and scalability – in order to determine if it is able to perform under expected and peak load conditions, plus scale sufficiently to handle increased capacity. We’ll return to stability and scalability in the following section.

As we’ve already seen, user satisfaction and conversions are two serious reasons why you should consider performance testing from a business perspective. What other metrics, relevant to your business, are influenced by application performance?

  • Cart abandonment. If it takes too long to check-out, cart abandonment goes up. According to Baymard Institute, a web research company in the UK, 1 in 5 online shopping carts are abandoned due to website errors or crash. Losing customers right before they’re about to pay is quite hurtful for the business.
  • Revenue. The period from Black Friday through Christmas is where sales really increase. These shopping days bring even 50-100% more revenue than normal shopping days. When systems are down or too slow, products remain simply unsold.
  • Brand reputation. Slowness or speed makes such an impact that it can become one of the brand values customers associate with an application. If enough people experience problems on a website, it can develop a reputation for being difficult or unpleasant to use. To contribute to a high profit, it’s important to avoid damaging your brand image and to invest in influencing your customers to keep coming back to purchase and keeping the churn as low as possible.

But how to decide which types of performance tests to run?

Performance testing covers many different tests which enable the analysis of different aspects of the application. They work depending on what you want to achieve: a better response time, stability, scalability.

The need for speed is universal. As expected, businesses also need to know if the application responds fast enough for the intended users. On one hand, the response time in terms of performance can give you a competitive advantage on the web. On the other hand, fast websites increase user satisfaction and they become more likely to make a purchase. Load testing shows you how a system behaves with a targeted number of users and what is the response time received for pages under various scenarios. For example, you’re running a load of 3.000 users to see if the page time exceeds or not 2 seconds, your objective.

But what if you’re interested to find out at which point – 5.000, 6.000 users – your application will not be able to keep the 2 seconds response time? Capacity testing helps you identify how scalable is your application and what is the maximum capacity of users your system can support, while not exceeding the targeted response time. Ideally, the application handles the expected user load and more without impacting the user experience.

Going further, let’s suppose that the capacity testing reveals that your system supports 5.000 users and when the load reaches 6.000 and keeps this load for a period of 10 hours, the page time starts exceeding the targeted 2 seconds. What happens when significant expected and unexpected load is extended over a long period of time? Is the application stable enough to handle the extended load without any deterioration of the response time? Endurance testing might bring the answers and we’ll explore them in the next post.


That being said, in the following article we’ll talk about solutions and we’ll arm you, as a Performance Tester, QA Manager, Test Manager or as a CMO, with five practices for making sure your e-commerce website is ready not only for Black Friday and the entire holiday season, but all the time. We’ll also have a look at the future of performance testing.


Note: This post is part one in a two-part series. Part two will be published next week.