methodology_cert_img_0009_mit_scr2

Methodology

“Measuring performance” means collecting many different test samples in order to put them into the right context and give valuable information about network quality.

Please scroll down for more Information

Testing networks for a better mobile experience

METHODOLOGY

We select typical customer-usage patterns and stretch network capability to its limits. P3 measures smartphone voice and data performance based on extensive drive-tests – from major metropolitan areas to smaller cities, roads and railways. We also perform walk-tests in big cities with special focus on busy intersections, on public transport and in buildings. To reflect the increasing importance of data to smartphone-users, we weight data with a 60% share, with voice accounting for 40% of the total result.

As the de-facto industry standard, the P3 benchmarking methodology focuses on customer-perceived network quality and covers a wide range of mobile services. P3 objectively defines the routes and test methodology and publishes results through certificates or public benchmark reports. Public benchmark reports are published with selected media partners.

DRIVE TEST

The P3 drive test scenario is carried out with two or more cars driving around in cities, small towns and on connecting roads. The cars are equipped with arrays of selected measurement devices that correspond with each other. The tests are divided into two main categories which are respectively, smartphone voice and data services.

In 2015 alone, P3 compiled more than 60,000 measurement hours in 47 countries across five continents, with its test vehicles covering more than 1,200,000 Kilometers.

DATA

VOICE

dataData services are tested by browsing the most popular websites, by transferring files and by playing YouTube videos. File transfer performance is evaluated on access success rate, stability and the data transfer rate. For the YouTube tests interruptions, buffering times – and video quality are reported.

voiceVoice services are tested by assessing smartphone-to-smartphone calls, including background data traffic in parallel. The success of the call setup, the call setup time and termination is evaluated as well as the voice quality perceived over the mobile connection.

CROWDSOURCING

Crowdsourcing is becoming a significant method to collect network performance data. With this app-based approach, many end-user smartphones become measurement devices delivering invaluable knowledge on individual voice calls, data and app usage.  When combining the user experiences of tens of thousands of smartphone users worldwide, you gain a comprehensive picture of how customers use and experience mobile communications, both, on mobile networks as well as Wi-Fi networks.

P3’s analyses are based on proprietary crowdsourced data, presenting a holistic view across smartphone user experiences. The data collected is anonymous, and no personal identifiers, i.e. MSISDN, IMSI, IMEI or IP addresses, are collected.

In the telecommunications industry, crowdsourcing is mostly understood as performing traditional “Speed Tests”. Traditional speed tests are manually executed tests, initiated by the end user who download artificial content for a defined period of time from a server and measure how much data was transmitted.

P3’s app-based, on-device data collection engine is sitting passively in the background of a user’s smartphone. One big advantage which comes with using passive data collection is the fact that the sample rate is much higher compared to traditional crowdsourced solutions. A particularly important advantage of the passive approach is that no real user interaction with the measuring component is necessary anymore. This allows P3 to integrate its framework into any application. Further, the overhead of artificial test data is removed. P3’s data collection is strongly tied to an explicit user consent.

Finally, P3’s crowdsourced data complements the drive testing activities and adds important dimensions (e.g. time, geography or variety in devices), which cannot be covered with reasonable effort using traditional testing methodologies.

ANALYTICS

All collected data are transferred via WiFi or mobile connections to the P3 Big Data Platform and are processed overnight. Automated aggregation with self-developed in-house software provides flexibility. Experienced engineers interpret the data according to the P3 score ensuring high quality at all times.

With testing mobile networks in over 47 countries and engineers working on measurement solutions every day, we have gained extensive experience in collecting, storing and analyzing huge amounts of data. By analyzing data according to predefined KPIs, we can provide information for the optimization of networks and also show if networks live up to expectation.

In constant exchange and communications with the industry, the test evaluation algorithms are updated every year to ensure they continue to reflect both the latest technology and how mobile services are consumed by end users. All of this has led to a wide acceptance of our tests and analytics.