2016│11│07 – The 2016 Mobile Network Test in the United Kingdom
Download the report as PDF here
For the third year in a row, the consulting, engineering and testing company P3 communications GmbH and connect magazine teamed up to evaluate the performance and service quality of the UK’s mobile networks.
Based on the findings of an extensive and comprehensive series of tests, the 2016 P3 connect Mobile Benchmark UK produced some surprises including two of the candidates achieving exactly the same score.
Results in a nutshell
The 2016 P3 connect Mobile Benchmark in the United Kingdom shows clear improvements both in the scores and KPIs of all four UK mobile network operators and produced a tie for the top Position.
P3‘s network benchmarks are widely accepted as being both highly authoritative and objective. The carefully designed methodology involves drive and walk tests in 13 large cities in the UK, as well as drive tests in smaller towns and along trunk roads and motorways. Great attention to detail includes the use of up-to-date LTE Cat 9 smartphones for the data tests and the use of the most comprehensive mobile plans available from each operator with adaptations to reflect real life conditions. The cities in which we tested accounted for more than 14 million people, or approximately 23% of the UK population.
After an extremly close race, EE and Vodafone came joint top, with both competitors achieving 803 out of a possible maximum of 1,000 points.
EE scored highly with its very strong data performance, while Vodafone achieved the best results in the overall voice category. Vodafone also achieved the biggest improvement on the 2015 scores. Three came third overall and was also third in the individual voice and data disciplines. This operator scored best in the voice tests conducted on connecting roads and also achieved good results in smaller towns.
O2 also achieved a considerable improvement in comparison to its 2015 results. The O2 network scores best in the voice tests conducted in London and ranks second in the voice category overall, leaving room for improvement, particularly in the data category.
The most positive outcome of this year‘s mobile benchmark however, is that three out of four operators achieved “good“ ratings – a first in the UK. This is proof not only of the huge efforts the providers took in extending their networks, but is also evidence of the influence of the P3 connect Mobile Benchmark on the development of the country‘s Networks.
THE UK’s OPERATORS
Of the four mobile networks competing in the UK, EE and O2 are the largest players in terms of subscribers, followed closely by Vodafone, with the smaller Three attacking with agressive tariffs.
With approximately 30 million customers, EE (formerly Everything Everywhere) is the biggest mobile network operator in the UK. Since January 2016, EE has been part of the British Telecom Group. EE started offering a 4G service in October 2012, that recently surpassed the 12 million subscriber landmark. The operator claims that its 4G coverage today reaches more than 95% of the UK population. The geographic 4G coverage currently is approximately 67%, with an ambition to hit 95% by 2020. EE operates its 4G network both at 1800 MHz and 2600 MHz.
First sites with Cat 9 LTE
The operator announced in September 2016 that it will support Category 9 (Cat 9) capabilities on its LTE Advanced network. So far, Cat 9 has only been activated in Wembley Stadium. Other sites in London, Birmingham and Manchester are to follow in 2016/2017.
Vodafone UK is part of the Vodafone Group which is also headquartered in the UK. The Vodafone Group owns and operates networks in 21 countries. Vodafone UK launched 4G/LTE in 2013. With around 19.5 million subscribers, Vodafone is the third largest mobile network in the UK after EE and O2. In June 2012, Vodafone and O2 signed a deal to “pool“ their network technologies, creating a single national grid of 18,500 transmitter sites. Both networks however announced they would continue to use their own independent spectrum. Vodafone currently operates 4G/LTE at 800 and 2600 MHz and has started to refarm also some of its 3G spectrum at 2100 MHz to 4G.
Carrier aggregation on its way
Under its current 4G+ roll-out, Vodafone is preparing to combine up to three frequency bands with “Carrier aggregation“ in high density areas. This technology could provide data rates of up to a maximum of 700 Mbps, with current Cat 9 devices making use of up to 450 Mbps.
Three UK is a subsidiary of Hutchison Whampoa and launched its mobile service in the UK in 2003. As a relatively young operator Three started as a 3G-only network supplemented by 2G via national roaming. In December 2013, Three began to roll out its 4G/LTE service and expanded it rapidly all over the UK. With about 8.8 million customers Three is the smallest mobile network operator in the UK but claims to carry over 40% of the nation‘s data traffic. Offering the cheapest price for 4G and unlimited data plans (excluding tethering) may well support this claim.
Aiming for large 4G coverage
In addition to 1800 MHz, Three offers 4G also at 800 MHz. Up to now, Three confines its 4G network to LTE Cat 3 or a maximum of 100 Mbps. The company claims to cover 98% of the UK‘s population with 3G. 4G coverage is somewhat lower, but should soon reach a comparable spread.
O2 claims to be the second largest mobile network operator in the UK with approximately 23 million customers. Formerly a subsidiary of British Telecom, O2 plc was purchased by the Spanish telecommunications company Telefónica in 2006. Today, the company also owns half of the mobile virtual network operator Tesco Mobile which operates on the O2 network in the UK.
O2 started providing its 4G/LTE service in August 2013 and has expanded this service across the UK since. In September 2016, O2 claimed to cover over 70% of the UK population with its 4G service, striving to meet a regulatory requirement of covering 98% of the indoor population by the end of 2017.
Focus on LTE footprint
O2 operates its 4G network mainly on 800 MHz with limited additional 1800 MHz coverage in London. Concentrating on enlarging its 4G footprint, the company does not offer data speeds exceeding 100 Mbps (LTE Cat 3) at the time of writing.
A CLOSE LOOK AT UK NETWORKS
In 2015’s mobile network benchmark, EE was the clear winner, followed at a distance by Three and Vodafone with O2 in last place. How did these competitors score one year later?
P3 communications GmbH, based in Aachen, Germany, is a world leader in mobile network testing. It is part of the P3 group, with over 3,000 employees worldwide and a turnover of more than €300 million. P3 is in partnership with the German telecommunications magazine connect, which has more than 20 years of editorial expertise and is one of the leading test authorities in Europe for telecommunications products and services.
Together, P3 and connect have been conducting the most important network benchmark test in Germany for nearly 15 years, extending it to Austria and Switzerland in 2009. Starting in 2014, P3 has also been conducting benchmarks in Australia and the UK, expanding its public mobile network benchmarks to cover the Netherlands and Spain last year.
In 2015 alone, P3 compiled more than 60,000 measurement hours in 47 countries, with its test vehicles covering more than 750,000 miles. As the de-facto industry standard, the P3 benchmarking methodology focuses on customer-perceived network quality – examining both voice telephony that makes up 40% of the total result as well as data connectivity that accounts for 60% of the score. P3‘s network benchmarks are widely accepted as a completely objective authority.
2016‘s results give surprises
Against this background, it was especially interesting to find out whether the UK carriers were able to improve on last year‘s results, bearing in mind that when compared with other European operators, all UK networks revealed room for improvement. See for yourself how they fared and get ready for some surprises.
„All UK mobile networks use network sharing which reduces cost and improves efficiencies. The P3 connect Mobile Benchmark UK reveals that this does not mean that all have exactly the same quality.
The good news is: All operators have improved compared to 2015‘s exercise and further developments are expected in the coming year with Voice over LTE and extended 4G coverage.“
While data communications are ever more prevalent, customers still expect excellent mobile telephony quality.
In order to check out the voice quality of the UK‘s mobile networks, P3 conducted drive and walk tests in 13 cities with more than 100,000 inhabitants and also covered a large number of towns and connecting roads all over the UK. In total, the test vehicles have covered 4,456 miles (see detailed description of the methodology on page 9).
Especially in the cities, Vodafone achieved the best results over its competitors, with O2 following at close distance. In the drive tests covering smaller towns, O2 excels, Three and Vodafone follow closely behind and EE comes last. Voice telephony on trunk roads and major highways obviously is a specialty of Three, with Vodafone ranking second and EE as well as O2 showing some weaknesses in this category.
Vodafone scores best in voice category
When looking at the call set-up durations, last year‘s overall winner EE exhibits a 1.5- to-2-second deficit behind its competitors. All in all, O2 and Vodafone managed to steal last year‘s top position in the voice category from Three. Success ratios and speech quality are mostly on similar levels, with a slight advantage for Three in rural areas as well as in terms of overall speech quality. While O2 is quite strong in smaller towns, it falls somewhat behind on connecting roads.
Voice Results At a Glance
In all tested voice scenarios, last year‘s overall winner EE is losing some ground. Three, which took the lead in the voice category in 2015, has to give way to Vodafone and O2 this year. 2016’s winner
in the voice category across all scenarios is Vodafone – with both O2 and Three following closely.
With the transmitted volume of data growing exponentially, all operators face challenges in providing a satisfying user-experience. Who manages to best meet the growing demand?
While all UK mobile network providers nowadays offer good coverage with their 4G services, there are differences in their rollout strategies. EE and Vodafone chase each other with continually growing data rates that currently go up to 450 Mbps, based on the so-called carrier aggregation (the combination of multiple carrier frequencies). In contrast, O2 and Three stick with a solid 100 Mbps and mainly focus on enlarging their 4G footprint.
P3‘s testing takes both aspects into account – the benchmarking of web-page downloads as well as file downloads and uploads rewards fast throughputs. At the same time, it assesses the networks‘ availabilty and stability by examining success ratios.
The testing of YouTube playback recognizes that this popular video service has been introducing adaptive bit rates recently. This decision of the streaming provider aims at a better user experience, surrendering pixel resolution in favour of stable playback. As a consequence, besides success ratios, start-times and the absence of interruptions, the average value of the obtained video resolution became another important performance indicator.
EE clear data leader in large cities
In the drive tests and walk tests that P3 conducted in the 13 large cities of the UK, EE takes a clear lead in the data category. With this year‘s results EE maintains the high performance level it showed in our 2015 benchmark. Vodafone is following at a distance of only 3 to 4 percentage points, while Three ranks third and O2 last – the latter showing obvious weaknesses especially in the walk test Scenarios.
Vodafone narrowly ahead in smaller towns
Vodafone emerged narrowly ahead of its competitors in the results from drive tests in smaller towns, with EE and O2 a mere 1% behind in this category. In this discipline, Three is falling back a little, showing slightly worse success ratios, taking slightly more time to download the test web pages and achieving lower download and upload rates in some test cases. Still, all four operators show viable results when it comes to data connectivity in smaller towns.
EE also leading on major roads
On the major, connecting roads, EE again takes a clear lead, providing fast data rates and achieving convincing success ratios. Three also shows strong results in this category, while Vodafone and O2 lag a little behind – especially with regard to the success ratios of web page downloads and file transfers. Nonetheless, with all success ratios being well over 90% even on the relatively demanding connecting roads, all four mobile network providers of the UK improved considerably over last year‘s results.
Data Results At a Glance
Especially in the data categories, the 2016 mobile network benchmark shows considerable improvement in all four UK Networks. EE takes a clear overall lead in data services, with Vodafone narrowly ahead in smaller towns and overall second in the data test.
Traditionally, P3 and connect take a closer look at the capital to see how the operators cover this busy Metropolis.
Being a major business hub and by far the most populous area in the UK, London is an especially demanding terrain in which to deploy and maintain a mobile network. This is why we regularly take a closer look to see how coverage in the capital compares with the rest of the Country.
O2 offers best London voice service
In the drive tests and walk tests measuring voice services, O2 scores best and takes a lead over a second-ranking Vodafone. EE and Three share the third spot in this discipline.
Also in London: EE number one in data
In the London-based data tests, EE reclaims the top position that it already earned in the nationwide data category. Vodafone follows closely, just five points behind.
Overall London champion: Vodafone
Combining the voice and data results achieved in London, Vodafone comes out 5 points ahead of EE in this discipline, due to a good performance in both categories.
London Results At a Glance
In comparison to 2015’s test, all four carriers improved their services in the London area. The overall winner in the capital is Vodafone, with EE not far behind. In any case, definite improvements are visible in London as well as in the rest of the UK.
The methodology of the P3 connect Mobile Benchmark is the result of P3’s many years of experience. It was carefully designed to evaluate and objectively compare the performance and service quality of the UK’s mobile networks from the users’ perspective.
The P3 connect Mobile Benchmark took place throughout September 2016. All samples were collected during the day, between 8am and 10pm. The network tests covered inner-city areas, outer metropolitan and suburban areas. Measurements were also taken in smaller towns and on trunk roads and motorways. The combination of test areas had been selected to provide a significant series of test results covering the UK population in England, Scotland, Wales and Northern Ireland. The areas chosen for the 2016 test account for more than 14 million people, or approximately 23% of the total population of the UK.
P3 conducted the tests with four drive-test cars, equipped with arrays of Samsung Galaxy S5 Cat 4 smartphones (Voice) and Samsung Galaxy S7 Cat 9 smartphones (Data) for the simultaneous measurement of voice and data services. Additionally, two teams conducted the walk tests, measuring voice and data performance.
For the voice walk tests, one walking operator contacted a counterpart located in one of the drive test cars. Furthermore, two smartphones per operator in each car were used for voice tests from car to car. The voice quality was evaluated based on the HD-voice capable and ITU standardised so-called POLQA wideband algorithm. In order to account for typical smartphone use scenarios during the voice tests, background data traffic was generated in a controlled way through random injection of small amounts of HTTP traffic. The voice test scores account for 40% of the total benchmark results.
Data performance was measured using one smartphone per Operator in each car. The radio access technology was set to LTE preferred mode in order to reflect the customer experience. All the tests were conducted with the best-performing mobile plan available from each operator and in a full drive and walk test mode covering big cities, small towns and major connecting roads, including motorways. The data scores account for 60% of the total results.
Routes and samples
The test routes are shown on page 1 of this report. In the big cities and smaller towns indicated, the cars had to follow predefined routes. Altogether more than 80,000 speech samples were logged per operator, about half of them in 4G-preferred-to-4G-preferred mode, while the other half were 4G-preferred-to-3G-preferred.
For the data benchmark about 50,000 samples were obtained per operator. About 60 % of the samples were obtained in big cities while the remaining 40% were collected in towns and on connecting routes.
Performance indicators and rating
To reflect the critical importance of service access and stability, the percentage of these two KPIs was the most heavily weighted. We also measured performance indicators for voice such as call set-up time and speech quality. For data, performance indicators included the average session time to access web pages or transmit small files, throughput rates and average start times and resolution for YouTube videos.
The score weighting reflects both the geographical distribution of the UK‘s population and a ranking of usage scenarios. Therefore, 675 of the total of 1,000 maximum points were assigned to the cities – 525 maximum points referring to the drive test results (data: max. 315 points, voice: max. 210 points in voice) and 150 maximum points reflect walk test results (data: max. 90 points, voice: max. 60 points). In towns and on connecting roads, only drive tests were carried out. In the towns, a maximum of 150 points is available, with a maximum of 90 points in the data, and 60 points in the voice category. To reflect the importance of connectivity while driving on the road network, a total of 175 points has been allocated to this category, divided into a maximum of 105 points for the data results and 70 points for the voice results. The table on page 2 shows the percentage of maximum points that each operator reached in each discipline.
Even with our fine-grained scheme awarding a maximum of 1,000 points, in 2016 we have two equally strong operators at the top.
The race was extremely close, and even our fine-grained scoring scheme ended up with a photo-finish deadheat between two winners: Scoring exactly 803 points out of the possible maximum of 1,000 points, EE and Vodafone share the winning position this year. EE achieves joint top-spot on the back of its very strong data performance, while Vodafone scores best in the voice category. O2 is nonetheless very close when considering voice-only but is held back from achieving a better total by its limited data performance. Interestingly, last year‘s distinct winner EE achieves the weakest score in the voice category. This however does not mean that EE is particularly bad in this discipline – but that its competitors successfully made inroads on their opponent in this field. Three ranks third with good overall results.
Scrutinising benchmarks certainly helped to improve the UK‘s mobile networks
Probably the most interesting and positive outcome of 2016‘s P3 connect Mobile Benchmark UK is that three out of four operators were rated “good“ despite the use of a more demanding scoring scheme than last year. This is an unprecedented result in this market. A glance at the development of the scores achieved in the last three years shows the great improvements that the UK operators implemented in this period. P3 and connect take pride in the assumption that our scrutinising benchmark has very probably played an important role in this process.
With excellent data performance, EE is clearly ahead of the competition in this category. So last year’s winner is still one of now two top operators in the UK. If EE had not lost some ground in the voice discipline, it might still be the uncontested number one. Regardless, EE customers enjoy one of the UK’s best mobile networks.
Compared to 2015, Vodafone hit the biggest score improvement – which reflects the advances of its mobile network. Vodafone ranks best in the voice discipline and stays narrowly ahead of EE’s overall results in the cities and towns. In spite of some room for improvement on roads, Vodafone is one of the UK’s best operators.
Two test category top-spots helped put Three in third position overall this year – which is also their respective rank in the voice and data categories. In the voice discipline, Three scores best on connecting roads and also achieves good results in smaller towns. Adding its nice data performance, Three deserves the overall grade „good“.
In comparison to 2015’s benchmark, O2 also achieves a considerable improvement in its score and KPIs. Scoring best in the voice tests conducted in London and ranking second in the voice discipline overall, the operator might consider putting additional efforts into strengthening and enlarging its data service.
Objective testing will be essential to the approaching 5G networks as well as emerging technologies like autonomous driving and smart cities. P3 communications is preparing for these future challenges.
Implementing super-fast 5G connectivity in the UK will require accurate and objective data about coverage and the user experience. However, with 5G‘s approach of assigning different network capabilities to different use cases, an extended approach to network testing will also be required. P3 communications is already planning for this not-too-distant future by developing concepts to benchmark and analyse future networks‘ end-to-end performance with advanced analytics and big-data Technology.
Extensions to the test routines
But the future is not a couple of years away – it actually starts tomorrow. Therefore, in order to augment its testing methodology, P3 communications has developed a comprehensive network-quality metric-collection method that also involves crowdsourcing. Starting as soon as the 2017 network benchmark season, crowdsourcing can be expected to become part of P3‘s scoring scheme.
Readers interested in taking part in this approach can already participate by downloading the “U get“ wireless performance rating app – see details on the right-hand side. In addition, Voice over LTE (VoLTE) will be an integral part of next year‘s testing programme in the UK, as 2017 will see a proliferation of devices supporting this with some operators already having launched VoLTE services this year.
Also, with Cat 9 becoming relevant for a larger user base and even the first Cat 12 devices appearing on the market, P3 will prepare for increasing data speeds as well as considering new environments and over-the-top Services.
Last year’s report can be downloaded here