Download the report as PDF here
2017│04│20 – Public benchmark report in the Netherlands
After conducting mobile network tests in the Netherlands in 2015 and 2016, the network testing professionals of P3 communications and connect have examined the Dutch mobile networks for the third time. As all four operators showed excellent results in the previous tests, the race is all the more exciting in 2017. How did the four candidates perform?
RESULTS IN A NUTSHELL
Coming from strong results in the 2016 P3 connect Mobile Benchmark in the Netherlands, the Dutch operators were able to maintain their impressive results in 2017. However, we observed some interesting changes too.
P3‘s network benchmarks are widely accepted for setting industry standards as well as being highly objective. The carefully designed methodology scheduled four drive-test cars to visit 17 large cities and 19 smaller towns in the Netherlands as well as the connecting roads. The areas in which we tested accounted for more than 5.1 million people, or about 30 per cent of the Dutch population.
All four Dutch operators achieved very good results with more than 900 out of a possible total of 1000 points. The overall winner is T-Mobile, surpassing even the 950 points threshold and thus receiving the rare grade „outstanding“. The test winner shows convincing results both in the data and in the voice category. The latter is especially remarkable, as T-Mobile was the only Dutch operator at the time of testing that did not yet support the modern Voice over LTE (VoLTE) technology.
With a considerable improvement in its voice performance over the 2016 result, Vodafone ranks second and is able to outrank the market leader KPN. In the data category, Vodafone performed a little behind KPN. However on the whole, the differences between the competitors are small.
KPN, which ranked second in 2016, is beaten by Vodafone with a gap of only one point and ranks third. The Dutch market leader already offers VoLTE, however at the time of testing did not yet support this feature on the Samsung S7 phones that were used for this benchmark. So the voice results of KPN are based on legacy telephony. Apart from long call setup times, KPN achieves very good voice results and also delivers a very good data performance.
The Netherlands‘ smallest operator Tele2 ranks last, but still with very competitive results. A distinct loss of points in the voice category might be explained by the disadvantageous effects of national roaming for voice services on the T-Mobile network.
Many users may prefer internet messaging to traditional voice calls. But in those cases when they are talking on the phone, customers expect reliable connections and good speech quality. How do the Dutch networks fulfill these expectations?
When the tests started, T-Mobile was the only Dutch operator that had not commercially launched VoLTE yet, and KPN did not yet support this feature on the Samsung S7 smartphones that were used for testing. The network-wide introduction of Voice over LTE and its support on a broad range of smartphones take time. So, the test phones were only partly able to use VoLTE, even if they were in areas where they could establish a LTE connection.
T-Mobile first with legacy technology, Vodafone shows most significant improvement
Despite not offering VoLTE yet, T-Mobile is the overall voice leader. The operator thus proves that fast call setup times and excellent speech quality can be achieved with legacy voice technology too. But Vodafone, which has a quota of 99.7 per cent of successful VoLTE calls, follows closely at only three points distance. Also, Vodafone shows the most significant improvement over last year‘s result, gaining 15 points in the voice discipline. Tele2 supports VoLTE in its 4G network, but outside its LTE footprint it realises legacy voice services via national roaming on T-Mobile. In this scenario, Tele2 does not have access to the higher quality voice codecs, which may explain its lag compared to the two leading operators. Due to the lack of VoLTE support on the Samsung S7 test phones on the part of KPN, voice connections in this network were established via circuit-switched fallback. Still, the largest Dutch operator delivers very good speech quality and high success ratios in all examined scenarios.
Voice at a Glance
The overall winner in the voice discipline is T-Mobile – although this operator is the only one in the Netherlands that has not yet introduced VoLTE. Vodafone follows closely behind, while Tele 2 loses points due to legacy national roaming. KPN offers VoLTE, but not yet on the S7 test phones. If the evaluation had considered only legacy calls, KPN definitely would have ranked better.
The volume of transmitted data is steadily growing. So, all operators face the challenge to keep data rates and reaction times at a high level. Which Dutch operator manages to best meet the growing demand?
Good news for Dutch customers: All mobile networks operating in the Netherlands show a very high degree of stability and performance in this benchmark. Whether our test teams drove through large cities, through smaller towns or on the connecting roads – the success ratios observed in each of these scenarios were altogether remarkably high.
In order to check the reliability and performance of data connectivity in the Dutch networks, one Samsung Galaxy S7 per operator was installed in our test cars. The phones constantly performed a suite of tests including web-page downloads, file downloads and uploads as well as Youtube video playbacks. For most of these checks, success ratios and average session times were logged. For the downloads and uploads we also recorded the average throughputs. In order to get an idea of the typical performance, we determined the minimum data rates that are available in 90 per cent of the cases. To also reflect peak date rates, we additionally recorded the speeds that were surpassed in 10 per cent of the cases.
P3‘s approach for testing Youtube playback recognizes that this popular video service has recently introduced adaptive bit rates. This decision of the streaming provider aims at a better user experience, surrendering pixel resolution in favour of stable playback. As a consequence, besides success ratios, start times and the absence of interruptions, the average value of the received video resolution became another important performance indicator.
T-Mobile also leads the field in data
Although all four operators reach very high scores in the cities, T-Mobile takes a distinctive lead over its competitors. Here, KPN reaches the second rank with a gap of only five points to T-Mobile. Tele2 comes in third in the cities. However, slightly weaker values particularly in the file upload category lead to clear distance to the results of the two candidates on the higher ranks. With a very close gap of only one point behind Tele2, Vodafone comes in last in the cities. This position originates in the loss of a couple of points especially in the web browsing and file upload categories. Again, we must emphasize that this ranking order >> is achieved at very high levels of performance and stability, and the overall differences between all four networks are small.
In towns, we see an interesting ranking where T-Mobile and KPN share the same number of points at the top rank, while Vodafone and Tele2 share the second position, again with an equal number of points. The distance to the leading duo in this scenario is six points.
Basically, the same order can be observed on the connecting roads: Again, T-Mobile and KPN are leading the field at the same level, while Tele2 and Vodafone rank second – in this scenario, however, with a very close distance of only two points.
All in all, T-Mobile leads the data category, followed closely by a strong KPN and at a little distance by Tele2 and Vodafone. However, the fact that all four operators perform similarly well in all tested scenarios emphasises the high level of service and performance that Dutch customers can expect.
Data at a Glance
T-Mobile is also the overall winner in the data category, but the gap over the other Dutch operators is small. KPN follows at close distance on a strong second rank, with Tele 2 and Vodafone falling a little behind. However, even the latter two provide very good results in the data category.
In 2000, the German company T-Mobile bought a minority of the Dutch mobile network operator Ben, which two years later was extended to a 100 per cent acquisition. In 2003, Ben was renamed T-Mobile Netherlands, with the brand „Ben“ becoming a „no-frills“ offer within its portfolio. In 2007, T-Mobile Netherlands additionally acquired the operator Orange which until then had belonged to France Télécom. T-Mobile NL is a full subsidiary of the German Deutsche Telekom. At the end of its fiscal year 2015, the company reported 3.7 million customers and a revenue of 1.4 billion Euros which equals a mobile market share of about 21 per cent.
T-Mobile Netherlands operates 2G mostly at 1800 MHz, 3G at 900 and 2100 MHz and 4G/LTE at 900, 1800, 2100 and 2600 MHz. The company claims a 4G coverage of more than 99 per cent of the Dutch population. With carrier aggregation, T-Mobile’s 4G+ network offers a maximum speed of 262.5 MBit/s.
Vodafone Libertel B.V., the Dutch subsidiary of the international Vodafone Group acquired the formerly independent operator Libertel in 2003. Today, Vodafone is the second largest mobile operator in the Netherlands with around 5.2 million mobile customers corresponding to a mobile market share of about 29 per cent.
In February 2016, Vodafone Netherlands announced a merger with the cable and fibre operator Ziggo previously owned by Liberty Global. The company now operates under the name VodafoneZiggo. As regulatory conditions required Vodafone to separate its former fixed-line business, this entity then be-came „T-Mobile Thuis“. Vodafone‘s mobile revenues for the 2016/2017 fiscal year were still due at the time of writing.
The company operates 2G networks (GSM, GPRS, EDGE) at 900 and 1800 MHz, 3G (UMTS up to HSPA+) at 2100 MHz and 4G/LTE at 800, 1800 and 2600 MHz. Vodafone claims to offer more than 95 per cent coverage with 4G (up to 150 Mbit/s). In a number of Dutch regions the company offers „4G+“ with 225 Mbit/s.
The Koninklijke PTT Nederland N.V. emerged from the privatisation of the formerly state-owned PTT in 1998. With about 8 million mobile customers, KPN claims to have a market share of 44 per cent in the Netherlands at the end of 2016. With these numbers, the company is by far the largest Dutch mobile operator. For 2016, the company reported a total revenue of 6.8 billion Euros.The company focuses on marketing its KPN brand, however with Simyo, Telfort and Ortel it also has offerings in the „no-frills“ segment.
KPN operates 2G networks at 900 and 1800 MHz, 3G at 900 and 2100 MHz and 4G/LTE at 800, 1800 and 2600 MHz. The company claims a 99 per cent 4G availability all over the country. Where KPN already offers carrier aggregation, the theoretical maximum speed of its LTE network is 300 Mbit/s. KPN markets this configuration as „4G+“. Otherwise, the top LTE speed is at 150 Mbit/s.
The Swedish telecommunications operator Tele2 acquired the former Versatel N.V. in 2005. The resulting Tele2 Netherlands Holding N.V. is a 75 per cent subsidiary of its Swedish parent company. Originally acting as a MVNO (mobile virtual network operator), Tele2 nowadays operates its own infrastructure and has a portfolio of fixed telephony, data, internet and mobile telephony products. At the end of 2016 Tele2 reported about 1 million mobile subscribers which equals a Dutch market share of roughly 6 per cent.
Tele2 operates its own LTE network at 800 and 2600 MHz. After considerable efforts to increase coverage, Tele2 now provides a 4G network to large parts of the country, with some gaps especially in the west and south. The operator even offers carrier aggregation with up to 225 MBit/s at some locations. As the company had to pay lower license fees for its 4G spectrum than its competitors, it can offer aggressive prices. Tele2 cooperates with T-Mobile NL by location sharing as well as national roaming for voice services.
DATA ACCESS IN BIG CITIES
LTE has been designed to offer mobile data access with very high speeds and a very high reliability. Can the Dutch network operators fulfill this promise? In big cities, each of the four Dutch network operators claims to offer almost complete 4G coverage. In order to measure the quality and reliability of mobile data services, the four P3 teams visited a total of ten Dutch cities each of which has more than 100,000 inhabitants. But in densely populated areas a large number of cells with comparably small diameters is necessary to provide high speed to every user. The real-life test scenarios applied by P3 (see also page 8) therefore concentrated on accessing both live and static web pages as well as downloading and uploading files and watching YouTube videos.
The methodology of the P3 connect Mobile Benchmark is the result of more than 15 years of testing mobile networks, today in 65 countries. It was carefully designed to evaluate and objectively compare the performance and service quality of The Netherlands mobile networks from the users’ perspective.
The P3 connect Mobile Benchmark in the Netherlands took place from February 27 to March 6, 2017. All samples were collected between 8am and 10pm. The network tests covered larger cities, smaller towns and connecting roads. The combination of test areas had been carefully selected to provide a significant series of test results cove-ring the Dutch population. The areas chosen for the 2017 test account for more than 5.1 million people, or about 30 per cent of the total population.
P3 conducted the tests with four drive-test cars, equipped with arrays of twelve Samsung Galaxy S7 Cat 9 smartphones each. Eight phones were used for the measurement of voice services, and four additional S7 captured the values for the data services.
Two smartphones per operator in each car were used for the voice tests, setting up test calls from one car to another. The audio quality of the transmitted speech samples was evaluated using the HD-voice capable and ITU (International Telecommunication Union) standardized so-called POLQA wideband algorithm (Perceptual Objective Listening Quality Assessment). All Dutch network operators offer 4G capable subscriptions.
To take the high share of LTE into account, speech samples were acquired partly in 4G preferred to 3G preferred mode and partly in 4G preferred to 4G preferred mode. As a consequence, the phones in most cases needed to switch („fall back“) to 2G or 3G when they were logged in to the LTE (so called „circuit-switched fall back“ or CSFB).
In order to account for typical smartphone usage scenarios during the voice tests, background data traffic was generated in a controlled way through random injections of small amounts of HTTP traffic. The voice test scores account for 40 per cent of the total benchmark results.
Data performance was measured using four smartphones in each car – one per operator. The radio access technology was set to LTE preferred mode in order to reflect the customer experience. The web tests accessed ten web pages according to the widely recognized Alexa ranking. In addition, the artificial (static) „Kepler“ test web page as specified by ETSI (European Telecommunications Standards Institute) for testing purposes was used.
In order to test the data service performance, files of 3MB and 1MB for download and upload respectively were transferred from or to a test server located on the Internet. In addition, the peak data performance was tested in uplink and downlink directions by assessing the amount of data that was transferred within a 10 seconds time period. Another discipline was the playback of Youtube videos, taking into account that Youtube dynamically adapts the video resolution to the available bandwidth. So, in addition to the success ratios, start times and playouts without interruptions, the Youtube measurements also determined the average video resolution.
All the tests were conducted with the best-performing mobile plan available from each operator. Data scores account for 60 per cent of the total results. Routes and samples are shown on page 1 of this report. In the 17 big cities and 19 smaller towns indicated, the cars had to follow predefined routes. Altogether, the four test cars covered more than 7900 kilometres, of which approximately 3500 km led through the big cities, while 4400 km were covered in smaller towns and on connecting roads.
Performance indicators and rating
The score weighting reflects both the geographical distribution of the Dutch population and the ranking of usage scenarios. Therefore, 600 of 1000 maximum points achievable were assigned to the cities – divided into 240 points reflecting the voice results and 360 points reflecting the data results. For the towns and the roads, 200 points each are available. In both categories, those are divided into 80 points referring to the voice and 120 points referring to the data category. The charts on page 2 and page 10 of this report show the percentage of maximum points that each operator achieved in each discipline.
All four Dutch operators score well over 900 out of 1000 points. Overall winner T-Mobile even surpasses the 950 points threshold, achieving the rare grade „outstanding“.
The results of the P3 connect mobile benchmark in the Netherlands had already reached a very high level last year. In 2017, again, all four Dutch operators show a remarkably high level of performance and stability. In a year-on-year comparison, Tele2 was able to improve in all tested categories. Vodafone takes over the second rank from KPN, which mainly results from Vodafone‘s distinct improvement in the voice category. If the evaluation had considered only legacy voice connections, KPN would have ranked better.
When it comes to data connectivity in the cities, T-Mobile and KPN lead the field, with Tele2 and Vodafone trailing a little behind, but still scoring at a very high level. In the smaller Dutch towns, the lead of the stronger two operators T-Mobile and KPN is slightly more pronounced. On the connecting roads, the distance between all four candidates is so close that no operator is a bad choice.
When comparing the results of this year‘s mobile network benchmark in the Netherlands to those from recent tests in other European countries, Dutch operators are among the top tiers. Even the fourth-ranking Tele2 scores at a level that would make it a suitable candidate for a winning position in countries with overall weaker mobile networks like Spain or the UK.
As in 2016, the third largest Dutch operator also wins the 2017 benchmark with a lead in the voice and data categories and the overall grade „outstanding“. The fact that T-Mobile achieves the best result in the voice category without offering VoLTE shows that legacy voice services can also offer perfect calls, when they are handled well.
With a considerable improvement in its voice performance, Vodafone is able to outrank the market leader KPN in this year’s benchmark. In the data category, this operator still achieves a third place. However, the differences are small, and especially on the roads and to some extent in smaller towns, Vodafone holds up well with its competitors.
In a tight competitive environment, the largest Dutch operator has to give way to the smaller operators, that rank on second and third position in terms of subscriber numbers. However, KPN is beaten by Vodafone with a gap of only one point. But the Dutch market leader still achieves very good results both in the voice and data categories.
The Netherlands’ smallest operator scores last in this benchmark. However, Tele2’s results are still very competitive and deserve the overall grade „very good“. The loss of points in the voice category can mainly be explained by the disadvantageous effects of national roaming for legacy voice services on the T-Mobile network.
Objective testing will be essential to the ongoing evolution of the Dutch networks as to emerging technologies like autonomous driving and smart cities. P3 communications is preparing for these future challenges.
As in the years before, all four candidates scored really well in this year‘s mobile network test in the Netherlands. This, however, cannot be taken for granted. Upcoming improvements like the further extension of 4G coverage, the ongoing network evolution with technologies like „4G Advanced“ and the continuing rollout of Voice over LTE (VoLTE) may change this picture.
P3 communications is well prepared to accompany the future developments of the Dutch telecommunications market. In this context, we are already preparing to include new technologies and enhancements mentioned above in next year‘s testing. Furthermore, P3 is also preparing for future challenges starting with new communications applications like autonomous driving and smart cities (see below), through to completely new mobile communications technologies like the upcoming super-fast 5G networks. However, 5G‘s approach of assigning different network capabilities to different use cases will also require an extended approach to network testing.This will involve concepts like advanced analytics and big-data technology.
Extensions to the test routines
Readers interested in taking part in our continuous efforts to evaluate the performance of mobile networks can already do so today – by downloading the „U get“ wireless performance rating app. See details on the right.
This crowdsourcing approach will give us valuable additional insights into the user experience and operational performance of mobile networks in the near future.
Crowdsourced network rating
P3 communications is increasingly focussing on aspects like the retainability of voice services, the integrity of data services and „operational excellence“. An important instrument for this approach is the „U get“ app that is available under uget-app.com or via the adjoint QR code. The Android app checks and visualises current network performance. Join the global community of users who understand their personal wireless performance, while contributing to the world’s most comprehensive picture of mobile customer experience.
The future of transport is quickly coming upon us – one kilometre loop at a time. With each new generation of car models, the automotive industry comes closer to its vision of highly automated driving cars. However, to ensure that driverless cars can maintain connectivity and thus optimal performance and safety, we need to ensure that the technological infrastructure can manage the increasing demand that machines will place upon it. Therefore, autonomous driving scenarios play an important role in P3 communications‘ concepts for the evolution of mobile network testing.
Today, 54 per cent of the world’s population lives in urban areas, a proportion that is expected to increase to 66 per cent by 2050. Mobile communications will be an essential component to delivering on smart city promises. To enable smart cities to thrive and host successful businesses in the digital era, their technological infrastructure must be capable of managing the increased demand on network usage. Therefore, it will be a future focus point of P3 communications to determine whether cities become truly smart by taking an even closer look at their advances in connectivity.