2016│11│29 – The Mobile Network Test in Germany, Austria and Switzerland
Download the report as PDF here
At the end of 2016, we conducted our annual mobile network benchmark for the 23rd time. Together with our renowned benchmarking partner P3 communications, once again we have investigated, which mobile operators in Germany, Austria and Switzerland are a cut above the rest – with utmost effort and our objective, customer-oriented testing methodology.
The network operators in Germany, Austria and Switzerland wait for the results of our annual mobile network benchmark with the highest tension imaginable. As in the years before, we have conducted this benchmark as part of our well-proven cooperation with the Aachen-based network testing specialist P3 communications.
However, we have never before experienced such harsh disputes with some of the test candidates in the forefront of our test. They argued about questions like which smartphone models were to be used for the measurements, how the test routes should be put together or how we should balance the individual voice and data results. Eventually, all their reasoning could be traced back to one purpose: Some of the candidates hoped to gain advantages in those areas where they believed to be particularly strong and having the edge over their competitors.
We did show good sportsmanship interpreting the pressure built up by some of the candidates in the run-up of our test as proof for the high relevance and acceptance of our benchmarks within the whole industry.
Quality benchmark for many years
Traditionally, fairness and transparency play a very big role in our network test. In any case, with all decisions concerning methodology and scoring, our highest priority is always to ensure the significance of our benchmark for you, our readers.
As a matter of principle, connect and P3 value objective and authoritative results about the actual quality and performance of the cellular networks second to none. Everybody interested can find an exact description of our methodology in chapter “Methodology”.
So, on the following pages you can read the answers to many thrilling questions: Has Deutsche Telekom been able to defend its top rank in Germany? How does O2 rank, after its owner Telefónica has started to combine this network‘s former radio cells with those of E-Plus, which they bought in 2014 (also see page 15). Which operators come out on top in Austria and Switzerland, where the contest is traditionally conducted on an especially high performance level?
Not all contestants will like all of the answers and results. This again proves to us that we have done a good job. Because this is the only way for us to guarantee that truly everybody can fully rely on our test results.
The quality and reliability of voice connections represent 40 per cent of the final score. Which operator offers the best network in this respect?
In some users‘ communications habits, voice telephony only plays a minor role. But conventional phone calls are far from being outdated. Otherwise, the three German network operators would probably not have gone to the lengths of implementing VoLTE – telephony based on sending data packets over the LTE network.
Therefore, quality and performance of voice telephony still played an important part in the drive tests and walk tests conducted by P3: For this purpose, both cars that were driving through 17 large and many smaller German cities carried six Samsung smartphones each. They permanently called their counterparts in the other vehicle. In order to simulate the everyday smartphone use, the phones would constantly transfer data in the background during the telephony tests.
An identical device configuration was used in the backpacks carried by the test staff who walked around in city centres and public buildings conducting the walk tests.
The devices had been configured to make sure that part of the connections would be established via VoLTE and another part would be transmitted via conventional circuit-switched telephony.
Distinct ranking order
The tests in the city centres already showed a clear picture: Both in the drive tests and in the walk tests, Deutsche Telekom is ahead. Vodafone follows at a distance of a few points. O2 is clearly defeated what can be seen in the table below by means of lower success ratios, longer call setup times and also a lower average speech quality.
As a result of their test drives through smaller cities and on connecting roads, the P3 testing team found basically the same ranking order: Deutsche Telekom leads, Vodafone follows at a comparatively small distance, and O2 comes in last with a considerably larger gap.
While Telefónicas network keeps up quite well in smaller towns, its distance to the leading two contenders grows larger on the connecting roads. On the whole, compared to last year‘s test, O2 improved in the voice category. For the voice calls examined by the testing staff in trains, even Telekom and Vodafone showed some weaknesses. But O2 scores again far behind them.
Regarding some measurement values like call setup times and speech quality, Vodafone is narrowly ahead. But while the Düsseldorf-based operator could claim a stage win in the voice category last year, in the 2017 test this title goes to Telekom.
Data tests account for 60 per cent of the final score. Who delivers the best performance in this category?
Data communication is the most prestigious category in connect‘s network test. Firstly, the results of these tests represent 60 per cent of the final score. And secondly, the test parcours to be completed by our candidates incorporates a large number of practice-oriented applications. For instance, the smartphones frequently access the most popular web sites according to the renowned Alexa ranking, as well as the static „ETSI reference web page“ also known as „Kepler page“. Measuring the speed and reliabilty of data transfers is the aim of our upload and download tests. We monitor uploads with test files sized 3 MB and downloads with 1 MB files. Additionally, we verify which amount of data travels over the network within ten seconds.
Another scope of our testing are Youtube videos. The popular video platform does not distinguish between standard definition (SD) and high definition (HD) resolutions any more. It rather dynamically adapts the video resolution to the bandwidth that is currently available. In order to respect this new strategy in our tests, we examined the success ratio of video playbacks, the start times, the percentage of playouts that took place without interruptions as well as the videos‘ average resolution or number of lines respectively. Both P3 test cars checked these indicators as part of their drivetest, and also the walk test teams had the same agenda. All data measurements in Germany were executed with the LTE Cat.6 smartphone Samsung Galaxy Note 4.
Strong Deutsche Telekom
In big cities, the results were similar to the voice category: Again, a very strong Telekom takes the lead, Vodafone follows with good results, and O2 is clearly defeated. Separate analyses show that Vodafone could definitely improve over last year‘s results in the data category. To some extent, this supports the Düsseldorf operator counterbalancing its shortfall in the data score. Also, Vodafone achieves especially good results at Youtube playback – in this category, the Düsseldorfers partly draw level with Telekom. Still, this is not sufficient to grant Vodafone a partial victory. The reason is that Vodafone scores a little worse than Telekom in the discipline of web page access and – slightly less distinctive – regarding file uploads and downloads.
However, when we look at O2, their deficits are even more obvious. While the inner city walk tests show a success ratio of 99.3 per cent for web page access in the Telekom network and 98.5 percent at
Vodafone, this value drops to 91.3 per cent in Telefónica‘s network. Statistically, this means that almost one out of ten web page views will fail when O2 customers are walking through city centres.
When observing indicators like the success ratios or average speeds of file downloads in the big city drive and walk tests, the results show the same trend that looks increasingly familiar this year: Telekom leads, Vodafone follows at not too big a distance, and O2 clearly comes in last.
For example, according to our walk tests, file downloads run at average data rates of more than 13578 kbps in 90 per cent of the cases in the Telekom network. In the Vodafone network, it is still more than 6801 kps, and O2 achieves only a minimum of 1568 kbps. So the latter candidate accomplishes not much more than a tenth of the speed offered by test winner Telekom.
Drive tests in smaller towns
The drive tests conducted in small towns gave equivalent results: Again, Telekom achieves the best measurement values, Vodafone follows at a distinct but not huge distance – and Telefónica once again comes in last.
Similar to the inner city drive tests, the success ratios of web page access via the O2 network are worse than in the networks of the leading two providers, but still better than O2‘s walk test results. Our download and upload tests show comparable results.
Once more, Vodafone turns out to be a Youtube star in smaller towns. Its top performance in this category is at the same level as test winner Telekom. Both operators seem to have a very performant “peering“ to the content delivery network of Google‘s video platform.
O2‘s weak overall scores can be explained to some extent with the ongoing integration of the former E-Plus network and the distortions coming along with it. While O2 at least improved in the voice category, the 2017 network test indicates a stagnation for this operator in the data category.
Modern cars rely heavily on connectivity. How does this actually work out on German roads?
It was approximately 6600 kilometres that P3‘s two test vehicles covered this year on German connecting roads – on top of the 5500 kilometres that each car covered driving through large and smaller cities. The point of this exercise: Gaining closer insights about the quality and reliability of the mobile networks on this particular type of roads.
Distinct ranking order on the roads
While Telekom and Vodafone were almost at level in the voice tests conducted on connecting roads, their offset becomes more obvious in the data category. Especially regarding the success ratios of web page access as well as downloads and uploads,
Telekom clearly ranks first and keeps its competitor from Düsseldorf at a distance. Still, this match takes place at a very high level when looking at the distance of O2.
Similar to the large and small cities before, Telefónica also loses valuable points on the connecting roads and thus falls back further behind the leading two German operators.
The test results in this category are quite obvious: Car drivers who need robust data connections on the road – whether for navigation, for communication or mobile entertainment purposes – currently cannot pass the mobile networks of Deutsche Telekom or Vodafone.
DATA ON RAILWAYS
Railways used to be the blind spot of German mobile network operators. Is this still true this year?
The testing staff spent about 33 hours on 15 different ICE trains during their railway tests conducted in 2016. Yet, the measurements did not only take place in these flagship trains of Deutsche Bahn, but also considered regional railway connections.
The test results from German trains should not surprise anybody who has read the outcome of the other categories: In the trains, Deutsche Telekom once again leads the pack, Vodafone scores second with viable results, and O2 brings up the rear.
Especially when comparing the partial results obtained on the railways, looking over the borders – specifically at the results of the respective tests in Austria and Switzerland – may make German railway customers quite envious. Both alpine countries are considerably ahead in this respect.
And even Telekom, which scores best in the measurements taken in German railways, cannot measure up with the results of the Austrian and Swiss operators – by far. Deutsche Bahn has recently started an initiative to enhance connectivity especially in its ICE trains in close cooperation with the German mobile network operators. But this does not seem to have much impact on this year‘s mobile network test.
Much need for improvement
When looking at the details, there are many similarities between the railways and the connecting roads: While Telekom and Vodafone were almost at level regarding voice phone calls in trains, their offset grows in the data category. Here, Telekom offers the best results – but still shows a lot of room for improvement. Vodafone‘s results are mid-level, and O2 once again comes in last. Success ratios, like those for web surfing, of around 85 per cent at Telekom, about 77 per cent at Vodafone and approximately 63 per cent at O2 convey a clear message: When it comes to connectivity in German railways, there remains a lot of work to be done.
Telekom wins this year‘s network test at a distinct distance to runner-up Vodafone. Both in the voice and data categories, the Bonn-based operator turns out to be the strongest conveyor. Although we have raised our requirements, last year‘s winner Telekom not only managed to defend its position, but actually continued to improve its score. Therefore Telekom absolutely deserves the first place in Germany – and this for the sixth time in a row.
Vodafone also improved both in the voice and data categories compared to last year‘s results. But still Telekom outperformed their Düsseldorf-based competitor in this year‘s voice tests. Anyway, the 2017 network test emphasizes Vodafone‘s clear improvements in the data scores. In this context, Vodafone‘s excellent Youtube results are particularly eye-catching. All in all, these results entail a second rank with a good overall score.
Compared to the results of last year, both the networks of O2 and E-Plus managed to improve. This is particularly underlined by the results of the voice measurements. A valid explanation for O2 not scoring any better may be the problems caused by the ongoing integration of both networks. So we hope in the best interest of O2‘s customers that this integration will continue to foster noticeable improvements.