2016|12|12 – THE P3 COMMSDAY MOBILE BENCHMARK IN AUSTRALIA
Download the report as PDF here
Welcome to the 2016 P3 CommsDay Mobile Benchmark: an in-depth objective comparison of the user experience for voice and data on Australia’s three mobile networks, operated by Vodafone, Optus and Telstra.
In 2016, the third year that we’ve run the benchmark, mobile network performance is of increasing interest to more and more Australians.
Just a few weeks ago, Deloitte’s Mobile Consumer Survey found that 84% of the country’s population now own a smartphone – a 5% increase through the last year, and ahead of this year’s global average of 81%. For the three months to June this year, the Australian Bureau of Statistics found that mobile handset users downloaded an average of 1.8GB of data per user per month.
And earlier in the year, CommsDay’s own analysis showed that Australia’s mobile carriers had one of the highest ratios of 4G sub-scribers to total mobile base in the world, indicating that Australians are particularly hungry for the latest and fastest mobile technology.
More than ever, customers are demanding the best possible performance not just from the latest smartphones but from their mobile networks. Vodafone, Telstra and Optus are continuing to spend billions of dollars on upgrading and expanding their networks to meet those demands. This benchmark gives an idea of how that investment is translating into an improved experience in practice for smartphone users.
In this report, you’ll find a detailed description of how we ran the tests for this year’s benchmark and how we’ve updated our methodology to capture the end-user experience more accurately than ever. You’ll find an in-depth look at performance on voice and data in large cities, smaller cities and towns and connecting roads as well as the all-important master score table, showing how each of the three operators compares to the others on key metrics. And there’s a comparison of network performance for each operator in 2014, 2015 and 2016, showing you just what’s changed in the last year.
Finally, if you’d really like to drill down into the details, you’ll find a complete break-down of the raw test data, drawn from a huge set of around 195,000 individual samples taken over weeks of scientific testing – significantly more extensive even than last year’s 150,000 measurements.
METHODOLOGY: UPDATING THE BENCHMARK FOR 2016
First published in 2014, the annual P3 CommsDay Mobile Benchmark was intended to provide a unique resource for consumers: a totally independent and impartial comparison of Australia’s three mobile networks, unprecedented in the scope, detail, and scientific rigour of the testing that underpinned it.
We launched the benchmark because smartphones are now an essential part of life for many Australians, and it should be as easy to compare mobile networks as it is to read reviews of new phones – after all, the latest and most advanced handsets need to be supported by equally high quality mobile networks.
Since launch, the benchmark has become an established annual event that shows how the experience on each network is changing over time. It has received widespread coverage from the press and consumer organisations, and is now keenly anticipated by the operators themselves.
We work closely each year with Optus, Telstra and Vodafone on the design of the tests and the results provide them with an objective view of their respective strengths, the effects of all their network investment, and areas for improvement.
As with previous years, 2016 has seen all three operators make significant investments in improving the reach and performance of their mobile networks, not least in terms of mobile spectrum. Radiofrequency spectrum is the specific part of the electro-magnetic frequency range that is used to broadcast signals for mobile phones, along with other things such as radio and TV. Operators pay significant licence fees to the Australian government for exclusive use of certain frequencies, ensuring nothing else can interfere with their mobile phone signals. It’s a key resource for operators; generally speaking, more spectrum means more bandwidth for data and faster mobile internet, and certain frequencies have other advantages as well.
This year, both Telstra and Optus have continued to increase the number of locations that use their 700MHz mobile spectrum – powering what Telstra calls its ‘4GX’ service and Optus refers to as ‘4G Plus’, both promising faster internet and an improved experience for users with compatible devices. They spent almost two billion dollars between them for the 700MHz licences in a 2013 auction, with Telstra buying twice as much as Optus; 700MHz is a relatively low frequency, and low frequency spectrum is important because it can transmit better over long distance and penetrate walls more effectively, providing better indoor service.
While Vodafone didn’t buy any spectrum in the 2013 auction, 2016 saw it finish a crucial ‘refarming’ process for its existing low-band spectrum. Refarming is a well-established process by which operators shift more of their spectrum to newer mobile access technologies; in March, Vodafone completed the move of its entire 2x10MHz holding in the 850MHz band to 4G, double the amount originally planned, again giving it stronger indoor penetration and range. It has also started to refarm 2100MHz spectrum from 3G to 4G in some areas.
2016 has seen an extra boost to regional mobile network coverage thanks to the Australian government’s Mobile Black Spot Program. This initiative matches Commonwealth, state and local government funding with specific investments from participating operators to establish mobile coverage and competition in previously underserved regional and areas. This is building 499 new and up-graded mobile base stations around the country. When completed, these are intended to provide new handheld coverage to 68,600 square kilometres, and dozens have al-ready been finished and activated.
The operators have also announced plenty of other network investments and up-grades through the year. Optus, for instance, has been targeting better coverage at major sports stadiums, trialing a centralised radio access network to better manage interference. And Telstra is prioritising A$250 million of its capital program to boost net-work resilience and performance; A$50 million of that specifically on its mobile net-work, and A$100 million on its core network.
As in previous years, we’ve updated our methodology for 2016 to capture as many of the latest network developments as possible. In particular, we’ve brought in a new walk test, using proven P3 technology to facilitate measurements indoors at shopping malls, on public transport, and at major tourist hotspots – some of the use cases where low-band spectrum becomes really important.
SCALE AND SCOPE
Australia’s sheer size, geography, and population distribution creates a major challenge in measuring network quality around the country.
Its current population of over 24 million people is distributed around a landmass of 7.7 million square kilometres, with most people concentrated in the large cities on the east, west and south coastlines but a significant proportion living in more remote areas. It’s also worth noting that the operators have focused to differing extents on providing coverage to regional Australia; Telstra, for example, claims the broadest network coverage and the most coverage of regional areas.
To provide the most representative results possible, however, our drive test vehicles covered a massive 17,000 test kilometres or so across several weeks of dedicated testing. While it’s simply not possible to test across the entire country, the routes and test areas were carefully selected to cover a significant share of the Australian population across different types of coverage areas – urban, suburban and regional, as well as connecting routes – to enable conclusions to be drawn on the overall state of the market.
We tested in Busselton, Bunbury, Perth, Brisbane, Tweed Heads, Canberra, Byron Bay, Gee-long, Coffs Harbour, Port Macquarie, Newcastle, Sydney, Wollongong, Nowra, Queanbeyan, Albury, Melbourne, Warrnambool, Ballarat and Adelaide – nine cities with a population of over 100,000, as well as eleven towns and smaller cities, and connecting roads running through more remote regional areas.
Our new walk tests also opened up a new level of granular and detailed testing in the five largest cities, giving us deep insight into network performance in central business districts, selected tourist hotspots and even major shopping malls in Sydney, Melbourne, Brisbane, Adelaide and Perth.
Altogether, the areas selected for the 2016 test accounted for well over 14 million people – more than 60% of the Australian population.
THE DRIVE TESTS
As usual, for our drive testing, we used paired test vehicles equipped with P3’s measurement hardware to collect voice and data samples; they tested 3G to 4G and 4G to 4G voice calling, as well as data transfer in 4G-preferred mode. The paired cars took the same overall routes around the country in parallel but were never in the exact same place at the same time, to avoid the risk of one distorting the measurements of the other; they also took different routes around cities to maximise the area covered.
One big change for 2016 was that we actually deployed two pairs of test vehicles, both starting in Sydney but then taking separate routes around the country. This enabled us to take an even more comprehensive volume of test samples than in previous years, while reducing overall measurement time – and thus also reducing any disruption caused as operators made sure their networks were in peak working condition for the tests. Using some of the new and compact testing technology that P3 developed for last year’s benchmark, we were again able to use a ‘fly-in, fly-out’ approach to drive testing. Our test personnel could store their equipment in a small suitcase, fly it into any location with an airport, install it into a fresh pair of hire cars within a few minutes and immediately begin testing. This meant that we could easily test in locations such as Perth, which would have been hard to reach over land.
THE WALK TESTS
New for 2016, we introduced walk tests into the benchmark. Just as with our drive tests, these included both 3G to 4G and 4G to 4G voice calling, as well as data measurement in 4G-preferred mode. But where our drive voice tests ran voice calls between our paired measurement vehicles, our walk testers in Sydney, Melbourne, Adelaide, Brisbane and Perth were calling a second party located in Sydney, in areas with strong signal coverage from all three operators.
The big advantage of walk testing was that it allowed us to gather comprehensive measurements in locations like shopping malls (for example Sydney’s Pitt St Mall, Westfields Marion in Adelaide or the Galleria shopping centre in Perth), central business districts, and tourist hotspots like Melbourne’s Southbank or The Rocks in Sydney. We were even able to test on public transport, such as buses or metro train lines, and used several public transport options in each city.
In addition, walk testing allowed us to specifically test indoor coverage, an important part of smartphone enduser experience. That in turn allowed us to phase out attenuation in our benchmark measurements.
Attenuation is a reduction in the strength of mobile phone signals; in the real world, it’s caused by factors such as the signal passing through walls or other obstacles. In previous years, we’d set up artificial attenuation to reflect these real-world conditions – first by using rather bulky hardware and arrays of rooftop antennas for our drive test cars, and later by using P3’s own compact Antenuatr technology.
However, the notion of using additional attenuation has sometimes been contentious in some of P3’s key markets, as well as introducing extra complexity into the testing process; walk testing solves this problem by making it possible to measure indoor performance in a dedicated way.
For 2016, in both drive and walk tests, we used arrays of Samsung Galaxy S5 Cat 4 smartphones for voice testing and Samsung Galaxy S7 Cat 9 smartphones for data measurements.
We used commercially available firmware for the handsets, corresponding to the original network operators’ branded versions, and the most comprehensive mobile tariff plan available from each operator. Telstra, Optus and Vodafone provided most of the SIM cards for the test, and made sure that fair use policies – sometimes used to limit download speeds for high-usage customers – would not interfere with testing. Our staff also anonymously purchased SIM cards for some of the testing as a control, ensuring the supplied SIMs matched those available in stores.
Just as with last year, one smartphone per operator in all drive testing (and, for 2016, walk testing as well) was dedicated to data measurements – and set to 4G-preferred mode to represent the typical Australian enduser experience. Telstra, Vodafone and Optus all still maintain 3G networks, but 4G is a very well-established technology now; operator investments in both radio spectrum and infrastructure are mainly focused on 4G, and almost all smartphones available in stores are 4G compatible.
However, smartphones will still fall back to the older 3G networks in areas where 4G is not available; since we tested in smaller towns and connecting roads as well as in major metropolitan areas, this was naturally reflected in our measurements.
Also similar to previous years, we ran a battery of different data tests: accessing some of the most popular web pages as well as static reference pages, downloading 3MB files and uploading 1MB files to simulate the transfer of photos, smartphone apps and other day to day data loads, and playing YouTube videos. We also repeated our ‘peak tests’ on both up-load and download, pushing as much data as possible across the network across a continuous 10-second window and measuring average throughput; this really demonstrated the maximum speeds possible on each network.
The most important factors of data performance for endusers are still stability and reliability. We all appreciate high speeds but, if emails fail to send or web pages time out without loading, we become frustrated very quickly. Therefore, as with the 2014 and 2015 benchmarks, our data scoring put a lot of emphasis on success ratios: that proportion of web pages that loaded completely, file up- and downloads that finished without a hitch, and so on. Other metrics included the average time taken to transfer files, startup times for YouTube videos, and of course the average throughput for peak testing. This year, for video tests, we also used YouTube’s adaptive bitrate feature to adapt the resolution to available network conditions – and measured the resulting resolution for all test samples.
Voice calling might be amongst the most basic features of a modern smartphone, but endusers tend to have quite demanding expectations of their voice service. They want to be able to make calls from anywhere, whether mobile or stationary, without drop-outs or interruptions – and often while they’re downloading emails or other data in the background at the same time.
Therefore, we set fairly challenging voice test parameters. Reliability is the single most important aspect of a voice call, with endusers having a very low tolerance for calls that drop out or fail to connect. So the heaviest score weighting was on call success ratios – that percentage of all voice call tests that were set up and completed successful-ly, with a minimum speech quality throughout. Other metrics included the time taken to set up the call, and speech quality – scored with the Perceptual Objective Listening Quality Analysis Wideband Algorithm, a standardised five-point quality scale.
All voice testing was run alongside continuous background data traffic to reflect a typical environment for today’s smartphone user. While you’re making a voice call, applications on your phone will often be sending data to and from the internet in the background, whether it’s from social media, instant messaging platforms or email clients. We simulated this in a controlled way with the random injection of small amounts of HTTP background traffic on our test phones.
As with the previous year, our voice testing needed to allow for the effects of a technology called Circuit Switched Fallback (CSFB). Because the 4G standard was not originally designed to natively support voice calling, making a call on a phone in 4G-preferred mode will actually result in a switch back to the 3G network to set up the connection. But CSFB has some side-effects; the switchover can result in paging problems and call setup delays, and if users are downloading data while making the call, their data transfer rate will drop to 3G speeds until the call is finished.
To account for this, we used two voice channels for all our testing. The first channel had the smartphones at both end of the call set to 4G-preferred; the second had one phone set to 3G-preferred and the other to 4G-preferred. The test results were drawn from a combination of calls across the various configurations, with the test scores reflecting aggregate performance.
New for 2016, we also ran a separate case study on Voice over LTE – a protocol which natively supports voice on 4G, and thus promises to tackle the issues around CSFB – outside the scope of the main benchmark.
CONSULTATION AND INDEPENDENCE
For 2016, as with previous benchmarks, we spent months consulting with Vodafone, Telstra and Optus before the start of testing, starting back in January – and kept communications open during the test period itself.
The operators know their own networks better than anybody, and it was critically important that we be as transparent as possible in designing the benchmark parameters and selecting the latest mobile technology to test. We did not disclose the exact test routes to the operators, and in fact the final routes were chosen by random lottery from a number of alternatives immediately before testing; this avoided any possibility of operators bringing additional capacity online in specific areas to influence the results.
However, we provided all three operators with an overview of the scope and timeline – giving each of them the opportunity to factor the data into their own network planning processes and, for instance, to avoid scheduling any planned outages (for network upgrades, additional mobile basestation deployments and the like).
Maintaining communications throughout the testing ensured that operators could provide any relevant information about unplanned outages should they happen to coincide with the test period. In such cases, P3 is able to evaluate the impact on measurement results and potentially consider the effects of these outages on the final results.
In other countries where P3 mobile network benchmarks are well established, P3 has observed that a side-effect of frequent network-testing is that it helps motivate operators to invest in constant performance improvements – meaning that customers eventually benefit.
This year’s voice scores were strong overall, with all operators very close to each other.
On the heavily weighted call success ratio – a key measurement of reliability and very important for enduser experience – all operators performed very well. Vodafone pulled ahead in drive testing in major metropolitan areas at 99.2%, almost an entire percent-age point ahead of Optus which was closely followed by Telstra. Major metro walk testing had Vodafone and Optus tied at 99.3% with Telstra just behind at 99%. Vodafone had even better success ratios in drive testing around smaller cities and towns, at 99.7% to Optus’ 99.3% and Telstra’s 99.1%.
However, with Telstra’s network covering the largest area of the country, it dominat-ed call success rates in the more regional areas represented by drive testing on connect-ing roads. Here, it scored a 95.7% call success ratio, against 93.6% for Optus and 86% for Vodafone.
Optus and Vodafone shared the laurels for fastest call setup times; both managed 5.5 seconds on average in metro drive testing, with Vodafone faster at 5.4 seconds in metro walk testing to Optus’ 5.6 seconds, but Optus averaging a very quick 4.8 seconds in drive testing in towns and smaller cities to Vodafone’s 5.8 seconds. Optus also did best for setup times on connecting roads, at 5 seconds on average to Vodafone’s 5.8.
Telstra was fairly consistent across the board, at 6.2 seconds on average for metro drive testing, 6 seconds for metro walk testing, 6.3 seconds towns and smaller cities and 6.3 seconds on connecting routes. Average speech quality was consistently high across all three operators, and markedly improved on last year’s results. All three scored 3.8 in all metro testing and 3.7 in drive tests along connecting roads; Telstra scored 3.7 in drive tests in towns and smaller cities, where Optus and Vodafone scored 3.8 apiece.
Ultimately, Optus’ consistently strong performance across major metropolitan areas, smaller cities and regional connecting routes gave it the highest overall score in the voice testing.
MOBILE DATA: MAJOR CITIES
Much of Australia’s population is concentrated in its major metropolitan areas, putting the heaviest load on mobile networks in the big cities; these areas are, therefore, a major investment focus for all three operators.
All the networks were shown to be extremely reliable in metro areas in both drive and walk testing; again, reliability is critical for a good enduser experience. Vodafone achieved success ratios of 99% or over in all metro data testing, with a perfect 100% success rate for small file downloads in metro drive tests; Telstra managed 99% or over in all major metro testing apart from small file uploads in walk tests, and hit a perfect 100% ratio in static webpage downloads in metro drive testing. Optus achieved 99% success ratios or higher in all metro data testing except live web page downloads in drive testing and small file uploads in walk testing, where it was slightly lower.
A lot of network investments and technology upgrades are dedicated to increasing mobile data speeds; even averaged out between all users sharing a network, higher speeds mean that everyday activities like downloading emails and browsing image-rich web pages go more smoothly, significantly enhancing the enduser experience.
The 2016 test results showed some huge speed improvements across the board and some key highlights for individual operators. Nowhere was this more apparent than in peak testing. Telstra hit a staggering 59.87Mbps average throughput on the download and 21.84Mbps upload in major metro drive testing, and 73.36Mbps download/23.34Mbps upload in major metro walk testing. Optus hit 56.99Mbps download and 16.02Mbps upload in major metro drive testing and 62.71Mbps download/19.14Mbps upload in metro walk testing. Vodafone peaked at 55.33Mbps average download in major metro drive testing but came in highest with 23.44Mbps on the upload; in metro walk testing it hit peaks of 52.98Mbps average download and again scored highest with 23.56Mbps upload.
Telstra and Vodafone split the highest scores between them on speed consistency indicators in major metro testing. Telstra, for example, had 90% of samples faster than 12.06MBps on small file downloads and faster than 4.9Mbps on small file uploads in metro drive testing, showing the strongest consistent speeds in these categories; Vodafone, meanwhile, had 90% of small file upload samples faster than 5.21Mbps and 90% of peak file upload samples faster than 6.44Mbps in metro walk testing, coming out ahead in those categories. On the other hand, Vodafone did better in many major metro categories at the top end of the samples; it had the highest speeds for the top 10% of samples for small file downloads and uploads, as well as peak upload testing, in both metro walk and drive tests. Telstra scored highest for the top 10% of samples in both metro walk and drive tests, however.
YouTube video performance was very good for all operators in all metro testing, with quick start times of around two seconds in all tests (Optus being the fastest at 1.8 seconds), and 99.9% or better video playouts without interruptions. Telstra achieved the highest video resolution in metro drive testing, tying with Vodafone in metro walk testing, but all operators managed 629p resolution or higher on average.
MOBILE DATA: SMALLER CITIES AND TOWNS
Here, with measurements taken solely through drive testing, the data scores separated somewhat. In terms of reliability, Telstra had the most consistently excellent success ratios, above 99% for all tests and hitting a perfect 100% for small file downloads, peak test uploads and YouTube playback. Optus did well, with a 100% success ratio of its own for small file uploads, but dipped substantially on live webpage downloads and YouTube videos with 93.4% and 93.8% success ratios respectively. Vodafone, similarly, had generally good success ratios but dropped to 96.2% on live web page down-loads, 97.2% on small file uploads, and 95.7% on YouTube videos.
Telstra also racked up the highest speeds in peak throughput testing in smaller city and town drive testing, with 45.91Mbps average download and 17.16 average upload. It also did the best on speed consistency, with the highest speeds in the top 90% of samples in all categories. However, Optus scored some wins at the high end; it had the highest speed for the top 10% of samples in small file downloads, small file uploads and peak throughput upload testing.
Telstra dominated YouTube testing in smaller cities and towns, with a perfect 100% video playouts without interruptions, the quickest start times, and the highest adaptive video resolution of 652p.
MOBILE DATA: CONNECTING ROADS
Again, with the largest network coverage area in Australia, Telstra performed very well in the more regional areas represented by drive testing on connecting roads and highways. It had the best reliability here, with the highest success ratios in all test categories apart from peak throughput file downloads, where Optus edged it out with the top score of 97.4%.
Telstra was also comfortably ahead on peak throughput speeds in these regional areas, with 40.1Mbps average throughput on the download and 14.73Mbps average throughput on the upload.
It should be noted, though, that even Vodafone’s third place peak download throughput of 20.35Mbps represented almost double its equivalent speed in 2015! And while Telstra also performed well in terms of consistent speeds, with the highest rates in 90% of samples in all but one cate-gory, Optus again pulled away at the high end in some tests; it had the highest speeds for the top 10% of samples in small file downloads, small file uploads and peak download throughput categories.
Optus also managed the best ratio of YouTube video playouts without interruption, at 99.5%, and tied with Telstra on setup time at 2.1 seconds – although Telstra had a higher overall success ratio and a considerably higher average video resolution, at 625p.