Update time again! As far as our network goes, the status is the same from Sunday – we’ve isolated the malicious traffic
and our DNS servers are back to normal load.
We do know that a number of you appear to be having issues that seem similar to the issues over the weekend – no or very
slow internet browsing. If this is you, can you please unplug your router for a minute, then plug it back in, then
reboot all your devices.
If this has not solved the problem, please send us a private message stating that you’ve done this, and then tell us
your account or phone number and the type of device(s) you are having trouble with.
For those that are interested, we have also included a run down on the situation as we know it in the comments [below]. There have been some changes following investigation overnight.
--
Sorry again for the hassle over the weekend and for any trouble you’re having today.
What has happened?
Cyber criminals based overseas appear to have been attacking web addresses in Eastern Europe, and were bouncing the
traffic off Spark customer connections, in what is known as a distributed denial of service (DDoS) attack.
The DDoS attack was dynamic, predominantly taking the shape of an ‘amplified DNS attack’ which means an extremely high
number of connection requests – in the order of thousands per second - were being sent to a number of overseas web
addresses with the intention of overwhelming and crashing them. Each of these requests, as it passes through our
network, queries our DNS server before it passes on – so our servers were bearing the full brunt of the attack.
While the Spark network did not crash, we did experience extremely high traffic loads hitting our DNS servers which
meant many customers had either slow or at times no connectivity (as their requests were timing out). There were
multiple attacks, which were dynamic in nature. They began on Friday night, subsided, and then began again early
Saturday, continuing over the day. By early Sunday morning traffic levels were back to normal and have remained so
since. We did see the nature of the attack evolve over the period, possibly due to the cyber criminals monitoring our
response and modifying their attack to circumvent our mitigation measures – in a classic ‘whack a mole’ scenario.
How did they get access through the Spark Network?
Since the attacks began we have had people working 24/7 to identify the root causes, alongside working to get service
back to normal. During the attack, we observed that a small number of customer connections were involved in generating
the vast majority of the traffic. This was consistent with customers having malware on their devices and the timing
coincided with other DNS activity related to malware in other parts of the world.
However, while we’re not ruling out malware as a factor, we have also identified that cyber criminals have been
accessing vulnerable customer modems on our network. These modems have been identified as having “open DNS resolver”
functionality, which means they can be used to carry out internet requests for anyone on the internet. This makes it
easier for cyber criminals to ‘bounce’ an internet request off them (making it appear that the NZ modem was making the
request, whereas it actually originates from an overseas source). Most of these modems were not supplied by Spark and
tend to be older or lower-end modems.
What remains clear is that good end user security remains an important way to combat these attacks. With the
proliferation of devices in households, that means both the security within your device and the security of your modem.
What did Spark do?
We have now disconnected those modems from our network and are contacting all the affected customers. We have also taken
steps at a network level to mitigate this modem vulnerability. We are now in the process of scanning our entire
broadband customer base to identify any other customers who may be using modems with similar vulnerabilities and will be
contacting those identified customers in due course to advise them on what they should do.
With respect to malware we continue to strongly encourage our customers to keep their internet device security up to
date, conduct regular scans and regularly update the operating software and firmware on their home network. We also
continue to advise customers not to click on suspicious links or download files when they are not sure of the contents.
We have also taken steps at the network level to make it more difficult for cyber criminals to exploit the DNS open
resolver modem vulnerability and we’re using the latest technology to strengthen our network monitoring and management
capabilities. For security reasons we can’t detail these steps, however this is an ongoing battle to stay one step ahead
of cyber criminals who are continually using more and more sophisticated tactics.
Why only Spark?
We can’t say what other networks experienced. However, it’s typical that cyber criminals look for clusters of IP
addresses to use in any particular denial of service attack. That makes it more likely that these IP addresses belong to
the customers of a single ISP – even more likely with a large ISP like Spark. They do this because it’s then easier for
them to monitor the steps the ISP is taking to mitigate the attack and change their tactics accordingly. We definitely
saw this happening over the weekend.