By Pattrick Smellie
Sept. 23 (BusinessDesk) - It had all been going so well.
Friday night's Rugby World Cup opening ceremony and first match, Japan vs Russia, rolled down the international
telecommunications cables to New Zealand so seamlessly for Spark Sport that freshly departed Spark chief executive Simon
Moutter couldn't contain himself.
Tweeting at just after midnight, he had a crack at the media for being "clearly disappointed that Spark Sport delivered
superbly tonight" and was still trying to "find a few customers who had issues."
At the coalface were some 200 staff on the Spark Sport help team - expanded from the usual dozen or so - fielding the
predictable flood of calls from customers with last minute trouble setting up their livestreams at home.
A team of senior Spark executives, including CEO Jolie Hodson, had "eyes on glass" in a situation room, monitoring with
technical teams a bewildering number of devices from high-end TVs to tablets and smartphones, taking the feed from
Tokyo.
"We were rapt with how the platform performed" on Friday, says corporate affairs manager Andrew Pirie. Of course, the
Spark team had "kick-arse" connectivity, unlike the many customers who were just discovering that fibre to their front
porch won't make the new TV in the lounge play the rugger if the two are connected by a weak wi-fi signal.
"Hundreds of thousands of New Zealanders watch Netflix and Lightbox every night," and streaming TV is second nature to
them, says Pirie."But there's hundreds of thousands who have never experienced Netflix and who need to get into the
world of streaming to watch their beloved rugby."
Spark's willingness to cop flak for forcing irritated punters to buy a Spark Sport subscription and risk the kind of
failure that occurred during the All Blacks vs Springboks match on Saturday evening boils down to this simple calculus:
having all those newbies switched on to livestreaming will be good for business in the long run.
By 6pm Saturday, customers had logged 10,000 set-up queries just that day by phone, online chat and social media. Three
hours before kick-off in Yokohama stadium, new subscriptions were still selling at a rate of 90 a minute. By
mid-evening, the Spark executives were daring to hope that the first really big game of the tournament for Kiwi fans
would go off without a hitch.
The Australia vs Fiji game had gone well, likewise Argentina vs France.
"There was pressure on the customer care channels, but from a technical point of view, we were prepared for that level
of capacity. The audience was building nicely," says Pirie.
But Spark's kick-arse connection at HQ also meant Hodson, Pirie and the other senior executives in the room could see
immediately when the picture began to play up, early in the first half of the All Blacks/Springboks clash.
"About 10 minutes in, there was a bit of a blip in the bit rates," he said. A few anxious moments later, the feed seemed
to come right.
"Five minutes later, it happened again."
There was concern but again, things seemed to come right.
"Then it happened again," says Pirie, and everyone knew they had to take action. If they were seeing a few blips on
their excellent connection, sure as eggs there would be customers seeing far worse effects.
"People were starting to huddle around the screens and we were getting onto our content partners asking what was
happening."
Watching the game at home and with no apparent problems was Spark's general counsel, Melissa Anastasiou, who was
surprised to get a call saying she needed to come into the office.
However, the call centres were getting hit with a spike in complaints, with some customers apoplectic to be missing
parts of the game.
"The real concern we had was that the symptoms customers might have seen were very similar to the symptoms they will see
with poor in-home connectivity," says Pirie.
Spark's contingency plan was to make judgements based on three key factors: numbers of customers affected; how bad it
was; and context.
That last factor - context - was crucial. As far as Spark could tell, perhaps 10 percent of all viewers would be seeing
a "degraded" livestream. While that wasn't great, the numbers seeing pixellation or the buffering 'wheel' would be far
smaller again.
However, this was the All Blacks vs the Springboks in the first round of the Rugby World Cup. The context dictated that
emergency measures were required.
"If this had happened in another game, would we have made the same decision? Depending on the game, we might not have,"
Pirie says.
By this time, word had reached the CEO at Spark Sport's American streaming partner, Akamai, and the Beehive was being
kept in the loop. Yet there was still no technical issue with the livestream itself and plenty of capacity in the New
Zealand broadband system.
However, with half time approaching, Pirie says it was a case of "getting the right people in the room and achieving a
quick consensus" that there was no choice but to switch to live broadcast on TVNZ's Duke channel. At TVNZ, a stand-by
team flicked the switch and, ironically, Duke was broadcasting the game before Spark had a chance to notify customers.
A flaw in contingency planning meant a banner on the Spark Sport livestream telling viewers they could switch to Duke
was delayed. That was largely, it seems, because Spark had planned for its worst nightmare: a complete service failure
with nothing on the screen at all, rather than the halfway house that befell it on Saturday.
While some customers ask why Spark didn't text all subscribers, the reality is that many would only have received the
text long after the final whistle.
By Sunday morning, a post mortem had determined where the problem lay: it was in the way livestream had been allocated
to various internet 'pathways' that brought the game to New Zealand.
While there was more than enough capacity across the whole system, an automated process caused bottlenecks by directing
more of the rugby content into one of those pathways. The issue was exacerbated by heavy mid-Saturday evening demand
from the rest of New Zealand who weren't watching the rugby, but were watching something on, say, Netflix.
"Algorithms did this," says Pirie, a little haplessly. Every other part of the process was working exactly as it should
and that there was no technical reason to prevent everyone being able to watch whatever they liked, without
interruption.
That bottleneck has now, Spark believes, been manually unpicked. It should not be possible for the exact same problem to
happen again. However, no one can be sure until the same level of demand hits the New Zealand broadband network again.
To be on the safe side, all games through Sunday were broadcast on TVNZ Duke as well.
"We're bracing ourselves for tonight for going back to Spark Sport only," says Pirie, who knows some customers are still
having trouble with their set-up at home and that any further failure will gain maximum publicity.
As to the refunds it's offering, Spark knows some people will claim their money back even if their service was OK. A
"surprising" number bought a one-match pass for Saturday's game and they can claim a refund, no questions asked, if they
say the quality was lacking and notify Spark before next Sunday night.
But, in the end, the company has no choice but to be seen to be doing the right thing, says Pirie.
"We accept that with the Rugby World Cup, you've got no choice, so we are giving people the option to opt-out if they
want to."