Despite offers of help, Rogers Communications Inc. could not transfer millions of customers to its competitors’ networks – during this month’s massive service outage, the company says in newly released documents.
The wireless giant’s chief technology officer reached out to his colleagues at BCE Inc. and Telus Corp. at 6 a.m. on July 8 to update them about the service disruption and warn them to be on the lookout for possible cyberattacks, according to a partial redacted document posted to the Canadian Radio-television and Telecommunications Commission website late Friday.
The outage left millions without wireless, internet, or phone service, and also impacted the Interac debit system and 911 services. Four emergency alerts, including three tornado warnings and one dangerous person report, were not delivered to Rogers’ wireless customers, the company said in Friday’s publications.
Rogers later determined that the service disruption was caused by an encoding error that had occurred as part of a maintenance upgrade of its core network.
Although BCE’s Telus and Bell Canada offered support, Rogers quickly found that its customers could not use its competitors’ networks because certain elements of the Rogers network, such as B. the centralized user database, were not accessible due to the failure.
“Furthermore, given the national nature of this event, no competitor’s network would have been able to handle the additional and sudden volume of wireless users (over 10.2 million) and associated surge in voice/data traffic,” stated Rogers in response to a litany known by questions from Canada’s telecoms regulator. “Without careful execution, such an attempt could have disrupted the operation of the other airlines’ networks.”
Industry Minister François-Philippe Champagne has instructed Canadian telecoms companies to reach a formal agreement to help each other out in the event of network outages and to allow customers emergency roaming on their networks.
Rogers said it will explore various options with its competitors as the telecoms companies work to formalize a mutual assistance pact to be presented to the minister in September.
The update that triggered the massive outage was the sixth phase of a seven-phase process that began on February 8th. The company said it went through an extensive planning process, including budget and project approvals, risk assessment and testing prior to launch. The first five phases went smoothly.
The sixth phase began at 2:27 a.m. on July 8 (Rogers said upgrades typically occur during times when network traffic is quietest). The coding error that triggered the cascade of events that led to the outage of the company’s core network was introduced at 4:43 a.m., Rogers said.
“While every effort was made to prevent and contain the outage, the consequences of the coding change affected the network very quickly,” the Toronto-based telecom wrote.
Many Telekom network employees were unable to connect to the company’s IT and network systems due to the outage. This slowed initial response as employees had to travel to central locations where they could access the network. To make matters worse, employees could not access the company’s core network through the VPN (Virtual Private Network) system, delaying efforts to identify the problem and restore services.
A number of Rogers employees were able to communicate with each other and coordinate recovery efforts via Bell or Telus SIM cards they were given in 2015 as part of an inter-carrier contingency plan.
A significant portion of the information Rogers disclosed to the CRTC has been redacted in the public-facing version of the document. The wireless giant said it was “as transparent as possible” to win back its customers’ trust, but noted that some of the information it shared with the regulator needed to remain confidential as it was “commercially and operationally sensitive”. .
“This is especially true for system designs and network operations that could be exploited by malicious actors trying to disrupt our systems,” Rogers wrote.
Telekom said it has already taken steps to prevent future outages and is reviewing its change, planning and implementation process. However, details of its plans — including short-, medium-, and long-term measures the company says it will implement over the coming days and weeks — are redacted.
Rogers said it took a number of steps after its wireless services went down for 16 hours in April 2021. However, many of these measures were specific to the circumstances of the 2021 outage, which the company says was caused by a software update from its vendor Ericsson. “Unfortunately, [these measures] did not prevent the special circumstances that led to the July 8 outage, although they positively contributed to its resolution,” the company wrote.
Rogers chief executive Tony Staffieri has vowed to make “any necessary changes and investments” to ensure the company does not experience another similar outage.
Earlier this week, Rogers replaced his chief technology officer, former Vodafone executive Jorge Fernandes, with Ron McKenzie. Mr. McKenzie has three decades of experience in the telecommunications industry and was previously President of Rogers for Business, the division that provides wireless and Internet services to enterprise customers.
The management change comes just days before a hearing Monday before the House of Commons Industry and Technology Committee investigating the outage.
The committee, made up of lawmakers from all four major federal parties, is expected to quiz Rogers executives about the five-day loan the company is providing to compensate its customers for the outage, and how Telekom will do so will prevent the problem from reoccurring.
Written by Globe editors, our Morning Update and Evening Update newsletters give you a concise summary of the day’s most important headlines. Sign up today.