SingTel fined a record $6m for Bukit Panjang exchange fire; OpenNet and CityNet also fined

The hour-long fire that crippled essential services from banking to health records retrieval across Singapore was caused by the use of an unauthorized blowtorch by a SingTel staff when performing maintenance works. It sparked a slow-burning fire that went undetected as a result of further fire safety lapses and human error.

Get the full story from The Straits Times.

Here is the full statement from the Infocomm Development Authority of Singapore:

The Infocomm Development Authority of Singapore (IDA) has completed its investigations into the service disruption caused by a fire at the Bukit Panjang Exchange on 9 October 2013. The fire damaged optical fibre cables and affected telecommunication services in the northern and western parts of Singapore, including fixed voice, broadband Internet access and mobile services. The service disruption also affected close to 270,000 telecom and broadcast subscribers[1], including residential users, some government agencies, businesses, financial institutions, and multi-application kiosks providing e-commerce and payment services. The affected services were fully restored on 17 October 2013.

IDA has found that SingTel, CityNet and OpenNet had not fulfilled their respective obligations, such as to provide sufficiently resilient telecommunication systems and services, and to restore services to affected end users as quickly as possible when the service disruptions occurred.

SingTel

As a provider of fixed line telephone services, SingTel was found to have breached its obligations under the Code of Practice for Telecommunication Service Resiliency (Service Resiliency Code) for the severe disruption to its services from 9 to 14 October 2013.

Based on IDA's investigations, IDA has concluded that the service disruptions were due to a fire in the cable chamber of the Bukit Panjang Exchange, which was most likely caused by hot works carried out by SingTel on its cables on 9 October 2013. IDA also found that the service disruptions could have been prevented had SingTel enforced its standard operating procedures and work safety practices. These included but are not limited to a lack of proper supervision when hot works were carried out at the Exchange.

IDA also noted that SingTel had certain business continuity management (BCM) policies and processes in place, and that SingTel had implemented these during the incident. However, SingTel's BCM did not have specific contingency plans to address serious service outage situations of such a scale as the Bukit Panjang incident.

IDA additionally observed weaknesses in SingTel's network and practices, such as: (i) having outdated work practice of using lead sealants in the cable chamber which required hot works to be carried out; (ii) a single point of failure in the Exchange where cable rings were temporarily rerouted due to construction work in the vicinity of the Exchange; and (iii) wrongly assuming that fibre paths were diverse for certain enterprise customer services without fully testing them. This is in spite of the fact that SingTel's network architecture is generally in line with international standards.

Accordingly, for failing to meet its obligations under the Service Resiliency Code, IDA has imposed a financial penalty of S$6 million on SingTel. Mitigating factors were taken into consideration by IDA in determining the financial penalty. These included SingTel providing affected users with alternative services during the incident and its offers of compensations to such users after the incident.

CityNet

As the owner and manager of the Exchange, CityNet did not meet its licence obligation as a Facilities-Based Operator to maintain and operate telecommunication systems, which included the cable chamber in the Bukit Panjang Exchange, in an adequate and satisfactory manner. In particular, IDA found that there were procedural lapses on CityNet's part, such as a lack of robust approving process for hot works in the Exchange, and the failure to ensure the safety of the work area and compliance with proper operating procedures. These lapses contributed to the incident. Accordingly, IDA has imposed a financial penalty of S$300,000 on CityNet.

OpenNet

OpenNet is the Network Company (NetCo) responsible for the design, build and operation of the passive infrastructure layer of the Nationwide Broadband Network (NBN). While not a direct contributor to the cause of the incident, OpenNet has been found to contribute to the delay in service restoration to its affected users.

IDA found that OpenNet could have expedited the service restoration and recovery process, such as by adopting more appropriate service restoration processes for large scale service disruptions. As a result, the duration of service disruptions to affected users was unnecessarily prolonged. In this regard, IDA found OpenNet to have contravened its obligations under the NetCo Resiliency Code, and has imposed a financial penalty of S$200,000.

A commitment to improve

Arising from the incident, the respective operators conducted their own network and service resilience reviews and have since proposed measures to address their respective shortcomings and enhance resiliency planning and practices.

Some commitments made by SingTel to improve include replacing duct seals in exchanges that will eliminate the need for hot works, enhancing fire safety measures and practices such as installing automatic fire suppression systems within cable chambers in all its exchanges, conducting regular training and assessment of its employees on workplace safety, and enhancing facilities monitoring such as installing more CCTV cameras at appropriate places. SingTel has also committed to IDA that it will be enhancing its Operations Support System and processes to ensure compliance to fibre path diversity requirements for enterprise customers with redundancy and path diversity needs.

CityNet has committed to improve procedural controls such as stepping up on the monitoring of maintenance works in its exchanges to ensure safety of work area, and enhancing its fire prevention, detection and protection measures.

OpenNet will also enhance its BCM and work with the other operators on an end-to-end BCM in the multi-layered NBN, as part of IDA's direction.

IDA will closely monitor the implementation of the improvement measures of the operators to ensure that the measures are put in place as committed and directed.

Review of telecommunication networks' resiliency

IDA has already embarked on a review of the resilience of all critical parts of Singapore's infocomm infrastructure and the review is expected to be completed in the second half of 2014. The review will help to identify any other potential weakness in the critical networks and seek to address them to minimise the risk of widespread service disruptions. In addition, IDA has worked with the industry to put in place a set of operating procedures to improve inter-operator communication and coordination during major service disruptions. To ensure that such a coordination process stays relevant to the operators and industry, IDA will review the coordination process from time to time.

Mr Leong Keng Thai, IDA's Deputy Chief Executive and Director-General (Telecoms and Post), said, "This is a very serious service outage of a magnitude that is unprecedented, but more importantly, that could have been avoided. With increasing dependency of businesses and consumers on telecommunications, a strong signal must be sent to telecommunication operators that they must take network resilience very seriously, and invest in necessary infrastructure, processes and training to prevent and minimise service outages. IDA will closely monitor the operators' undertaking of the corrective measures and ensure that they are implemented."

Mr Leong added, "Notwithstanding this incident, there is overall resiliency in Singapore's telecommunication networks and services in the form of multiple operators, networks and technology platforms. Resiliency does not mean that incidents will not happen. An important part of resiliency is to be prepared for incidents and to recover quickly. IDA will continue to review its regulatory frameworks to ensure that telecommunication operators take proactive steps to enhance the resilience of their networks and services. IDA will also take steps to further strengthen the resilience of Singapore's telecommunication networks."