BOOK A CALL BACK
Have a question? Fill in the form below to book a call
About:
Menu
REMOTE SUPPORT
 NETWORK STATUS
Quarantine yourself, not your business... Delivering high performance cloud solutions 24x7, keeping your business connected
020 7471 3277
Book a call
ALL BLOGS
Upstream Voice Provider Issues
January 31st, 2020
Categories:

One of our upstream voice providers, Gamma, is experiencing a network outage which is causing crackly outbound voice on a subset of users.

Updates:

 

8:45am- Gamma Engineers are continuing to monitor the restoration of service and can confirm traffic is successfully transiting the Gamma network as expected. 

We are aware of registration and functionality issues on the Horizon platform and will be working to resolve these throughout this morning. If you are experiencing any Horizon registration issues we would recommend you reboot any impacted devices and contact the support teams if the registration issues continue. Please note, you may experience functionality issues on the Horizon platform throughout this morning as we work to fully synchronise all services.

We are also working to resolve issues on the Highlight monitoring platform to ensure this reflects the current status of the circuit connectivity accurately.

 

6:10am – The fibre issue in the London area has now been resolved as of 05:01, our monitoring and testing has confirmed that service is restoring across Gamma platforms and we will continue to monitor network stability throughout this morning.
We apologise for any disruption this incident has caused to your service. 

 

01/02 1:30am- To clarify the communication update at 8am –  We will provide any material changes in status as soon as those updates are available, however we expect to provide a fuller update at 8am across all of the areas our engineering teams are working on.

If we have any new updates that give us an indication on how long it will take to restore services we will provide that as soon as available.

 

01/02 1am-  Work on the stabilisation of re-routed traffic across the temporary network connection remains a challenge and we have engaged our technology partner to work alongside us on this.  This will continue to be a priority throughout the evening, but unfortunately we can’t yet provide an outline timescale on when this will complete.

Connection to our Gamma Portal has now been restored and we have tested the number divert capability and this is functioning correctly.

Fibre engineers have identified the break between two fibre joints in the London area and are investigating the options to replace this with a local spare.  Once we know the availability of a spare fibre we hope to be able to provide an estimated time to restore.

The next update will be at 08:00, but any material changes to the status will be updated before then if they become available.

 

01/02 00:00-  The network routing change is being implemented and we are starting to move network traffic across a newly established network connection.  As we stabilise voice and data traffic, customers will continue to see disruption to service.  Our engineering team will continue to focus on this activity as a priority.  We can’t yet provide a timeframe to complete but will do so at the earliest opportunity.

Fibre engineers have now isolated the fault in the London area to a smaller patch and are in the process of testing fibre sections in that patch.

Next update will be by 01:00 or sooner if we can.

 

11pm – The fibre engineers are in the process of isolating the fibre break in the London area and we are expecting a clear view on the fibre restoration before 00:30.  We have encountered further technical challenges on the network routing change to restore services in the interim, this was planned for 22:00 but is still work in progress and we will provide a further update on that change before 00:00.

The next update will be provided before 00:00.

We apologise for the ongoing disruption and all of our resources remain committed to restoring service as soon as we can.

 

9:30pm- Fibre engineers are now on site and beginning work on the restoration.  Our network team have been continuing the work on the network re-routing of voice and connectivity services and we expect to restore those services by 10pm tonight.  We’re confident the re-routing work will fully restore services, however services remain at risk until the fibre restoration work is completed.  We will provide a further update by 11pm.

 

8pm-  We now estimate engineers to arrive at the fibre break in the London area at 9pm at which point they can assess the extent of the work required to restore the fibre connectivity.  Work will begin at 9pm on fibre break within the Leicester area, however we know the restoration work here is extensive due to an electrical incident in that area.  We have taken steps within our network to re-route voice and connectivity services and this action will restore the majority of services, but we expect some ongoing intermittent problems on voice quality and connectivity speed until at one of the two fibre breaks is resolved.  The next update will be provided at 21:30.

 

6:11pm –  We have isolated the fault to a dual fibre break on the East coast and the London area of our national network.  Our technology partner has been engaged and they are deploying engineers to address both breaks.  The incident continues to impact our voice and connectivity services and we are taking mitigating action at a network level to try to restore connectivity where we can. 

 We expect to be able to provide a clearer view on time to resolve by 20:00 hours as deployed engineers establish the extent of the work at each break.

 

5:11pm – We are currently experiencing a Gamma national network fibre break which is impacting Voice and Data connectivity services. Customers will be experiencing voice connectivity and call quality issues due to the congestion as a result of the fibre break.

Gamma engineers are currently working to identify the actual cause of the fibre break at the moment whilst in parallel we are working to review alternative traffic routing measures to restore connectivity and alleviate traffic congestion.

We apologise for any inconvenience caused and will provide a further update within 1 hour.

If you are affected please contact us for further details.