North Lincolnshire Monday 28th August 2023 13:58:00

Maintenance Window ITY2309003 - Closed

Conclusion -

Service is no longer considered at risk. Despite the nature of the work, very few failed packets were observed during changes. A new upgraded original backup link has been installed moving traffic closer to the affected area and improving performance. We will continue with strict monitoring on both this backup link and the main link before preparing to move traffic back to normal operations which due to this morning's work will be seamless.

Maintenance Window ITY2309003 - Opened

Scheduled Maintenace Window in Relation to this incident, Impact - At Risk, Date: Friday 8th September 2023, Time: 05:00am-06:00am

Description:

Our Networks Operations team will complete essential maintenance works between 5:00am -6:00am on Friday 8th September 2023. During this maintenance window, services are considered at risk.

At risk indicates that services are not expected to be impacted; however, as our Network Operations Team will be working on critical infrastructure, the risk of service impact is heightened.

We are carrying out these works at a time where traffic levels, for affected areas, have historically been observed as being at their lowest.

Works are required as step one of the process of returning to normal operation for all customers. 

If access to the internet is critical between these times, we do hope that our prior notice will enable you to make alternative arrangements. You can stay up to date with the progress of the Maintenance Window here.

If you have any questions or queries, please contact our Support Team.

Email Communications are on their way to customers however the campaign is expected to take a few hours to get out to all customers so we wanted to notify you as soon as possible.

Update from Third Party Following Barton Link Failure -

We are happy to confirm this is still stable and they are clearing the fault as of today following their monitoring window of 72 Hours. We will now enter our monitoring window of an additional 72 Hours.

Updates to follow on Traffic management plans.

Network Monitoring has reported disconnects and performance issues between the following times : 2023-09-07 07:30:07 - 2023-09-07 07:42:02 . The cause of this is being investigated with more updates to follow.

Third-Party Provider Update on Main Link -

I'm happy to confirm this is still stable. They will continue to check until Friday.

The backup link is still installed and working well.

Important Notice -

Can anyone still experiencing any disconnects in any form since 08:00 yesterday please contact our customer services team for investigation.

Updates to follow

The Third-Party Provider has confirmed that work on the Faulty Main Barton link was completed last night

The change was fully completed at 3:49 am.

The card has been replaced and monitored and reported all good

Our latest update from our Third Party Provider of the main link to the Barton Upon Humber Cabinet is the following:

The OTN card is being swapped at Nottingham under change conditions tonight. This is due to it impacting multiple other ISP Links. They will inform us once the change has been done and this link will then be monitored for an extensive period before non-service-impacting traffic is tested to see if the link remains stable.

In terms of the backup connection:

We have secured a link via a completely different area of Lincolnshire. We have moved some services onto this router.

This route is to avoid having to use the faulty link we have from Barton to London and allows us to alleviate some traffic from the backup connection we installed last bank holiday weekend to Wootton that has been slowly saturated over the last 24-36 hours, making speeds much lower than customers should receive and causing some disconnects.

We once again apologise for the inconvenience caused over this period and promise we are taking all steps possible to remove any vulnerabilities such as this from the network of which this newly established link is a major part of this. We would like to thank you for your patience and support in this matter and promise to return to delivering the service that you and us expect. We will have more updates on the closure of this incident in the coming days and will then publish a full detailed after-outage report detailing:

1. Timeline
2. Summary of Incident
3. Customer impact
4. Resolution summary
5. Actions required to mitigate future occurrences

Virgin Media attended our port at Grimsby POP today and found what they believe to be a fault on our connected port. This port has now been changed. We monitored the stability of the link for several hours today and have not seen the same issues with the link that we have been seeing over the last few days. We have returned some non-customer impacting services to this link to test reliability further. All alternative services are working to specification at the moment and our performance issue status has been removed.

Virgin Media attended our Ferriby Road cabinet last night. The circuit that routes traffic from Barton to London was inspected, and a fibre light loss of -7Db from expected levels was found. This was repaired.

We moved some traffic back to this link, as opposed to using the new backup link we installed over the weekend, to test the repair. This morning, the circuit failed again. We have informed Virgin Media and await their arrival.

The backup link we have installed is now in use again. This circuit is working well, however, the transition of services between links should be seamless - However, some customers are reporting this is not the case, whereas for others, it is.

We are currently investigating why this is the case and working on how we ensure any failover is instant and does cause any internet disruption for customers. Further updates will follow.

We just experienced a link down. Provisions tested last night were activated by our network operations team and traffic is going out via an alternative route. To ensure a smoother connection for all please avoid speed testing during this period as this will potentially decrease your performance.

Risk has been downgraded from high to medium as there are still some parameters in place that are beyond our reasonable control. Engineers have finished fibre splicing works to create another exit route for traffic from affected areas if the third-party link were to fail again. Our network operations team have established the link and will complete testing between 3am and 4am. Communications with the third party remain open while a reason for the outage is still to be established.

Communications remain ongoing with our third-party providers. The link has remained stable since restoration. The risk has now been downgraded to High from Very High. Anyone who has factory reset routers is urged to contact our customer services team as soon as possible to enable us to restore your service through means of remote reconfiguration.

Services have been restored. We are watching this very closely at present and services are still considered at a Very High Risk of failure. Anyone still having issues should power off their router for 30 seconds and power it back on again. DO NOT FACTORY RESET ROUTERS WITH THE RESET BUTTONS, THIS WILL LEAD TO FURTHER DOWNTIME