Our Network operations team are aware and working with third-party providers to fully rectify a service performance issue
Incident Closed. Traffic was moved back seamlessly as of 11/09/2023 12:00:00 and the link has remained stable. We are now back to normal operations. Anyone still experiencing issues of any form is advised to contact our Customer Services Team. After outage report to follow as soon as this has been finalised.
Maintenance Window ITY2309003 - Closed
Conclusion -
Service is no longer considered at risk. Despite the nature of the work, very few failed packets were observed during changes. A new upgraded original backup link has been installed moving traffic closer to the affected area and improving performance. We will continue with strict monitoring on both this backup link and the main link before preparing to move traffic back to normal operations which due to this morning's work will be seamless.
Maintenance Window ITY2309003 - Opened
Scheduled Maintenace Window in Relation to this incident, Impact - At Risk, Date: Friday 8th September 2023, Time: 05:00am-06:00am
Description:
Our Networks Operations team will complete essential maintenance works between 5:00am -6:00am on Friday 8th September 2023. During this maintenance window, services are considered at risk.
At risk indicates that services are not expected to be impacted; however, as our Network Operations Team will be working on critical infrastructure, the risk of service impact is heightened.
We are carrying out these works at a time where traffic levels, for affected areas, have historically been observed as being at their lowest.
Works are required as step one of the process of returning to normal operation for all customers.
If access to the internet is critical between these times, we do hope that our prior notice will enable you to make alternative arrangements. You can stay up to date with the progress of the Maintenance Window here.
If you have any questions or queries, please contact our Support Team.
Email Communications are on their way to customers however the campaign is expected to take a few hours to get out to all customers so we wanted to notify you as soon as possible.
Update from Third Party Following Barton Link Failure -
We are happy to confirm this is still stable and they are clearing the fault as of today following their monitoring window of 72 Hours. We will now enter our monitoring window of an additional 72 Hours.
Updates to follow on Traffic management plans.
Network Monitoring has reported disconnects and performance issues between the following times : 2023-09-07 07:30:07 - 2023-09-07 07:42:02 . The cause of this is being investigated with more updates to follow.
Third-Party Provider Update on Main Link -
I'm happy to confirm this is still stable. They will continue to check until Friday.
The backup link is still installed and working well.
Important Notice -
Can anyone still experiencing any disconnects in any form since 08:00 yesterday please contact our customer services team for investigation.
Updates to follow
The Third-Party Provider has confirmed that work on the Faulty Main Barton link was completed last night
The change was fully completed at 3:49 am.
The card has been replaced and monitored and reported all good
Our latest update from our Third Party Provider of the main link to the Barton Upon Humber Cabinet is the following:
The OTN card is being swapped at Nottingham under change conditions tonight. This is due to it impacting multiple other ISP Links. They will inform us once the change has been done and this link will then be monitored for an extensive period before non-service-impacting traffic is tested to see if the link remains stable.
In terms of the backup connection:
We have secured a link via a completely different area of Lincolnshire. We have moved some services onto this router.
This route is to avoid having to use the faulty link we have from Barton to London and allows us to alleviate some traffic from the backup connection we installed last bank holiday weekend to Wootton that has been slowly saturated over the last 24-36 hours, making speeds much lower than customers should receive and causing some disconnects.
We once again apologise for the inconvenience caused over this period and promise we are taking all steps possible to remove any vulnerabilities such as this from the network of which this newly established link is a major part of this. We would like to thank you for your patience and support in this matter and promise to return to delivering the service that you and us expect. We will have more updates on the closure of this incident in the coming days and will then publish a full detailed after-outage report detailing:
Virgin Media attended our port at Grimsby POP today and found what they believe to be a fault on our connected port. This port has now been changed. We monitored the stability of the link for several hours today and have not seen the same issues with the link that we have been seeing over the last few days. We have returned some non-customer impacting services to this link to test reliability further. All alternative services are working to specification at the moment and our performance issue status has been removed.
Virgin Media attended our Ferriby Road cabinet last night. The circuit that routes traffic from Barton to London was inspected, and a fibre light loss of -7Db from expected levels was found. This was repaired.
We moved some traffic back to this link, as opposed to using the new backup link we installed over the weekend, to test the repair. This morning, the circuit failed again. We have informed Virgin Media and await their arrival.
The backup link we have installed is now in use again. This circuit is working well, however, the transition of services between links should be seamless - However, some customers are reporting this is not the case, whereas for others, it is.
We are currently investigating why this is the case and working on how we ensure any failover is instant and does cause any internet disruption for customers. Further updates will follow.
We just experienced a link down. Provisions tested last night were activated by our network operations team and traffic is going out via an alternative route. To ensure a smoother connection for all please avoid speed testing during this period as this will potentially decrease your performance.
Risk has been downgraded from high to medium as there are still some parameters in place that are beyond our reasonable control. Engineers have finished fibre splicing works to create another exit route for traffic from affected areas if the third-party link were to fail again. Our network operations team have established the link and will complete testing between 3am and 4am. Communications with the third party remain open while a reason for the outage is still to be established.
Communications remain ongoing with our third-party providers. The link has remained stable since restoration. The risk has now been downgraded to High from Very High. Anyone who has factory reset routers is urged to contact our customer services team as soon as possible to enable us to restore your service through means of remote reconfiguration.
Services have been restored. We are watching this very closely at present and services are still considered at a Very High Risk of failure. Anyone still having issues should power off their router for 30 seconds and power it back on again. DO NOT FACTORY RESET ROUTERS WITH THE RESET BUTTONS, THIS WILL LEAD TO FURTHER DOWNTIME
We are currently investigating reports of service loss in the North Lincolnshire area. Our team are currently investigating this and an update will be posted in the next few moments.
Service has now fully restored.
Network Operations are advising that services are restoring. Whilst the network is being tested, services are still considered at risk during this time.
Our Network Operations Team have identified a hard-down fault with the third party providers connection to Internetty.
An urgent fault has been raised with the third party provider and an update is expected within the hour.
Our Network Operations Team are continuing to investigate the source of the issue. Engineers are on the way to the cabinet to aid investigations.
Our Network Operations Team have advised that some but not all services in the Barton and Barrow area are affected by the outage. Further investigations are being completed and updates will follow shortly.
No incidents reported
No incidents reported
Our monitoring systems have detected a loss of power error to the main router which services Brunswick park leading to a full outage at the site. Updates to follow
Following extensive work today, the power vulnerability has been eradicated. Services are no longer considered at risk.
Night Conceirge have assisted in getting equipment back on. Appears stable. We will continue to monitor. Engineers will attend tomorrow AM to review
Efforts to restore service on this occasion have been unsuccessful. Updates to follow
Following further contact with building management they are going to reattend to work with our engineers to restore services once again. We have already put in place for our engineers to attend first thing in the morning to investigate why this is happening however it is of upmost importance that we restore service as quickly as we can this evening to minimise disruption. Updates to follow on progress of investigations
Our monitoring systems have detected that the issue has occurred again. Network operations are investigating the route cause of this and will update accordingly.
All services have now reinstated. Outage marked as resolved by Network operations as of 21:08:00 24/08/2023 . Anyone still having issues is urged to contact our customer services team.
Building management have now restored power to the equipment which is now coming back online. Our Network Team will now monitor the equipment. Services remain at risk during this time.
Building management are now on site and are now working with our Network Team to restore power.
Building management have been dispatched to site and have provided an eta of 20 minutes. Further updates to follow.
No incidents reported
No incidents reported
Service Outage - Engineers are on site investigating
Services are restoring. We await information from the provider as to reason for outage. Services still considered at risk while we await further information
Tests are still being conducted to find the source of the issue
Description of Fault:- Primary national circuit is offline. 3rd party provider has been informed. Awaiting call back from out of hours team once preliminary tests are completed