Planned Start Date: 29/08/2023 Planned Start Time: 20:00 Planned End Date: 30/08/2023 Planned End Time: 06:00
This window is classified as Affected.
This means that service is expected to be hard-down between the hours of 20:00 to 06:00 and full service impacting.
From Vendor: 3rd party provider network fibres respliced restoring services. From ITY - Services seen to restore from 4.15am. Have been monitoring since and confirm services restored. Case can be closed and mark resolved pending RFO Report. Expected 5-7 working days.
Vendor Update:
Dear Customer, We have just been advised that engineers are on site and are setting up. Further updates will be shared as we receive them. Regards.
Vendor Update:
Dear Customer, Our 3rd party has provided an eta of 02:45 BST (GMT +1) for their engineers who have been instructed not to leave the site until we have confirmed that services have been restored. Further updates to follow. Regards.
Vendor Update -
Our 3rd party has identified a work order that took place at a location that lines up with the fault seen and the timing of our services going hard down.
The contractor that performed the work has all the records required to resolve the issue and has been escalated to dispatch engineers to resplice.
The next update will be when we have been provided with an ETA.
Vendor Update:
A conference call with NEOS senior management and fibre engineers along with our 3rd party provider is being arranged to formulate an action plan and a way forward to resolve this fault.
A further update will be provided once this call has taken place.
Vendor Update: Dear Customer,
We are still actively chasing our third-party vendor for an ETA for engineer to site and are pushing them through our internal escalation process.
This fault is escalated internally to director level at NEOS.
Our own fibre engineers remain on standby for a joint meet to assist with resolution.
The next update will be in 1 hour unless we receive an significant update before then.
Latest Update:
We are still actively chasing our third-party vendor for an ETA for engineer to site.
As soon as this has been provided, we will dispatch our own fibre engineers for a joint meet to assist with resolution.
The next update will be in 1 hour unless we receive an significant update before then.
Our vendor update:
Dear Customer,
Please note, we have now escalated this to the next level with our third-party vendor at level 5.
Our NOC Management have been on a conference call with our vendor to discuss all previous findings to assist with resolution.
We are still actively pushing them to provide an ETA for engineer on site and hope to have this soon.
The next update will be in 1 hour.
Our Vendor Update:
Dear Customer, Following our escalation with our third-party, they have now accepted our diagnostics and are sending out resources to our identified break location. As soon as we have engineers ETA we will inform you. Please be aware this has been escalated to our CEO and COO who are actively engaged.
The next update will be in 1 hour.
The issue has been pinpointed to 3 fibres provided via a supplier other than VirgiN Media. Virgin Media have now cleared their major incident. A secondary provider has now been actioned to attend. They will accept all previous reports and testing information as their own and have not reset the service level agreement window of repair or resolution, so this is still being treated as a fibre fault that is 27 hours in. This is therefore at the highest escalation level and will be restored at the earliest opportunity. We do not have an estimated resolution time at this moment, but we do expect to have more quality information and detail within the next 2 hours.
Third Party Provider Update: We do apologise for the delayed update. We are actively escalating with our third party vendor for a plan of action on resolution. As soon as they have provided us with an ETR we will update you.
Third Party Provider Update: Please be advised that we are continuing to work alongside our 3rd party vendor to help them to establish the fault on their network and assist in any way possible to aid and speed up the resolution.
Whilst the 3rd party are trying to identify their network issue, we do not currently have an ETR however, we are pushing for further updates as soon as possible.
The next update will be at approximately 16:00 unless we have any updates from the vendor prior.
Third Party Provider Update: Testing has confirmed the break location that was identified last night was incorrectly associated with our network due to a crossed parallel fibre. All this testing information has now been shared with our 3rd party to speed up their investigation and fault resolution.
Our engineers are still onsite and available for any co-op required with the 3rd party vendor where required.
Next update will be at approximately 15:00 unless we have any further major developments before.
Third Party Provider Update: Further testing has identified their may be a further issue at an alternative location on leased fibres. We have now engaged the 3rd party vendor to investigate this issue and are awaiting their initial response.
Our engineers are also continuing to work alongside Fibre teams to find a resolution as soon as possible including looking at the prospect of a re-route.
Next update will be at 14:00 unless there are any other major developments before.
Third Party Provider Update: Our engineers and Tier 3 Fibre teams are continuing to investigate and work to find a resolution as soon as possible however, currently we have no further updates to share.
We will continue to keep you updated as we receive any further details.
The next update will be at approximately 13:00 unless we have any major changes before.
Third Party Provider Update: Currently our engineers are onsite working in conjunction with our Tier 3 Fibre Transmission team to investigate and find a resolution to this fault following failure of the initial repair.
They are looking at several multiple options including alternative routes, additional working spares and potentially an installation of a strap cable to bypass the damage. However, due to the complex nature they are having to take all care when deciding on a solution to ensure further disruption is not caused.
The next update will be at approximately 12:00 unless we have any further major developments.
Third Party Update: Engineers are continuing to investigate for a fix following the initial spare fibres not providing resolution.
Due to the complex nature of the joints involved, engineers are having to take extra care in fibre identification.
We will continue to update you on an hourly basis with the next update at approximately 11:00.
Third Party Provider Update: We have now had to engage further engineers to site due to regulations on working times with the current engineers. Full handover has been completed.
Initial identification of spare fibres has been unsuccessful in resolving the issue, we are working to identify alternative options and a resolution as soon as possible
Next update will be at approximately 10:00am.
Following completion of one joint being spliced, engineers are continuing further investigations at another joint trying to identify the spare fibres.
We will continue to monitor for further updates from the engineers and be in touch when anything else is received.
Next update will be at approximately 09:00.
Third party update -
Engineers have tested and identified fibres which go the full distance and are currently prepping the fibres before splicing. Engineers have finished splicing at one joint and are moving to the next to complete their splicing.
Third Party Update -
Our engineers have discovered that civil engineering works undertaken yesterday in Elgar Road Reading have damaged a 216f cable affecting customer services.
Engineers have been working through the night to identify spare fibres but all the fibres tested so far have been damaged.
Testing is ongoing with no ETR.
Internetty -
This now returns to fault status
Fibre engineers have entered a live fibre joint and are currently carrying out OTDR testing to pinpoint the location of the damage fibres. Due to working in the proximity of fibres carrying live traffic, this work is very delicate and can't be rushed. Next update at 05:00 unless significant progress has been made.
Both NEOS and 3rd party fibre engineers are at the location of the suspected fault and are walking the route to see if there are any obvious signs of civil engineering/building works taking place in the area. Please be advised that the SLA for this fault is P1 fibre with a fix time of upto 16 hours. Next update at 01:00
We have already responded and called that 16 hours from point of first issue woudl extend beyond 6am and that this is not acceptable. They have not yet responded to our request for information.
Virgin Media engineers have OTDR tested the fibre and found the issue to be 800m from Reading Easynet PoP site, the 3rd party fibre provider have been requested to attend to pin point the issue.
A further status update will be provided at 23:00
We can now confirm engineers have arrived on site and are in the process of locating the affected fibre. Once located, they will proceed to conduct the ODTR testing.
Statement from Third Party-
We can now confirm that both engineers have arrived on site and are in the process of locating the affected fibre. Once located, they will proceed to conduct the ODTR testing.
A further status update will be provided within 1 hour.
We are now within Maintenance Window ITY2309002 but will continue to update you on the fibre break.
Engineers have yet to confirm arrival to site in order to begin ODTR testing - We have chased our supplier for more information on this. This fault will now extend into the maintenance window. We await further details on the estimated time for restoration of services under the maintenance window.
Although it has not yet been confirmed, we believe this outage will roll into the scheduled maintenance window due to commence 20:00hrs 29th August 2023. We will await confirmation from the third-party provider,
Statement from Neos Networks:
Please be advised this fault is related to a wider outage that we believe is a suspected fibre break between Reading and Froxfield.
We have engaged our 3rd party vendor regarding this as well as our own Fibre engineers to investigate further and to pin point the fault.
There is currently no ETA however, we will endeavour to keep you updated as soon as we receive further information.
Internetty -
We are constantly requesting updates and understand customer frustrations. We are also looking into available alternatives to restore service faster if possible.
Urgent support request Created: 29/08/2023 15:15:14 BST
Priority 1 - Critical - has been raised with provider
If you are currently without service please leave any equipment on to ensure a smooth restoration, further updates/advice to follow. DO NOT FACTORY RESET ROUTERS, THIS WILL LEAD TO FURTHER DOWNTIME
Telehouse North 2, Coriander Avenue, London,E14 2AA to Liverpool Exchange BT Liverpool Royal,Royal House, 100 Wood Street, Liverpool, MERSEYSIDE,,L1 4LA - NEOS Networks reporting Hard down Fault. Awaiting their reponse
Our Network operations team are aware and working with third-party providers to fully rectify a service performance issue
Incident Closed. Traffic was moved back seamlessly as of 11/09/2023 12:00:00 and the link has remained stable. We are now back to normal operations. Anyone still experiencing issues of any form is advised to contact our Customer Services Team. After outage report to follow as soon as this has been finalised.
Maintenance Window ITY2309003 - Closed
Conclusion -
Service is no longer considered at risk. Despite the nature of the work, very few failed packets were observed during changes. A new upgraded original backup link has been installed moving traffic closer to the affected area and improving performance. We will continue with strict monitoring on both this backup link and the main link before preparing to move traffic back to normal operations which due to this morning's work will be seamless.
Maintenance Window ITY2309003 - Opened
Scheduled Maintenace Window in Relation to this incident, Impact - At Risk, Date: Friday 8th September 2023, Time: 05:00am-06:00am
Description:
Our Networks Operations team will complete essential maintenance works between 5:00am -6:00am on Friday 8th September 2023. During this maintenance window, services are considered at risk.
At risk indicates that services are not expected to be impacted; however, as our Network Operations Team will be working on critical infrastructure, the risk of service impact is heightened.
We are carrying out these works at a time where traffic levels, for affected areas, have historically been observed as being at their lowest.
Works are required as step one of the process of returning to normal operation for all customers.
If access to the internet is critical between these times, we do hope that our prior notice will enable you to make alternative arrangements. You can stay up to date with the progress of the Maintenance Window here.
If you have any questions or queries, please contact our Support Team.
Email Communications are on their way to customers however the campaign is expected to take a few hours to get out to all customers so we wanted to notify you as soon as possible.
Update from Third Party Following Barton Link Failure -
We are happy to confirm this is still stable and they are clearing the fault as of today following their monitoring window of 72 Hours. We will now enter our monitoring window of an additional 72 Hours.
Updates to follow on Traffic management plans.
Network Monitoring has reported disconnects and performance issues between the following times : 2023-09-07 07:30:07 - 2023-09-07 07:42:02 . The cause of this is being investigated with more updates to follow.
Third-Party Provider Update on Main Link -
I'm happy to confirm this is still stable. They will continue to check until Friday.
The backup link is still installed and working well.
Important Notice -
Can anyone still experiencing any disconnects in any form since 08:00 yesterday please contact our customer services team for investigation.
Updates to follow
The Third-Party Provider has confirmed that work on the Faulty Main Barton link was completed last night
The change was fully completed at 3:49 am.
The card has been replaced and monitored and reported all good
Our latest update from our Third Party Provider of the main link to the Barton Upon Humber Cabinet is the following:
The OTN card is being swapped at Nottingham under change conditions tonight. This is due to it impacting multiple other ISP Links. They will inform us once the change has been done and this link will then be monitored for an extensive period before non-service-impacting traffic is tested to see if the link remains stable.
In terms of the backup connection:
We have secured a link via a completely different area of Lincolnshire. We have moved some services onto this router.
This route is to avoid having to use the faulty link we have from Barton to London and allows us to alleviate some traffic from the backup connection we installed last bank holiday weekend to Wootton that has been slowly saturated over the last 24-36 hours, making speeds much lower than customers should receive and causing some disconnects.
We once again apologise for the inconvenience caused over this period and promise we are taking all steps possible to remove any vulnerabilities such as this from the network of which this newly established link is a major part of this. We would like to thank you for your patience and support in this matter and promise to return to delivering the service that you and us expect. We will have more updates on the closure of this incident in the coming days and will then publish a full detailed after-outage report detailing:
Virgin Media attended our port at Grimsby POP today and found what they believe to be a fault on our connected port. This port has now been changed. We monitored the stability of the link for several hours today and have not seen the same issues with the link that we have been seeing over the last few days. We have returned some non-customer impacting services to this link to test reliability further. All alternative services are working to specification at the moment and our performance issue status has been removed.
Virgin Media attended our Ferriby Road cabinet last night. The circuit that routes traffic from Barton to London was inspected, and a fibre light loss of -7Db from expected levels was found. This was repaired.
We moved some traffic back to this link, as opposed to using the new backup link we installed over the weekend, to test the repair. This morning, the circuit failed again. We have informed Virgin Media and await their arrival.
The backup link we have installed is now in use again. This circuit is working well, however, the transition of services between links should be seamless - However, some customers are reporting this is not the case, whereas for others, it is.
We are currently investigating why this is the case and working on how we ensure any failover is instant and does cause any internet disruption for customers. Further updates will follow.
We just experienced a link down. Provisions tested last night were activated by our network operations team and traffic is going out via an alternative route. To ensure a smoother connection for all please avoid speed testing during this period as this will potentially decrease your performance.
Risk has been downgraded from high to medium as there are still some parameters in place that are beyond our reasonable control. Engineers have finished fibre splicing works to create another exit route for traffic from affected areas if the third-party link were to fail again. Our network operations team have established the link and will complete testing between 3am and 4am. Communications with the third party remain open while a reason for the outage is still to be established.
Communications remain ongoing with our third-party providers. The link has remained stable since restoration. The risk has now been downgraded to High from Very High. Anyone who has factory reset routers is urged to contact our customer services team as soon as possible to enable us to restore your service through means of remote reconfiguration.
Services have been restored. We are watching this very closely at present and services are still considered at a Very High Risk of failure. Anyone still having issues should power off their router for 30 seconds and power it back on again. DO NOT FACTORY RESET ROUTERS WITH THE RESET BUTTONS, THIS WILL LEAD TO FURTHER DOWNTIME
We are currently investigating reports of service loss in the North Lincolnshire area. Our team are currently investigating this and an update will be posted in the next few moments.
Service has now fully restored.
Network Operations are advising that services are restoring. Whilst the network is being tested, services are still considered at risk during this time.
Our Network Operations Team have identified a hard-down fault with the third party providers connection to Internetty.
An urgent fault has been raised with the third party provider and an update is expected within the hour.
Our Network Operations Team are continuing to investigate the source of the issue. Engineers are on the way to the cabinet to aid investigations.
Our Network Operations Team have advised that some but not all services in the Barton and Barrow area are affected by the outage. Further investigations are being completed and updates will follow shortly.
No incidents reported
No incidents reported
Our monitoring systems have detected a loss of power error to the main router which services Brunswick park leading to a full outage at the site. Updates to follow
Following extensive work today, the power vulnerability has been eradicated. Services are no longer considered at risk.
Night Conceirge have assisted in getting equipment back on. Appears stable. We will continue to monitor. Engineers will attend tomorrow AM to review
Efforts to restore service on this occasion have been unsuccessful. Updates to follow
Following further contact with building management they are going to reattend to work with our engineers to restore services once again. We have already put in place for our engineers to attend first thing in the morning to investigate why this is happening however it is of upmost importance that we restore service as quickly as we can this evening to minimise disruption. Updates to follow on progress of investigations
Our monitoring systems have detected that the issue has occurred again. Network operations are investigating the route cause of this and will update accordingly.
All services have now reinstated. Outage marked as resolved by Network operations as of 21:08:00 24/08/2023 . Anyone still having issues is urged to contact our customer services team.
Building management have now restored power to the equipment which is now coming back online. Our Network Team will now monitor the equipment. Services remain at risk during this time.
Building management are now on site and are now working with our Network Team to restore power.
Building management have been dispatched to site and have provided an eta of 20 minutes. Further updates to follow.
No incidents reported