šŸ’¤ Unscheduled downtime

Good to see normal service is resumed then

:lou_wink_2:

1 Like

Sorry about the downtime. Another process on that box was slowing everything else down

I think we had our biggest outage ever today. Wasnā€™t just us. The whole of hostinguk was down, something of a fucking PR disaster for them, as it turns out that just like us, they are buying their capacity from someone else.

That someone else had major problemos.

Crap service from the hosts, really. Their tech support line just went straight to a dead tone. Crossword Solver, presumably a site that shares space with Sotonians, aptly sent a lot of cross words their way, which is why I didnā€™t.

All good now. Hopefully a reminder of not knowing what youā€™ve got until itā€™s gone.

6 Likes

I call a Sotonians meeting over this to brainstorm ways to extend this PR disaster to express how 7 middle-aged men couldnā€™t post! This is an outrage. As proposee (new word bletch) I set the venue as Mordenā€¦ If not Wimbledonā€¦ 3rd option Wetherspoonā€™s Eastleigh if beer and expenses included.

6 Likes

What about the Semi-Final, Tokes? Ideal opportunity.

3 Likes

Iā€™ll let you off @pap , I think my employers actually appreciated my heightened output for themā€¦

2 Likes

Has anyone started a semi final meet up thread yet? I refuse as I will not be drinking :lou_angry:

Iā€™m still undecided about going yetā€¦

Wealdstone FC at Ruislip for our coach of pissheads. Short tube ride to Wem-ber-lee.

1 Like

How so?

1 Like

Cost? Travelling time? And Iā€™ll be at the Bournemouth home game so i feel guilty about leaving the kids with Mrs Btripz for 2 days in 6ā€¦

Most of this is outweighed by the exceptional company and the chance of a showing from @barry-sanchez and @tokyo-saint ā€¦

1 Like

I would recommmend buying your ticket. Either use it or sell it.

2 Likes

1 Like

Oh you know Iā€™m going to be thereā€¦is there such a thing as pre-buyers remorse?

Central London / Waterloo for pre beers surely?!

OK, that was a biggy!

Our hosting provider (Hostinguk.net - feel free to troll them on Twitter) had a massive outage from about 9 pm last night.

Apparently, and I shit you not, a fibre cable that had been run under a farmerā€™s field was cut. I

mean, farmers digging up a field, whoā€™d have predicted that?

Anyway, we are up again BUT WE MAY STILL EXPERIENCE MORE DOWNTIME as this is a temporary overground fix.

What is really strange about this is that hosting companies build redundancy into each area of their operations so that if one part fails another one will kick in until full service can be restored. It SEEMS as though Hosting UK did have multiple fibre routes to their data centre, but they all went under farmer Gilesā€™ field.

If youā€™re a geek, or are particularly warped, hereā€™s the whole brutal saga. See here for continued updates.

  • Update (31/05/2018 20:47): Our NOC has identified a network incident affecting several data centres on our UK network.
  • Update (31/05/2018 21:19): It is suspected that the convergence of two diverse fibre paths, from an underlying provider Zayo, may have been severed resulting in network isolation for DC 8 (St. Asaph). Engineers are working on three resolution paths (transit re-route, fibre repair and new fibre provisioning) with an ETA of 2-12 hours for full service restoration.
  • Update (31/05/2018 21:30): No new information available. Updates to follow within 15 minutes.
  • Update (31/05/2018 21:45): Zayo are investigating the fibre failures in North West England
  • Update (31/05/2018 22:00): No new information available. Updates to follow within 15 minutes.
  • Update (31/05/2018 22:15): No new information available. Updates to follow within 15 minutes.
  • Update (31/05/2018 22:30): No new information available. Updates to follow within 15 minutes.
  • Update (31/05/2018 22:45): We are awaiting confirmation, but current investigation would indicate the break of two, diverse fibre paths.
  • Update (31/05/2018 23:00): There appears to be a fibre event 27km from Manchester. Once we have an ETA from our fibre repair group it will be provided. Further updates to follow as more information is available.
  • Update (31/05/2018 23:15): It is confirmed that there was no planned works occurring at the time of the break. Updates to follow.
  • Update (31/05/2018 23:30): No new information available. Updates to follow within 15 minutes.
  • Update (31/05/2018 23:45): No new information available. Updates to follow within 15 minutes.
  • Update (01/06/2018 00:00): A technician is en route to the break location with an ETA of 20 minutes. We will begin testing as soon as the technician arrives.
  • Update (01/06/2018 00:35): No new information available. Updates to follow within 15 minutes.
  • Update (01/06/2018 00:45): The on-site tech has determined this is due to a large cable cut and narrowed the potential damage location. Our underlying fibre provider, Zayo, have OSP construction and splice crews en-route to a common Splice Enclosure just North of Stretford. Further updates to follow as more information is available.
  • Update (01/06/2018 01:00): No new information available. Updates to follow within 15 minutes.
  • Update (01/06/2018 01:15): No new information available. Updates to follow within 15 minutes.
  • Update (01/06/2018 01:45): Our fibre provider, Zayo, are still attempting to locate the exact location of the fibre break. It has however been narrowed down significantly.
  • Update (01/06/2018 02:00: No new information available. Updates to follow within 15 minutes.
  • Update (01/06/2018 02:15: The location of the fibre break has been narrowed down further. We are awaiting a further update from Zayo.
  • Update (01/06/2018 02:30: No new information available. Updates to follow within 15 minutes.
  • Update (01/06/2018 02:45: The fibre cut has been located and the damage is being assessed now. Zayo note that from the looks of things, there are two 96 count under ground cables that appear to be damaged.
  • Update (01/06/2018 03:00: No new information available. Updates to follow within 15 minutes.
  • Update (01/06/2018 03:15: No new information available. Updates to follow within 15 minutes.
  • Update (01/06/2018 03:30: No new information available. Updates to follow within 15 minutes.
  • Update (01/06/2018 03:35: Zayo have updated: Engineers are still on the ground trying to locate the exact damage location. The Cable crew is en route with an ETA of 4:00 AM BST. The construction crew is en route with an ETA of 5:00 AM BST. Further updates to follow as more information is available.
  • Update (01/06/2018 03:56: No new information available. Updates to follow within 34 minutes.
  • Update (01/06/2018 04:16: After walking the span where the damage should be the OSP crew were unable to locate any sign of construction or damage that would cause the outage. The OSP are now going to open the SEPā€™s on each side of the suspected damage and trace the exact distance in order to give them the exact location of the physical damage. Further updates to follow as more information is available.
  • Update (01/06/2018 04:30: No new information available. Updates to follow within 30 minutes.
  • Update (01/06/2018 05:03: OSP have identified fibers between the Computer Centre and one of their Splice Enclosures. The location matches the 1.2 km from another one of their SEPs. OSP is currently looking for signs any visible damage. As it is now starting to get light outside, it should assist in finding the damage. Updates to follow within 30 minutes.
  • Update (01/06/2018 05:30: No new information available. Updates to follow within 30 minutes.
  • Update (01/06/2018 05:54: The Zayo OSP has located the damage in a field where it appears a trencher has hit the cable and duct severing the full cable. The crew are currently working to devise a plan to get into the joints on each side of the damage as well as what they will do to repair the conduit. Unfortunately, this is in the middle of a field so there does not look to be spare duct run in the area. Once a plan has been devised we will share the plan as well as push our OSP for a potential estimated time to repair. Updates to follow within 36 minutes.
  • Update (01/06/2018 06:08: The Zayo OSP crew has identified the location of the cut, there is a 5" (five inch) micro trench that is approximately 2 meters in depth, that has been dug across a field our fiber traverses. This trench has severed multiple conduits and completely severed their cables.f 2m. . Zayo is currently assessing the damage to find the nearest chambers with the cabler team in an effort to identify the quickest possible restoration, whether that be a temporary (section throw/splice) or permanent cable pull/splice. The location is in the middle of a farmers field which has added time to our ability to locate. Updates to follow within 30 minutes.
  • Update (01/06/2018 06:40: No new information available. Updates to follow within 30 minutes.
  • Update (01/06/2018 07:02: No new information available. Updates to follow within 30 minutes.
  • Update (01/06/2018 07:20: Zayo have found the 2 chambers either side of the fibre fault and found some slack cable but the chambers are 900 and 1200m away. The preferred option is to dig and pull the slack to create 1 extra joint, if they are unable to do this they may have to add extra joints. They are currently trying to gain access to the fields due to multiple ownership to access areas to where the fault is located but are in conversation with the landowners to get to the fault. Updates to follow within 30 minutes.
  • Update (01/06/2018 07:44: No new information available. Updates to follow within 30 minutes.
  • Update (01/06/2018 08:02: OSP is currently excavating in the field to try get to the cables, and sub-duct. They are doing this to try, and pull the slack cable through. Updates to follow within 30 minutes.
  • Update (01/06/2018 08:30: No new information available. Updates to follow within 30 minutes.
  • Update (01/06/2018 08:50: No new information available. Updates to follow within 30 minutes.
  • Update (01/06/2018 09:20): The excavation continues. We will update you as soon as we any new information.
  • Update (01/06/2018 09:50): Civil crews are still working to complete excavation at this time. Crews expect to be completed soon, and should be pulling slack on the cable to prep for splicing. More updates to follow.
  • Update (01/06/2018 10:00): No new information available. Updates to follow within 30 minutes.
  • Update (01/06/2018 10:30): Excavation has been completed, but the cable did not have enough slack to pull through. Crews are now prepping a new 50m strap cable between the 2 exposed ends of the cable. More updates to follow.
  • Update (01/06/2018 10:55): Fibre engineers estimate a repair schedule of 1-7 hours, with a possible ETA of 18:00
  • Update (01/06/2018 11:20): Fibre engineers intend to run a temporary fix over ground splicing to commence around 12:00 for services to start coming back up.
  • Update (01/06/2018 11:40): Crews have completed prepping the cable into the joints and are now taking the cable over to the excavation location to be prepped and spliced. We estimate we will begin splicing at 12 GMT.
  • Update (01/06/2018 12:00): No new information available. Updates to follow within 30 minutes.
  • Update (01/06/2018 12:30): No new information available. Updates to follow within 30 minutes.
  • Update (01/06/2018 12:40): Crews are nearing completion of all the cable prep and should begin splicing in 30 minutes. Will provide an additional update when splicing begins.
  • Update (01/06/2018 12:45): We have partially restored DNS services for customers. Please note that DNS responses may be intermittent and slower than usual.
  • Update (01/06/2018 13:00): No new information available. Updates to follow within 30 minutes.
  • Update (01/06/2018 13:20): Crews are now starting to splice the cable. You will begin to see services restore as splicing continues. No ETR at this time. Will provide additional updates as splicing continues.
  • Update (01/06/2018 13:45): Connectivity has been restored to our DC 8 (St. Asaph) facility. A temporary overground fix has been put in place. Please note that service is still at risk and there are likely to be interruptions.
  • Update (01/06/2018 14:15): Repair work continues on a proper fix for the fibre cuts. At the moment, both of our paths are back online albeit with a temporary fix. We will provide further updates within the next 60 minutes.

So what happened?

1 Like

Fucking heā€™ll I was on the edge of my seat reading that. Gripping or what?!

At least it wasnā€™t Mrs C_S blocking the website using the parental controls in an effort to make interact with the familyā€¦

2 Likes

And who said IT people are tedious dullards?

5 Likes