[HamWAN PSDR] Capitol Peak cable system failure

Sun Feb 7 14:18:53 PST 2021

I think it's fair to assign majority fault to installation technique.  
However, since the cable bundle was not entirely torn apart, I wonder if 
a slightly stronger cable would have remained operational.  I wonder if 
the water resisting tape would have prevented corrosion and failure at 
the PoE injectors.

There will always be sections of cable (eg: literal corner cases) where 
it won't be possible for it to have steel above it.  This is where 
better equipment selection can keep us online longer.

If you look at this through the economic lens, we have an abundance of 
available dollars, but a shortage of available man-hours.  The unit cost 
increase isn't that great either.  Our normal TC-Pro costs are 
$150/box.  The new CAT6 would be $200/box.  Meanwhile we just spent $50 
of gas (for me alone), and time-value of me & Dale to do the inspection 
visit.  We haven't even paid the cost of replacement visits yet.  Who 
will be willing to climb the tower in the winter?  Who will be willing 
to do ground crew in the snow?

If you look at this through a site reliability lens, since each cable is 
a single point of failure, there isn't much else to be done except 
maximize the cable's reliability.  The modems cannot take 2 cables for 
redundancy.  We also incur reputational damage every time we have a 
reliability issue like this.  That is harder to quantify and regain.

I agree with your point that all sites may not need this (eg: Beacon, 
QueenAnne), but I was implicitly referring to our tower sites.

I also agree with your point about site inspections, and think we should 
do them on a schedule, but that's a separate process improvement to the 
stuff I wanted to focus on here.

What makes this site susceptible to ice fall damage is fairly constant 
high winds, moist air directly from the pacific, and the fact that it's 
pretty tall (looks like 180ft?) which allows the ice to build up a lot 
of speed before it hits.

In my earlier UPS email I broadly stated I would like us to focus on 
reliability in 2021.  The power system improvements are direction #1, 
better hardware and installation techniques are #2, regularity of 
inspections #3, and monitoring / management software improvements will 
be #4.  These aren't in priority-order, just enumerated to keep them 
distinct and delegatable.

--Bart

On 2/7/2021 1:24 PM, Rob Salsgiver wrote:
>
> I’m not opposed to exploring better / tougher alternatives, but I 
> wonder how much is due to the quality of our previous selections vs 
> our installation experience and knowledge level at some of these 
> earlier sites.  We’ve done pretty well in most installations for 
> several years now.  If our existing equipment selections are 
> sufficient in most cases as long as we incorporate lessons learned on 
> installation specs (under steel members, ice risks, etc), we may not 
> need to incur the added expense for every site going forward. This is 
> not a slam against anyone who’s done prior installs, as we all 
> continue to learn even today.  It is what it is.
>
> That said, having tougher alternatives already explored and tested 
> gives us a path if the above proves to be malarkey.
>
> On future visits to existing sites we should have a little added focus 
> on photos/examination of the condition of our existing installs and 
> put them in file for the record to keep track of how well things are 
> doing over time at each location.  I think this is pretty normal when 
> we go on site, but this is an expensive example to reinforce the idea.
>
> Is there anything notable about this site that makes it worse for ice 
> fall risks?  I suspect not as I’ve seen ice damage examples on 
> non-HamWAN equipment on almost every site visit, but worth asking.
>
> Cheers,
>
> Rob
>
> *From:*PSDR [mailto:psdr-bounces at hamwan.org] *On Behalf Of *Bart Kus
> *Sent:* Sunday, February 7, 2021 1:05 PM
> *To:* Puget Sound Data Ring
> *Subject:* [HamWAN PSDR] Capitol Peak cable system failure
>
> Hello,
>
> I recently noticed we had multiple modems offline at Capitol Peak.  
> Dale & I went to visit Capitol Peak a few days ago to inspect the 
> situation.  This is what we found:
>
> https://photos.app.goo.gl/KXajGkDGLsW41dQz7 
> <https://photos.app.goo.gl/KXajGkDGLsW41dQz7>
>
> It's a little hard to see in the photos, but our cable bus has been 
> ripped off the tower.  When examining the cabinet where the cables 
> terminate, I found the cables to be wet, and we discovered water 
> pooling in the bottom of the cabinet:
>
> https://photos.app.goo.gl/kWwAWs2FvA6ghhC28 
> <https://photos.app.goo.gl/kWwAWs2FvA6ghhC28>
>
> So not only are the cables dangling, they're also broken to the point 
> where we lost communications and are now taking on water.  This was 
> likely caused by ice falling onto the cables, since that bottom bundle 
> was mounted in such a way that it was protruding inside the tower, 
> instead of being under the cover of angle iron like the rest of the 
> horizontal coax nearby. It's hard to pin down exactly when this 
> happened, but the data I've seen points to Jan 23rd for S3 falling 
> offline.
>
> There is another cable of ours that had its support removed about 80ft 
> up the tower.  It feeds the BawFaw link.  The damage was again along a 
> horizontal run, from the cable ladder to the dish on the opposite side 
> of the tower from the cable ladder.  We typically run these under the 
> steel members, so I'm not sure what the failure there was.  I wasn't 
> able to take good pictures of the dangling cable.
>
> The whole bottom bundle is impacting on the edge of angle iron 
> whenever the wind blows.  This has likely cut through the cables, 
> although poor weather didn't allow close visual inspection.
>
> I think this situation calls for some engineering changes in 
> addressing this and other future deploys:
>
> 1) All horizontal runs must be protected from falling ice, by having 
> steel members above them.
> 2) We should evaluate the use of tougher cable that is more resistant 
> to physical damage.
> 3) We should evaluate the use of cable that is water-resistant when it 
> does get breached.
> 4) We should evaluate the use of drip-loops (removing a section of 
> jacket) before the cable reaches expensive equipment.
> 5) We should measure the forces required to remove beam clamps and 
> establish minimum torque requirements.
> 6) A camera system may have spotted these cable failures before we 
> experienced communication outages.  Deployment has been approved.
>
> There is some good info on addressing (3) on this page, thanks to 
> Dale's research:
>
> https://www.truecable.com/blogs/cable-academy/into-the-great-outdoors-running-ethernet-cable-outside 
> <https://www.truecable.com/blogs/cable-academy/into-the-great-outdoors-running-ethernet-cable-outside>
>
> To address (2) I think we can look at stepping up to CAT6, which is 
> AWG23 instead of AWG24, and has a larger outer diameter.  The presence 
> of a spline will also make the cable stronger.
>
> This leaves us with a cushion problem, since our cushions are only 
> designed for 1/4" cable and CAT6 is thicker than that. It seems we 
> have gotten exceedingly lucky here!  Valmont has just recently 
> released a new cushion design that accepts 7 runs of 4mm-9mm cable!  
> The part # is BCU158M7.
>
>
>
> We also need to make sure these new cable selections will be 
> compatible with our current-spec EZ-RJ45 ends.
>
> I'm going to send a separate VOTE email to approve the purchasing of 
> samples of this new hardware, so I can verify it will all work 
> together.  Let's use this thread to discuss the problems, and if 
> anyone has other/better solutions please share here.
>
> --Bart
>
>
> _______________________________________________
> PSDR mailing list
> PSDR at hamwan.org
> http://mail.hamwan.net/mailman/listinfo/psdr

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.hamwan.net/pipermail/psdr/attachments/20210207/9e81ab75/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 29114 bytes
Desc: not available
URL: <http://mail.hamwan.net/pipermail/psdr/attachments/20210207/9e81ab75/attachment.jpg>