[HamWAN PSDR] Capitol Peak cable system failure

Sun Feb 7 14:47:07 PST 2021

I totally agree with you.  Nm5f,  New Mexico. 

Sent from my iPad

> On Feb 7, 2021, at 3:19 PM, Bart Kus <me at bartk.us> wrote:
> 
>  I think it's fair to assign majority fault to installation technique.  However, since the cable bundle was not entirely torn apart, I wonder if a slightly stronger cable would have remained operational.  I wonder if the water resisting tape would have prevented corrosion and failure at the PoE injectors.
> 
> There will always be sections of cable (eg: literal corner cases) where it won't be possible for it to have steel above it.  This is where better equipment selection can keep us online longer.
> 
> If you look at this through the economic lens, we have an abundance of available dollars, but a shortage of available man-hours.  The unit cost increase isn't that great either.  Our normal TC-Pro costs are $150/box.  The new CAT6 would be $200/box.  Meanwhile we just spent $50 of gas (for me alone), and time-value of me & Dale to do the inspection visit.  We haven't even paid the cost of replacement visits yet.  Who will be willing to climb the tower in the winter?  Who will be willing to do ground crew in the snow?
> 
> If you look at this through a site reliability lens, since each cable is a single point of failure, there isn't much else to be done except maximize the cable's reliability.  The modems cannot take 2 cables for redundancy.  We also incur reputational damage every time we have a reliability issue like this.  That is harder to quantify and regain.
> 
> I agree with your point that all sites may not need this (eg: Beacon, QueenAnne), but I was implicitly referring to our tower sites.
> 
> I also agree with your point about site inspections, and think we should do them on a schedule, but that's a separate process improvement to the stuff I wanted to focus on here.
> 
> What makes this site susceptible to ice fall damage is fairly constant high winds, moist air directly from the pacific, and the fact that it's pretty tall (looks like 180ft?) which allows the ice to build up a lot of speed before it hits.
> 
> In my earlier UPS email I broadly stated I would like us to focus on reliability in 2021.  The power system improvements are direction #1, better hardware and installation techniques are #2, regularity of inspections #3, and monitoring / management software improvements will be #4.  These aren't in priority-order, just enumerated to keep them distinct and delegatable.
> 
> --Bart
> 
> 
> On 2/7/2021 1:24 PM, Rob Salsgiver wrote:
>> I’m not opposed to exploring better / tougher alternatives, but I wonder how much is due to the quality of our previous selections vs our installation experience and knowledge level at some of these earlier sites.  We’ve done pretty well in most installations for several years now.  If our existing equipment selections are sufficient in most cases as long as we incorporate lessons learned on installation specs (under steel members, ice risks, etc), we may not need to incur the added expense for every site going forward.  This is not a slam against anyone who’s done prior installs, as we all continue to learn even today.  It is what it is.
>>  
>> That said, having tougher alternatives already explored and tested gives us a path if the above proves to be malarkey.
>>  
>> On future visits to existing sites we should have a little added focus on photos/examination of the condition of our existing installs and put them in file for the record to keep track of how well things are doing over time at each location.  I think this is pretty normal when we go on site, but this is an expensive example to reinforce the idea.
>>  
>> Is there anything notable about this site that makes it worse for ice fall risks?  I suspect not as I’ve seen ice damage examples on non-HamWAN equipment on almost every site visit, but worth asking.
>>  
>> Cheers,
>> Rob
>>  
>> From: PSDR [mailto:psdr-bounces at hamwan.org] On Behalf Of Bart Kus
>> Sent: Sunday, February 7, 2021 1:05 PM
>> To: Puget Sound Data Ring
>> Subject: [HamWAN PSDR] Capitol Peak cable system failure
>>  
>> Hello,
>> 
>> I recently noticed we had multiple modems offline at Capitol Peak.  Dale & I went to visit Capitol Peak a few days ago to inspect the situation.  This is what we found:
>> 
>> https://photos.app.goo.gl/KXajGkDGLsW41dQz7
>> 
>> It's a little hard to see in the photos, but our cable bus has been ripped off the tower.  When examining the cabinet where the cables terminate, I found the cables to be wet, and we discovered water pooling in the bottom of the cabinet:
>> 
>> https://photos.app.goo.gl/kWwAWs2FvA6ghhC28
>> 
>> So not only are the cables dangling, they're also broken to the point where we lost communications and are now taking on water.  This was likely caused by ice falling onto the cables, since that bottom bundle was mounted in such a way that it was protruding inside the tower, instead of being under the cover of angle iron like the rest of the horizontal coax nearby.  It's hard to pin down exactly when this happened, but the data I've seen points to Jan 23rd for S3 falling offline.
>> 
>> There is another cable of ours that had its support removed about 80ft up the tower.  It feeds the BawFaw link.  The damage was again along a horizontal run, from the cable ladder to the dish on the opposite side of the tower from the cable ladder.  We typically run these under the steel members, so I'm not sure what the failure there was.  I wasn't able to take good pictures of the dangling cable.
>> 
>> The whole bottom bundle is impacting on the edge of angle iron whenever the wind blows.  This has likely cut through the cables, although poor weather didn't allow close visual inspection.
>> 
>> I think this situation calls for some engineering changes in addressing this and other future deploys:
>> 
>> 1) All horizontal runs must be protected from falling ice, by having steel members above them.
>> 2) We should evaluate the use of tougher cable that is more resistant to physical damage.
>> 3) We should evaluate the use of cable that is water-resistant when it does get breached.
>> 4) We should evaluate the use of drip-loops (removing a section of jacket) before the cable reaches expensive equipment.
>> 5) We should measure the forces required to remove beam clamps and establish minimum torque requirements.
>> 6) A camera system may have spotted these cable failures before we experienced communication outages.  Deployment has been approved.
>> 
>> There is some good info on addressing (3) on this page, thanks to Dale's research:
>> 
>> https://www.truecable.com/blogs/cable-academy/into-the-great-outdoors-running-ethernet-cable-outside
>> 
>> To address (2) I think we can look at stepping up to CAT6, which is AWG23 instead of AWG24, and has a larger outer diameter.  The presence of a spline will also make the cable stronger.
>> 
>> This leaves us with a cushion problem, since our cushions are only designed for 1/4" cable and CAT6 is thicker than that.  It seems we have gotten exceedingly lucky here!  Valmont has just recently released a new cushion design that accepts 7 runs of 4mm-9mm cable!  The part # is BCU158M7.
>> 
>> <image001.jpg>
>> 
>> 
>> We also need to make sure these new cable selections will be compatible with our current-spec EZ-RJ45 ends.
>> 
>> I'm going to send a separate VOTE email to approve the purchasing of samples of this new hardware, so I can verify it will all work together.  Let's use this thread to discuss the problems, and if anyone has other/better solutions please share here.
>> 
>> --Bart
>> 
>> 
>> 
>> _______________________________________________
>> PSDR mailing list
>> PSDR at hamwan.org
>> http://mail.hamwan.net/mailman/listinfo/psdr
> 
> _______________________________________________
> PSDR mailing list
> PSDR at hamwan.org
> http://mail.hamwan.net/mailman/listinfo/psdr
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.hamwan.net/pipermail/psdr/attachments/20210207/bd2e2f83/attachment.html>