[HamWAN PSDR] Westin maintenance @ 2:30PM PST

[Tom Hayward]

I'm just trying to share details about the network. I figure that's what
people want to see here.

We recently migrated Seattle-ER1 from an RB2011 to a VM on an ESXi host.
This was done for a few reasons, but most importantly, crypto is much
faster in the new x86 CPU than in the RB2011. This will allow us to support
more VPN terminations. (I believe the RB2011 maxed out around 18 Mbps of
VPN traffic.) Seattle-SRV1 is another VM on the same host.

Today's outage was for a reboot of the ESXi host. Bart is more familiar
with the details, but I understand the reboot was required to get the VMs
to show up properly in vCenter server. This momentarily took out the two
VMs, Seattle-ER1 and Seattle-SRV1. During the outage, the network re-routes
around Seattle-ER1, but services only provided on Seattle-SRV1 remains
inaccessible. After a few minutes, everything boots back up and service is
restored.

The topology at the Westin looks something like this:

Seattle-ESXi
    Seattle-ER1
    Seattle-SRV1
Seattle-QueenAnne PtP

Each of these systems participates in OSPF, so if we have a failure of
Seattle-ER1, Seattle-SRV1 will begin routing all traffic through Queen
Anne. My guess is this will eventually make its way back to the Internet
via our Tukwila datacenter. (The other possibility is exit through
Corvallis via a VPN tunnel.)

For VM failover, we'd need a second ESXi host and configuration for
automatic failover. My preference is that we make services redundant rather
than systems, so if a VM or ESXi itself fail, another system at another
site continues to provide the service. This is how we have DNS configured,
for instance. I bet no one noticed any downtime for DNS during this
maintenance event. Unfortunately, DNS is a lot simpler to make redundant
than a SQL-backed website like hawan.org.

Tom


On Fri, May 22, 2015 at 3:31 PM, John D. Hays <john at hays.org> wrote:

> You did note the smiley face?
>
> BTW, two MikroTik routers in parallel can failover nicely.
>
> On Fri, May 22, 2015 at 2:36 PM, Tom Hayward <tom at tomh.us> wrote:
>
>> The network itself has automatic failover and redundancy. The website
>> hamwan.org (non-redundantly hosted on the Westin server) does not. IIRC,
>> that's the only service at Westin that is not duplicated elsewhere.
>>
>> Tom
>>
>>
>> On Fri, May 22, 2015 at 2:34 PM, John D. Hays <john at hays.org> wrote:
>>
>>> What? No automatic failover and redundancy!!!
>>>
>>> :)
>>>
>>>
>>> On Fri, May 22, 2015 at 2:22 PM, Bart Kus <me at bartk.us> wrote:
>>>
>>>> Hello,
>>>>
>>>> I'll be doing a reboot of the Westin server @ 2:30PM PST today. This
>>>> means all traffic and services there will stop for a few minutes.  I'll
>>>> send a second email when it's all completed.
>>>>
>>>> --Bart
>>>>
>>>> _______________________________________________
>>>> PSDR mailing list
>>>> PSDR at hamwan.org
>>>> http://mail.hamwan.net/mailman/listinfo/psdr
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> ------------------------------
>>> John D. Hays
>>> K7VE
>>>
>>> PO Box 1223, Edmonds, WA 98020-1223
>>> <http://k7ve.org/blog>  <http://twitter.com/#!/john_hays>
>>> <http://www.facebook.com/john.d.hays>
>>>
>>>
>>> _______________________________________________
>>> PSDR mailing list
>>> PSDR at hamwan.org
>>> http://mail.hamwan.net/mailman/listinfo/psdr
>>>
>>>
>>
>> _______________________________________________
>> PSDR mailing list
>> PSDR at hamwan.org
>> http://mail.hamwan.net/mailman/listinfo/psdr
>>
>>
>
>
> --
>
> ------------------------------
> John D. Hays
> K7VE
>
> PO Box 1223, Edmonds, WA 98020-1223
> <http://k7ve.org/blog>  <http://twitter.com/#!/john_hays>
> <http://www.facebook.com/john.d.hays>
>
>
> _______________________________________________
> PSDR mailing list
> PSDR at hamwan.org
> http://mail.hamwan.net/mailman/listinfo/psdr
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.hamwan.net/pipermail/psdr/attachments/20150522/a8415212/attachment-0001.html>