[HamWAN PSDR] Mesh backbone

[Bart Kus]

On 3/12/2013 9:35 PM, ZPO wrote:
>> I view the NW-MESH and HamWAN projects as complimentary.  HamWAN in
>> general and the PSDR component in particular makes an excellent
>> backbone transport network to move data around the wider area.
>> NW-MESH works well to cover the last mile.
>>
>>
>> This is where I think HamWAN is being sold short, since it also works great
>> on the last mile.  Reason being is HamWAN sites are all planned to be
>> non-ground-level, so LoS probability increases which makes last mile happy.
>> Your trip may be physically longer by a few miles, but the latency and speed
>> will still be good, if not better when compared to multiple ground hops for
>> the same distance.  In close-proximity situations, the end-user-site to
>> end-user-site direct connectivity of NW-MESH I believe provides an
>> optimization.  A COUPLE hops of MeSh might indeed be superior to traversing
>> a busy sector.
>>
> I don't think I'm selling HamWAN short, just looking at the likely
> timelines to have those cell-type sectors densely deployed and seeing
> it stretch into years if ever.  It is good to have multiple things in
> the network gene pool.  Let the use cases shake themselves out.  My
> thought process is that a few clubs with advantaged sites can form the
> initial connections from MESHland to the PSDR.  I'm being brief, this
> makes a lot more sense with a whiteboard.

That's a common misconception, that the cell sites need to be densely 
populated.  We're not like the cellular telcos you are familiar with who 
have to cater to tiny antennas in people's pockets that run barely any 
power, and shoot through buildings.  We're targeting fixed installs 
which can provide 30dBi gain and run 1W of power, via LoS links.  The 
projected coverage radius of a cell site in point-to-multipoint mode is 
25 miles.  Just a handful of those cover much of the Puget Sound.  We're 
not talking years.  Improving coverage would be an ongoing effort, sure, 
but to initially blanket the whole Puget Sound is not years.

>
> Maybe we can grab a cold beer at SEAPAC and I'll tell you the joys of
> trying to tie together US and NATO networks in Astan.  All kinds of
> messiness.

Sounds like a good time.

> Breaking between these two paragraphs as this is one of the first
> spots where the ethos of HamWAN and the MESH programs (the one here
> and others) tends to differ.  There is no "constitution" in MESHland.
> That is by design.  There are sets of interoperability recommendations
> that can be thought of as barebones versions of the RFCs underpinning
> the global Internet.  I don't particularly care what hardware somebody
> brings to the party as long as it has the right interfaces and can
> speak the right protocols.  This allows the network to grow
> organically with each node extending the network a bit more.  It also
> tends to make things very resilient.

Yeah, all the constitution/organizational business in HamWAN land is so 
that we can channel money in such a way that noone gets offended.  It 
would be good if HamWAN were hardware independent, but WiMAX/LTE gear is 
just too expensive to make this a reality.  So we've taken a compromise 
on the hardware front.  Technically HamWAN doesn't care what hardware 
you use either as long as you speak the protocols, but the protocols 
happen to be offered by one vendor (right now).

More deeply though, I believe it is the lack of centralized funding that 
keeps projects like SeattleWireless from taking off.  The investment is 
made either on the ground by individuals buying nodes, which have a high 
attenuation path between each other, or by a handful of enlightened 
individuals who seek height, but for a lack of structured broad funding, 
run out of personal money to really build and sustain something large.  
HamWAN pools the financial resources of the larger community and invests 
where the biggest bang for the buck exists: at high, strategic, 
locations.  Such a communal investment naturally gives rise to 
expectations (on part of the donors) to have it protected, and therefore 
the need for the HamWAN organization which maintains the investment and 
keeps it in good health.

The raw MESH approach does have theoretical merit, but the window of 
optimal behavior is quite small.  Too low a user density, and you can't 
see each other to MESH.  Too high a user density, and you've got a slow 
or unusable network.  There exists a small window of just the right user 
density that allows a MESH network to work properly, but the dependency 
on such a small window is what gives me the impression of it being a 
fragile design.

In the HamWAN design, a low user density is best served by sparse, tall 
sites.  Increased density (which comes with increased funding) is met 
with more numerous site deployments, each having a smaller coverage 
area.  This model scales continually as the network evolves in either 
direction (bigger or smaller) through time.

In terms of the resiliency, what's stopping a NW-MESH node from 
announcing a whole bunch of nonsense routes and hijacking traffic?

>
>> This is where I think you, Rob, and the rest of the NW-MESH team need to get
>> together with the HamWAN people to spell out exactly what it is that each of
>> us is trying to accomplish.  Do you have a mission statement you can share?
>> Do you have a favorite low-noise public meeting location with power + wifi?
>> Starbucks is sounding likely.
>>
>>
>>    I'd like to see some more detail
>> from Bart on what he sees as the large issues with bidirectional
>> interconnection (I think I know most of them) or simply using the PSDR
>> as transport via tunneling between access nodes.
>>
>>
>> So even though it's possible for HamWAN to act as a dumb pipe for NW-MESH,
>> this is not a fair arrangement for HamWAN.  If this were a peering agreement
>> at an Internet exchange, it would be considered a violation of the terms of
>> service set out in every Internet exchange I'm aware of.  A fair peering
>> agreement is where each network moves equal amounts of data to and from a
>> peer, where said traffic is between clients of each network.  This means not
>> via a peer network, and the traffic is from/to clients of the same
>> originating network.
>>
> This concept probably needs to be fleshed out a bit.  I didn't
> adequately define it in my initial post.  Better with a dry erase
> board..  My goal would be to have every node in MESHland offering
> services to the wider network do so via a 44-net address which will be
> advertised as an HNA4 on the network.  This is easy to link into
> HamWAN at interconnection points via BGP.  That completely isolates
> the internal 10.x.y.z/8 2.4GHz addressing (which isn't a viable
> long-term solution for lots of reasons) from HamWAN.

OK, but this doesn't allow for the MESH nodes to talk to each other via 
HamWAN should their ground links fail, right?  Seems kinda limited in 
utility.  Will these 44-net addresses be advertised to the Internet?  
Individuals getting allocations of 1-16 IPs directly from AMPR won't be 
allowed to announce.  You need to aggregate somehow up to at least a /24 
to qualify.  This in most cases implies suballocating from an upstream 
carrier who is announcing for you. That's the HamWAN plan for its 
users.  What's the plan for NW-MESH users?

What IS the viable long-term solution for the NW-MESH internal 
addressing if 10/8 is not it?


>    This is a good
> thing.  On the other hand, there are some functions and features which
> add value to MESHland that require direct communication with OLSR ETX
> values intact, and routing staying on 10.0.0.0/8.  Some folks may
> choose to not go with 44-net addresses and advertise their services to
> the wider world.  Someone traveling out of area may want their MESH
> devices to transparently link up with other networks.

For the HamWAN mobility case, I'd like to have other networks with their 
own 44-blocks all dialed into a VPN to support the exchange of mobile 
traffic.  That means if a user leaves their HamWANOregon network and 
carries a sub-allocation of his 44.x.y.z/28 or whatever, and then joins 
the HamWANWashington network, his personal subnet is still fully 
routable on the Internet and via RF by way of HamWANOregon and 
HamWANWashington exchanging that route via VPN, even though the /28 
announcement would be too small for the Internet and would probably 
violate BGP route filters with our respective ISPs.  This way noone 
needs to compromise.  Permanent re-location of networks might cause some 
excessive traffic though, so re-addressing of the user's subnet might 
have to be done.  But that's where DNS makes things nice.

> The statement " If this were a peering agreement at an Internet
> exchange, it would be considered a violation of the terms of service
> set out in every Internet exchange I'm aware of" reflects a commercial
> service provider mindset.  This is not such an environment.

Well, the commercial rules arose out of general principles of fairness.  
I'm not suggesting money change hands to offset traffic imbalances, but 
I would like to make the traffic exchange fair. What "fair" means 
exactly is a complicated thing, and we need to hold a few discussions 
about it I'm sure.

>    If the
> ham community as a whole contributes to fund, build, and support
> HamWAN then HamWAN needs to accept uses that may not fit its model.

In reality, HamWAN donors will be driving the creation of the network.  
It is to them that I feel primarily beholden, not the general ham 
community.  For example, if sector users are finding their speeds going 
to zero due to inter-MESH traffic, HamWAN will likely react to make the 
service work for the people who made it happen in the first place.  I 
consider this a fair approach, although it is certainly not an 
everyone-is-equal approach.  You don't face this problem in NW-MESH 
since you're not centralizing your funding, so everyone can be equal.  
But on the flip side of the NW-MESH approach, any joker can come online 
and deny the network for others, no?  How do you deal with that kind of 
situation?  In the HamWAN case once abusive behavior is identified, the 
offending client can be booted from the network.

>    I
> don't expect there will be huge amounts of traffic flowing via
> tunneled connections, but it is an important feature.

I'd like to see detailed models on this.  What sticks in my memory is 
the Auburn meeting I attended, and just the one room of routers (20-30 
of them maybe?) produced so much traffic that you could barely get a 
ping through the MESH there.  I expect they were all running 54Mbit to 
boot, due to their close proximity.  Sadly noone was able to do an 
over-the-air protocol analysis to see what was going on.

>    What I don't
> want to see happen is money, site availability, and goodwill expended
> needlessly operating two parallel networks.  There will be compromise
> and adaptation on both sides.

Yup!

>
> The most important thing in such an interconnection is harmonizing
> DSCP marking for QoS.  If we want to get very granular and complex,
> DISA UCR 2013 (
> http://www.disa.mil/Services/Network-Services/UCCO/UCR-2013-Draft )
> provides a detailed setup (section6 -
> http://www.disa.mil/Services/Network-Services/UCCO/~/media/Files/DISA/Services/UCCO/UCR2013-Draft/06UCR2013Section6.pdf
> ).  In the long run, as long as both domains use the same QoS, things
> are manageable.  I've played the game where DSCP tags are remarked on
> traffic ingress/egress.  It isn't fun and generally ends up breaking.
>

Yup!  And QoS is an open engineering problem on the HamWAN side right 
now.  I don't think the IPv4 DS field will be the answer either.  We may 
actually be looking at rolling VLANs to solve this without compromise.  
The problem is we actually want to extend QoS down to the clients' 
internal networks so that your VoIP phone for example will have a higher 
QoS than your PC.  At the same time, it'd be handy for the same PC to 
normally ride a "normal" QoS tier, but when you know you're gonna move 
100GB of data and you're not in a hurry, that same PC ought to declare 
to the network "hey, use a best-effort QoS".  This may not be 
implementable given current technology, but we'll shoot for the stars 
and hopefully reach the moon.

Have not seen that mil spec.  Looks daunting.  :)

>   --snipping the BCWARN stuff since it isn't germane --

*gasp!*

> In most cases, the difference between Part97 and Part15 is a state of 
> mind. It is only when you start using frequencies outside the ISM/UNII 
> bands or running higher EIRPs on point-to-multipoint links that things 
> become definitely Part97. For example, a 5.8GHz PtP link between two 
> sites running 500mW radios and 26dBi antennas could be considered 
> Part15 or Part97.

This is not the case in 5GHz land.  EIRP is limited to 1W under U-NII.  
Anything beyond that and you gotta claim part 97 (or possibly ISM?).  I 
have not looked at 2.4GHz, but you may want to. I suspect the rules will 
be similar.  See our Spectrum Allocation 
<https://www.hamwan.org/t/tiki-index.php?page=Spectrum+Allocation&structure=HamWAN> 
page for a detailed to-the-point analysis of the official publications 
with sources cited.  Apologies in advance for the wide format.

>    There may be value in considering it
> Part15 as we may want to run the link with encryption or perhaps run
> an encrypted tunnel for a served agency through the link.  In such an
> arrangement, we might get the served agency to fund the link, we
> install/maintain it, and they get a portion of the available bandwidth
> as a backup communications channel.  That isn't a bad deal for any of
> the involved parties.  Such a setup is being mulled now for one
> group/area in particular.
>
> Probably the best idea is to continue discussions at an exploratory
> level and then spend some time with a couple dry-erase boards at
> SEAPAC.  I will be there Friday-Monday (driving back Monday).  We can
> enjoy some frosty beverages and dry-erase our way through most of the
> challenges.
>

I agree on the white boarding, but the timeline you propose is really 
far away.  It's going to become harder to make any necessary changes on 
HamWAN/NW-MESH as the networks grow over the next few months.  I'd like 
to see us reach some agreements well before SeaPac.

--Bart


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.hamwan.net/pipermail/psdr/attachments/20130313/e1ce2af0/attachment.html>