[HamWAN PSDR] Mesh backbone

[ZPO]

Finally some ATUs to devote to a proper response..

On Sat, Mar 16, 2013 at 1:34 PM, Bart Kus <me at bartk.us> wrote:
> KY9K de AE7SJ
>
> Awaiting your response, but I'll address Gerard's comments here since he's
> waited quite a while.
>
>
>
> On 3/14/2013 12:01 AM, Gerard Hickey wrote:
>
>
> First of all, (Bart I am not trying to start a fight, just making an
> observation)
>
>
> No one is trying to start a fight, but we do have inherently contentious
> issues to discuss.  So let's keep a cool head and keep the communication
> going for the good of the community.
>
>

KY9K:  How about we set a starting point for the discussion.  HamWAN
and NW-MESH are and shall remain independent activities.  We may
decide to route traffic between the networks and leverage the networks
for mutual gain. If we get to a point where we have agreed
interconnection that makes HamWAN valuable to the NW-MESH activities,
I am willing to vote with my wallet and provide support to HamWAN.
One network will not fall under the command and control of the other.


> HamWAN is not a mesh network. It is an architected network with links (both
> real and planned) spread out throughout the region.
>
>
> This is where we hit the inevitable confusion over the phrase "mesh network"
> which I addressed at the top of my first response so there wouldn't be any
> such confusion.  Let me find it ...
>
> -=[ QUOTE ]=-
>
> First, let's speak the same language.
>
>
> There's a lot of confusion around the word "MESH".  I'm also not sure why
> it's always capitalized.  It's not an acronym as far as I know.  The term
> "mesh network" describes nothing more than the logical topology of a
> network.  There's a really good write-up on this at the wikipedia page.  I'm
> pretty sure that when you (and others on this email) use the phrase "mesh
> network" it is not the wikipedia definition you're intending.  It means
> something different, including:
>
> 1) Using a common RF channel
> 2) Promiscuous neighbor discovery + association
> 3) Automatic IP configuration based on MAC
> 4) Nearly open node authentication
> 5) Omnidirectional operation
> 6) WDS-style operation
>
> These are the typical traits of a SeattleWireless style mesh network (which,
> BTW, has been attempting to bootstrap itself for the last 13 years).  This
> type of definition of "mesh network" is quite a different animal from the
> canonical mesh network definition (wikipedia's, derived from network
> theory).
>
>
> The reason I want to make the distinction clear is that nearly all large
> networks in the world are indeed mesh networks, but nearly none of them
> possess the qualities of 1-6.  So when I say something like "HamWAN is a
> mesh network", I want it to be clear that I'm referring to the network
> topology only (the wikipedia definition).  I'm not sure what is the right
> word to describe the other type of network; perhaps capitalizing all the
> letters is a good differentiator after all.  :)
>
> -=[ /QUOTE ]=-
>

KY9K:  I've been around networks long enough to agree with Gerard.
While you can play semantic games with the words, the entire
networking industry knows the difference between MANET/MESH and
centrally controlled service provider networks.  Gerard is making a
very important distinction between a centrally managed infrastructure
network and an organic ad-hoc network.  Most of already participate in
centrally managed networks.  They are called "ISPs" and "Wireless
phone carriers."  Such networks have advantages in some cases, ham
radio experimentation, EMCOM, etc aren't on the list.


>
> Two of the qualities of a mesh network is self-healing and zero
> configuration.
>
>
> So, admittedly I did pull the list of the above 6 properties out of my butt.
> It's my best guess.  Is there actually a formal list somewhere?  Please
> provide it.  I don't think there's actually any clear divide between MESH or
> not, it's more like a spectrum of features that are there or not.
>
> I would argue the "zero configuration" claim is not actually true.  There's
> configuration in MESH nodes, it's just stored in the node router, much as it
> will be in HamWAN client routers.  You set things like frequency, bandwidth,
> SSID.  In fact, as I understand it, there's usually a custom firmware that's
> uploaded which comes with I don't know how many settings that I'm guessing
> are set "just right" so that things work nicely.  So here, we're not so
> different, as both client router devices will have a config set.
>
> The other way I can interpret this is that once you get your fully
> configured client router device, you simply attach it to an antenna and it
> finds and connects to the wireless network.  This is exactly the scenario
> we're enabling for HamWAN as well, it's not unique to NW-MESH.  OK, maybe
> one slight difference: in the HamWAN case there will be user credentials you
> enter into the router for it to authenticate onto the network.  But isn't
> there some OLSR key in NW-MESH?  I'm really not up to speed on that as much
> as I should be.
>
>

KY9K:  I suggest you do additional research to understand NW-MESH and
HSMM-MESH.  Perhaps then we can speak from a common base on knowledge.
 There is a driving concept of "conservation of complexity" inherent
in the design for NW-MESH/HSMM-MESH.  While there are settings that
have to be "just right", the vast majority of them are configured at
build time and hidden from the user.  There are 3-4 settings entered
by the user upon loading the firmware.  Once those are done, that node
can be taken anywhere in the area, turned on, and it will work.  In
this case, the complexity moves to the packager of the firmware.  In
other cases, such as wired linking multiple radios on multiple bands,
the complexity increases, but only for the individual who has actively
chosen to do that work.  Once done, those links benefit the entire
network without any additional action by other operators.


> Yes, HamWAN does have self-healing properties but it is an engineered
> healing mechanism rather than a intrinsic part of the technology.
>
>
> I'm failing to understand the difference between "engineered" and
> "intrinsic" here.  If we require that HamWAN run OSPF, this becomes an
> intrinsic property of HamWAN.  OLSR is REALLY close to OSPF.  I would say in
> the self-healing aspects, the networks are not very different at all.
>
>
> I also can not just by the same hardware that HamWAN is using, turn it on
> and point it to a HamWAN point of presence and have it work. There are
> specific network and routing parameters that need to be set in order to come
> up on the network.
>
>
> I addressed this "zero config" claim above, but do correct me if I'm wrong.
>
>
> Just the same as if I have a configured HamWAN installation, I could not
> move it to another part of the region, point it to another point of presence
> and be back on the network (actually there is one or two ways, but it makes
> the routing butt ugly).
>
>
> This is actually one of the scenarios we're definitely enabling.  If you
> have a static IP or subnet, it will follow you around on the network no
> matter what sector you connect to.  In one of my previous replies I also
> described how that static (and publicly routable!) address(es) can follow
> you to partner networks as well through use of VPN peering.  Nice, ya? :)
> This is why we're using beefy routers, cuz we expect the routing tables to
> be large.
>
>
>
> With these statements, I am just trying to provide a clearer distinction
> that Bart started in the second or third email of this thread. By no means
> am I trying to elevate one technology or the other as a superior technology.
> I think that there is plenty of room for every one to experiment and play
> with the various technologies.
>
>
> I whole-heartedly encourage any activity which gets hams playing with more
> modern digital stuff.  The whole concept of "superior" or "best" is relative
> to your goals.  That's why I'm trying to get some sense out of the NW-MESH
> crowd of what the goals really are.  A mission statement or something.  Like
> I've said, I think our goals are actually pretty close in most areas, and
> it's the differences I'd like to explore to see if we can minimize them or
> respect them and work on a peering model.  If we develop a clear
> understanding of each others' goals, only then we can begin the technical
> work necessary to align the networks together.
>
>

KY9K:  There isn't a mission statement, single goal, governing cabal,
or anything similarly rigidly defined in MESHland.  That is by intent.
 The purpose is to provide a set of tools to build a network that
enables meeting the goals and needs of participating individuals and
groups.

>
> I have stated since I first heard Bart speak about the HamWAN project last
> year at Summer Gathering that it works very well as a backbone technology. I
> think it compliments the NW-MESH work very well in the sense that it can tie
> together various NW-MESH networks and provide the backbone between them. And
> NW-MESH provides an economical way for hams to participate in a community
> network.
>
>
> The economics of onboarding to HamWAN (~$200) vs. NW-MESH (~$30) is a fair
> point.  I can envision a neighborhood cluster of $30 nodes sharing a $200
> uplink.  But for this to become reality, we must agree on how to align.  I'm
> starting to sound like a broken record.  :)  The initial thinking of "just
> give us a tunnel" is a raw deal for HamWAN, so we must work harder to strike
> a better balance.  I'd like HamWAN clients and MESH clients to be mutually
> directly addressable, for one.  I have other concerns about route injection
> into HamWAN, client authentication on NW-MESH and potential collisions with
> people's RFC1918 space networks since HamWAN does intend to announce routes
> into people's home networks.  I'd also like to understand the QoS, default
> gateway, DNS, IPsec and IPv6 stories.  There's a LOT of work to sort out.
> Let's not waste time in jumping to it.
>
>

KY9K:  I have never said "just give us a tunnel."  I have said that a
tunnel to link the mesh islands is a required feature.  I have also
said that my desire would be for all announced services of interest
outside NW-MESH to be sourced from a 44-net address.  That will take a
software mod to the current software.  The very good thing is that
nodes not announcing services of interest outside the NW-MESH network
don't need to make any changes to utilize those services.  Again,
conservation of complexity.  The OpenWRT builds can make this change
now without new software, but the config will be a bit more involved
than the typical "10 clicks or less" model of the HSMM-MESH software
upon which the WRT54G NW-MESH build is based.

KY9K - INTERCONNECTION - The interconnection process isn't that
difficult.  BGP with a private AS on at least the NW-MESH side with
only 44-net IPs flowing back and forth.  There will be a requirement
to group the NW-MESH addresses into a defined supernet, but I'm
confident to required coordination with John (K7VE) can be
accomplished without too much trouble.  That answers the mail on the
need to announce /24s or larger.

KY9K - QoS - DSCP with common values on both sides or a translation
table between them if HamWAN decides to do something other than DSCP
or use different values.  Having engineered, built, and maintained
some very large tactical networks; I can say that DSCP is the way to
go.  While more complicated models are possible, they provide very
little additional bang for the buck of time and effort.  DSCP is the
lingua franca of QoS across policy domains just like BGP is the
dominant exterior gateway protocol.

KY9K - DNS - Should be a bit interesting, but not something that has
to be resolved for a general theory of interconnection is hammered
out.  I've registered "hsmm.us" a while back with the intent of
getting someone in each state to grab <state>.hsmm.us and administer
it as they see fit.  "mesh.wa.hsmm.us" is the obvious place to put the
mesh stuff.  "hamwan.wa.hsmm.us" for the HamWAN stuff?  That would
easily allow for other states to build "HamWAN-like" implementations.

KY9K - IPSec - While AH is arguably allowed, ESP is an absolute no-go
on Part97 networks.  In the Part15 space, IPSec is perfectly kosher.

KY9K - IPv6 - You've been reading the mail between Gerard and I.  This
is more of a mid/long-term NW-MESH goal.  To move the wireless links
to IPv6 addressing and utilize a single 44-net IPv4 address on each
node for handling IPv4 traffic.  It removes the whole address
assignment issue.  I can remove the RFC-1918 issue as well, but that
would be a different choice.  I expect the 10.x.y.z/8 utilization on
2.4GHz will be with us for a long time.  That will remain an internal
issue to the NW-MESH network.  It won't be exported to HamWAN.

**snip**
>
> What I will offer is conference rooms at ebay in Redmond. We have plenty of
> white boards and projectors / 60 monitors for presenting so that everyone
> does not have to crowd around one laptop. We can meet pretty much most
> nights or weekends except for Tuesdays and the 2nd and 4th Monday of the
> month. This Friday and Saturday is also out since I will be at the Cascadia
> IT conference. I will also be in California the week of 3/25. Beyond that I
> think I can pretty much make anything else work.
>
>
> I'm game.  Since I work in Redmond, I can be available there any weekday
> (after work), and am willing to make weekend trips whenever others can do
> likewise.  Let's get the ball rolling here.  During the first meeting(s)
> whoever is attending should be prepared to discuss their goals and
> philosophy about these networks.
>

My current travel schedule (subject to change on short notice of
course - it has already changed twice in a week).  The 5/6-5/10 is
probably going to move and I'll end up going out to MA somewhere in
that mix.  I work in downtown Seattle, but I can drive in to work and
then head down for a meeting.

3/17-3/22 - CA
3/23 - Microhams
4/8-4/19 - CA
5/6-5/10 - CA

** snip **

Bart's earlier email.....

On 3/12/2013 9:35 PM, ZPO wrote:
>> I view the NW-MESH and HamWAN projects as complimentary.  HamWAN in
>> general and the PSDR component in particular makes an excellent
>> backbone transport network to move data around the wider area.
>> NW-MESH works well to cover the last mile.
>>
>>
>> This is where I think HamWAN is being sold short, since it also works great
>> on the last mile.  Reason being is HamWAN sites are all planned to be
>> non-ground-level, so LoS probability increases which makes last mile happy.
>> Your trip may be physically longer by a few miles, but the latency and speed
>> will still be good, if not better when compared to multiple ground hops for
>> the same distance.  In close-proximity situations, the end-user-site to
>> end-user-site direct connectivity of NW-MESH I believe provides an
>> optimization.  A COUPLE hops of MeSh might indeed be superior to traversing
>> a busy sector.
>>
> I don't think I'm selling HamWAN short, just looking at the likely
> timelines to have those cell-type sectors densely deployed and seeing
> it stretch into years if ever.  It is good to have multiple things in
> the network gene pool.  Let the use cases shake themselves out.  My
> thought process is that a few clubs with advantaged sites can form the
> initial connections from MESHland to the PSDR.  I'm being brief, this
> makes a lot more sense with a whiteboard.

AE7SJ:  That's a common misconception, that the cell sites need to be
densely populated.  We're not like the cellular telcos you are
familiar with who have to cater to tiny antennas in people's pockets
that run barely any power, and shoot through buildings.  We're
targeting fixed installs which can provide 30dBi gain and run 1W of
power, via LoS links.  The projected coverage radius of a cell site in
point-to-multipoint mode is 25 miles.  Just a handful of those cover
much of the Puget Sound.  We're not talking years.  Improving coverage
would be an ongoing effort, sure, but to initially blanket the whole
Puget Sound is not years.

KY9K: Unless you're going to cut down all the "vertical water columns"
(aka - trees), I don't see lots of 25-mile coverage areas without
bouncing around once you get past the "club gateway station" (my term)
via NW-MESH connections.  I get pushback from some individuals when I
recommend any solution that costs more than $30 to fully implement.
$200 (more like $300 when it is all said and done) is likely to result
in a much smaller uptake, especially when you add in desired donations
for the backbone itself.  A more sustainable model would likely be to
have clubs fund/own/operate the distribution nodes.  I already send
money each month to a couple companies for providing me IP services -
ISP and Verizon.  Many are likely to feel the same way.  When you
bring the existing club organization into the mix, at least you
leverage the existing rather than trying to invent new.  Those are
natural places for the interconnection between HamWAN and NW-MESH.  If
folks *want* to spend the bucks on a PtP link, they can.  If they
don't want to spend that money, they come in via NW-MESH.  NATed
services can also be implemented to let pure-users talk to things that
live out on HamWAN.

**snip**


> Breaking between these two paragraphs as this is one of the first
> spots where the ethos of HamWAN and the MESH programs (the one here
> and others) tends to differ.  There is no "constitution" in MESHland.
> That is by design.  There are sets of interoperability recommendations
> that can be thought of as barebones versions of the RFCs underpinning
> the global Internet.  I don't particularly care what hardware somebody
> brings to the party as long as it has the right interfaces and can
> speak the right protocols.  This allows the network to grow
> organically with each node extending the network a bit more.  It also
> tends to make things very resilient.

AE7SJ:  Yeah, all the constitution/organizational business in HamWAN
land is so that we can channel money in such a way that noone gets
offended.  It would be good if HamWAN were hardware independent, but
WiMAX/LTE gear is just too expensive to make this a reality.  So we've
taken a compromise on the hardware front.  Technically HamWAN doesn't
care what hardware you use either as long as you speak the protocols,
but the protocols happen to be offered by one vendor (right now).

KY9K:  What protocols do you see as required that aren't implemented
in an open vendor-neutral manner?  You've got NV2 (Mikrotik) and
AirMAX (Ubiquiti) that are both TDM protocols to solve the DCF and
hidden-node CSMA issues.  Other vendors have implemented similar
things, but I'm not as familiar with them.  The long-haul PtP links
only need to have the same equipment at both ends of the link.  The
link protocol in use only need to be supported at the single-link
level.  If you're talking about MPLS, I say run away very fast.  Works
great in dense service provider networks with dedicated (and
overworked) engineering and network management staffs - is a huge time
suck with little reward on a ham network supported by available brain
cycles.  KISS is my operating principle for ham networks.  The more
complex the network is made, the more likely it is to fail.  Establish
some open standards, implement a few links, and let it run.


AE7SJ:  More deeply though, I believe it is the lack of centralized
funding that keeps projects like SeattleWireless from taking off.  The
investment is made either on the ground by individuals buying nodes,
which have a high attenuation path between each other, or by a handful
of enlightened individuals who seek height, but for a lack of
structured broad funding, run out of personal money to really build
and sustain something large.  HamWAN pools the financial resources of
the larger community and invests where the biggest bang for the buck
exists: at high, strategic, locations.  Such a communal investment
naturally gives rise to expectations (on part of the donors) to have
it protected, and therefore the need for the HamWAN organization which
maintains the investment and keeps it in good health.

KY9K:  We've got clubs and various organizations already that can
provide support.  I'm willing to provide monetary support, but only if
I know I'm going to be able to use the network for my own needs.
Locking things away behind a service provider model will turn many
people off - me included.


AE7SJ:  The raw MESH approach does have theoretical merit, but the
window of optimal behavior is quite small.  Too low a user density,
and you can't see each other to MESH.  Too high a user density, and
you've got a slow or unusable network.  There exists a small window of
just the right user density that allows a MESH network to work
properly, but the dependency on such a small window is what gives me
the impression of it being a fragile design.

AE7SJ:  In the HamWAN design, a low user density is best served by
sparse, tall sites.  Increased density (which comes with increased
funding) is met with more numerous site deployments, each having a
smaller coverage area.  This model scales continually as the network
evolves in either direction (bigger or smaller) through time.

KY9K:  More "service provider focus".  Let lower tiers of the network
figure that out.  If density gets too high in a NW-MESH area, the
operating folks in the area can decide on their own to add some PtP or
PmP cabability to take load off the shared channel.  I'd like to get
to that problem space.  One of the key operating principles of NW-MESH
is that it pushes such engineering and problem solving down to the
lowest possible level.  It allows decisions to be made based on local
needs rather than a wide area corporate overhead level.

AE7SJ:  In terms of the resiliency, what's stopping a NW-MESH node
from announcing a whole bunch of nonsense routes and hijacking
traffic?
OK, but this doesn't allow for the MESH nodes to talk to each other
via HamWAN should their ground links fail, right?  Seems kinda limited
in utility.  Will these 44-net addresses be advertised to the
Internet?  Individuals getting allocations of 1-16 IPs directly from
AMPR won't be allowed to announce.  You need to aggregate somehow up
to at least a /24 to qualify.  This in most cases implies
suballocating from an upstream carrier who is announcing for you.
That's the HamWAN plan for its users.  What's the plan for NW-MESH
users?

KY9K - Route Hijacking - A NW-MESH node certainly could announce a
bunch of nonsense routes either due to user error or malicious
activity.  Just like a ham can time out repeaters for fun, jam nets on
HF, and a myriad of other things.  We accept those risks as part of
the game in ham radio.  We're essentially operating organic
peer-to-peer networks.  The tech history of ham radio is littered with
the carcasses of grand experiments that tried and failed.  The larger
they are and the more corporate overhead, the more likely they are to
fail.

KY9K - Routable Addresses and Interconnection - The 44-net assignment,
as mentioned above, would need to come from somewhat contiguous blocks
to meet the /24 criteria.  That isn't a difficult hurdle to jump.
Those 44-net addresses are just HNAs in NW-MESH and are easily
reachable with current software.  I did some testing with it - easy.
The whole issue of interconnection of a Part97 network with the public
Internet is filled with landmines both operational and regulatory.
The prohibitions on encryption, commercial content, and profanity come
immediately to mind as major issues.  That issue isn't germane though
to HamWAN-MESH interconnection.


AE7SJ: What IS the viable long-term solution for the NW-MESH internal
addressing if 10/8 is not it?

KY9K:  Multiple solutions are possible.  Some are cleaner than others.
 Not really germane to HamWAN-MESH interconnection as long as only
44-net IPs and the mesh-to-mesh tunnels flow across the HamWAN
interconnection.


>   This is a good
> thing.  On the other hand, there are some functions and features which
> add value to MESHland that require direct communication with OLSR ETX
> values intact, and routing staying on 10.0.0.0/8.  Some folks may
> choose to not go with 44-net addresses and advertise their services to
> the wider world.  Someone traveling out of area may want their MESH
> devices to transparently link up with other networks.

AE7SJ: For the HamWAN mobility case, I'd like to have other networks
with their own 44-blocks all dialed into a VPN to support the exchange
of mobile traffic.  That means if a user leaves their HamWANOregon
network and carries a sub-allocation of his 44.x.y.z/28 or whatever,
and then joins the HamWANWashington network, his personal subnet is
still fully routable on the Internet and via RF by way of HamWANOregon
and HamWANWashington exchanging that route via VPN, even though the
/28 announcement would be too small for the Internet and would
probably violate BGP route filters with our respective ISPs.  This way
noone needs to compromise.  Permanent re-location of networks might
cause some excessive traffic though, so re-addressing of the user's
subnet might have to be done.  But that's where DNS makes things nice.

KY9K:  We don't even have an operating "HamWANWashington" yet.  I'd
hate to create an architecture where every state has to adopt the same
practices, standards, etc in order to allow mobility.


> The statement " If this were a peering agreement at an Internet
> exchange, it would be considered a violation of the terms of service
> set out in every Internet exchange I'm aware of" reflects a commercial
> service provider mindset.  This is not such an environment.

AE7SJ:  Well, the commercial rules arose out of general principles of
fairness.  I'm not suggesting money change hands to offset traffic
imbalances, but I would like to make the traffic exchange fair.  What
"fair" means exactly is a complicated thing, and we need to hold a few
discussions about it I'm sure.

KY9K:  The commercial rules arose out of a small group of providers
protecting their turf from newcomers.  I know.  I was there.  I did it
on both the telco and Internet side.  The whole concept of "fairness"
was an thinly veiled smokescreen to protect revenue models.  In the
telco world, the ILECs insisted on termination settlements for their
interconnection agreements with CLECs when the ILECs assumed the
lion's share of traffic would be terminating on their networks.  They
deemed that "fair" and sued as required to ensure the FCC and PUCs
played ball.  As soon as the CLECs proved they weren't stupid by
signing up every dial-up ISP they could find and the termination
revenues went from "CLEC pays ILEC" to "ILEC pays CLEC" suddenly the
interconnection agreements were no longer "fair" and the ILECs started
suing and making campaign contributions to get interconnection fees
removed.

KY9K:  Back when the ARPAnet was first formed, there was the concept
that every network and host attached to the network increased the
value of the network and expanded its reach.  This was a network of
equals.  In the commercial ISP space, you have a producer-consumer
layout where lawyers fight over which is more valuable to the network
- consumers or content.  You see this now with the various
cable/satellite companies and the various owners of programming.  They
fight over who should be paying who and how much.


>   If the
> ham community as a whole contributes to fund, build, and support
> HamWAN then HamWAN needs to accept uses that may not fit its model.

AE7SJ:  In reality, HamWAN donors will be driving the creation of the
network.  It is to them that I feel primarily beholden, not the
general ham community.  For example, if sector users are finding their
speeds going to zero due to inter-MESH traffic, HamWAN will likely
react to make the service work for the people who made it happen in
the first place.  I consider this a fair approach, although it is
certainly not an everyone-is-equal approach.  You don't face this
problem in NW-MESH since you're not centralizing your funding, so
everyone can be equal.  But on the flip side of the NW-MESH approach,
any joker can come online and deny the network for others, no?  How do
you deal with that kind of situation?  In the HamWAN case once abusive
behavior is identified, the offending client can be booted from the
network.

KY9K - Traffic "fairness" -  Please reread my comment.  I said "ham
community as a whole contributes to fund, build, and support....."
That assumes some level of monetary support for HamWAN from
individuals/groups/clubs that access the network form the NW-MESH side
of things.  Thus that ties into your concept of fairness.  Most ham
projects derive their monetary support from 20% or less of the user
base.  Clubs somewhat fix that by supporting things like repeaters,
packet nodes, etc via club dues that are paid by all members.

KY9K:  Sector-Overload If HamWAN finds that inter-MESH traffic on a
given sector is consuming all available bandwidth, and there is non
inter-mesh traffic being QoSd off the network or droped outright, then
there is a problem to be fixed.  If all the traffic happens to be
flowing inter-MESH because the traffic is flowing to/from addresses
within the MESH AS, and no non-MESH traffic is being dropped, then
there isn't a problem to fix.  Assuming the MESH-HamWAN interconnects
are running BGP, the required level of networking ability to implement
those interconnections limits their number.  That implies the
individual/group/club running it has the requisite ability to
rate-limit the outbound traffic.  Even nodes wihch are pure tunnel
interconnections (may or may not be allowed - point of discussion) can
set a rate limit on traffic.  A lot of this loops back around to QoS
and the interconnection architecture.

>   I
> don't expect there will be huge amounts of traffic flowing via
> tunneled connections, but it is an important feature.

AE7SJ:  I'd like to see detailed models on this.  What sticks in my
memory is the Auburn meeting I attended, and just the one room of
routers (20-30 of them maybe?) produced so much traffic that you could
barely get a ping through the MESH there.  I expect they were all
running 54Mbit to boot, due to their close proximity.  Sadly noone was
able to do an over-the-air protocol analysis to see what was going on.

KY9K:  There was a rather special kind of FUBAR going on in that room
at that time.  There were more like 60+ routers in the room at the
time.  The network had partitioned itself due to a few people running
on different channels with the same SSID.  We've since corrected that
issue.  Mix in the AuburnAccess traffic and a few other things and it
was a total and utter mess.  That level of loading would even be tough
on a TDM sector deployment.  Though of course the TDM deployment
wouldn't do the CSMA DCF on existing 802.11 traffic sharing the
channel and might actually perform worse.

***snip***

>
> The most important thing in such an interconnection is harmonizing
> DSCP marking for QoS.  If we want to get very granular and complex,
> DISA UCR 2013 (
> http://www.disa.mil/Services/Network-Services/UCCO/UCR-2013-Draft )
> provides a detailed setup (section6 -
> http://www.disa.mil/Services/Network-Services/UCCO/~/media/Files/DISA/Services/UCCO/UCR2013-Draft/06UCR2013Section6.pdf
> ).  In the long run, as long as both domains use the same QoS, things
> are manageable.  I've played the game where DSCP tags are remarked on
> traffic ingress/egress.  It isn't fun and generally ends up breaking.
>

AE7SJ:  Yup!  And QoS is an open engineering problem on the HamWAN
side right now.  I don't think the IPv4 DS field will be the answer
either.  We may actually be looking at rolling VLANs to solve this
without compromise.  The problem is we actually want to extend QoS
down to the clients' internal networks so that your VoIP phone for
example will have a higher QoS than your PC.  At the same time, it'd
be handy for the same PC to normally ride a "normal" QoS tier, but
when you know you're gonna move 100GB of data and you're not in a
hurry, that same PC ought to declare to the network "hey, use a
best-effort QoS".  This may not be implementable given current
technology, but we'll shoot for the stars and hopefully reach the
moon.  (**removed spaces for convenience***)Have not seen that mil
spec.  Looks daunting.  :)

KY9K:  I strongly suggest that DSCP be used at the points of
interconnection.  That way, the internal QoS architecture of either
network doesn't have to match.  As long as they commonly tag their
packets at the point of interconnection, each network can translate to
its own internal methodolgy.  Requiring PCs to change their QoS values
based on what they intend to do over a single service is a level of
control far too deep.  QoS is usually implemented by traffic type.
There is plenty of prior-art on the subject.  Since we can copy the
DSCP tags from the encapsulated packets to the outer headers, QoS can
even work to shape traffic moving via tunnels.  Been there.  Don that.
 It works very well.  I've had SATCOM links at 100% utilization for
12-16 hours per day.  Everything continued to flow as it should -
voice calls were perfect, IM coordination still flowed without issue,
web browsing slowed down a bit, and it took a bit longer to deliver
large emails.  Take a look at the flow based QoS shaping disciplines
(RED and CoDel are the usual suspects) and you can see how the example
"PC moving 100GB of data" can be prevented from whacking everything
else in the same QoS bucket.  Hopefully, that computer would move the
data using something like FTP that can be easily matched and tossed
into a low-priority queue anyway.


> In most cases, the difference between Part97 and Part15 is a state of mind. It is only when you start using frequencies outside the ISM/UNII bands or running higher >EIRPs on point-to-multipoint links that things become definitely Part97. For example, a 5.8GHz PtP link between two sites running 500mW radios and 26dBi antennas >could be considered Part15 or Part97.

AE7SJ: This is not the case in 5GHz land.  EIRP is limited to 1W under
U-NII.  Anything beyond that and you gotta claim part 97 (or possibly
ISM?).  I have not looked at 2.4GHz, but you may want to.  I suspect
the rules will be similar.  See our Spectrum Allocation page for a
detailed to-the-point analysis of the official publications with
sources cited.  Apologies in advance for the wide format.

KY9K:  BZZZT!  Please try again. :)  Read FCC Part15.407 para A-3:  (I
added some "*" to make the PtP easier to see)
--------
(3) For the band 5.725-5.825 GHz, the maximum conducted output power
over the frequency band of operation shall not exceed the lesser of 1
W or 17 dBm + 10 log B, where B is the 26-dB emission bandwidth in
MHz. In addition, the peak power spectral density shall not exceed 17
dBm in any 1-MHz band. If transmitting antennas of directional gain
greater than 6 dBi are used, both the maximum conducted output power
and the peak power spectral density shall be reduced by the amount in
dB that the directional gain of the antenna exceeds 6 dBi. ***********
However, fixed point-to-point U-NII devices operating in this band may
employ transmitting antennas with directional gain up to 23 dBi
without any corresponding reduction in the transmitter peak output
power or peak power spectral density. For fixed, point-to-point U-NII
transmitters that employ a directional antenna gain greater than 23
dBi, a 1 dB reduction in peak transmitter power and peak power
spectral density for each 1 dB of antenna gain in excess of 23 dBi
would be required. ************** Fixed, point-to-point operations
exclude the use of point-to-multipoint systems, omnidirectional
applications, and multiple collocated transmitters transmitting the
same information. The operator of the U-NII device, or if the
equipment is professionally installed, the installer, is responsible
for ensuring that systems employing high gain directional antennas are
used exclusively for fixed, point-to-point operations.
---------


**snip**