Virtual Cell is NOT going to be part of Fortinet’s upcoming .11ax access point product line. I’m not certain of the exact reason but it would seem that the cost of engineering a chipset with proprietary features and efficiency gains from the advancement in technology brought by the new standard, it wasn’t feasible to move forward with keeping Virtual Cell in the .11ax product line. Keep in mind that many of these advancements and efficiencies in .11ax are very closely related to the benefits that Virtual Cell was originally created to solve. Virtual Cell solved many co-channel interference issues using methods very similar to BSS coloring. Also many of the efficiency gains of OFDMA seemingly outweigh the need to continue the development of Virtual Cell within the .11ax standard.
*** Virtual Cell WILL continue to be supported in existing Virtual Cell capable access points. Thanks to Colin Hardacre for the reminder**
Only for the 802.11ax (FAP-U431F & FAP-U433F) when they release later in the quarter. There will still be support for VC/SCA on all the legacy AP's (wave1/2) 11ac. But yes 802.11 r/k/v and .ax has reduced the advantages of the solution, it was a bitter pill to swallow #RIPSCA
As many of you know, Virtual Cell is very near to my heart. We have one of the largest Virtual Cell networks in existence and we have been very successful with it. I would be remiss if I didn’t acknowledge that Virtual Cell has been beneficial to my career. I always felt it was cool that I was able to work with technology that was different than all the others and have it work great despite what others said. The proof is in the pudding. Virtual Cell does work at scale and works very well. Although I will miss all the positives and negatives that come with going on #SingleChannelAdventures, I understand the move to discontinue Virtual Cell development going forward.
I’d like to take this time to thank Paul Lambert, Harish Gnanasambandam, Kaushik Dash, Vikas Banerjee, and Praveenkumar Subramanian for all of the support while working with Virtual Cell. The support from these guys have been invaluable while working with a network that supports upwards of 80k concurrent devices daily. I look forward to working with them as we move towards (and beyond) .11ax.
A very special thanks to Dr. Vaduvur Bharghavan, Srinath Sarang, Joseph Epstein, and Sung-Wook Han. Without these four men, it is quite possible the wireless network industry may look a little different… especially 802.11ax.
Here is datasheet for new 802.11ax access points from Fortinet.
Check out Fortinet’s blog post about their new 802.11ax access point product line here.
Note – Lighthouse image was used with permission from Ranveig Marie Photography. Check out her stuff here: Instagram, Facebook, and Flickr.
This is part three of a multi-part post describing the tuning of Single Channel Architecture/Virtual Cell and taming ChromeOS roaming behaviors. Part one can be found here. Part two can be found here.
We are very densely populated with access points to accommodate at least 50-60 devices per classroom at any given time. Because of this AP density it is necessary to make adjustments to the network on occasion.
As indicated in part two, pcaps determined that we were decoding beacon frames with valid FCS into the high -80s dBm, all with the same BSSID (Virtual Cell). Even though there were several different devices consuming wifi in the area, Chromebooks were hyper sensitive to the fluctuating SNR and were the only device type continuously performing four way handshakes due to a triggered roam (SNR <=18).
Since we are so densely populated with access points (to meet requirements), the only way to tune away the issue was to increase the minimum base transmit rate even further (from 24 Mbps).
24 Mbps Base Transmit Rate – 5 GHz Radio
TX power is already reduced to a level that provides proper overlap between classrooms to satisfy requirements. Increasing the minimum base transmit rate to 36 Mbps was done first but did not yield the results that I wanted to see. I then increased the minimum base transmit rate to 48 Mbps (also leaving the supported rates of 24 and 36 Mbps) and ran another pcap.
48 Mbps Base Transmit Rate – 5 GHz Radio
This time I was receiving most beacons with valid FCS in the mid -70s dBm. Further testing with the Chromebooks showed the results I wanted to see.
Unnecessary (triggered) roams were dramatically reduced and performance was much better. Beacon overhead was decreased by 50% which is indicative of moving from 24 to 48 Mbps minimum base rate.
We ended up testing this configuration change at a couple of our high schools of similar layout, design, and requirements with very good results.
Note – Lighthouse image was used with permission from Ranveig Marie Photography. Check out her stuff here: Instagram, Facebook, and Flickr.
This is part two of a multi-part post describing the tuning of Single Channel Architecture/Virtual Cell and taming ChromeOS roaming behaviors. Part one can be found here.
It is no secret at this point that I maintain a very large Single Channel Architecture/Virtual Cell wireless network. From time to time there are certain nuances that force our hand into making adjustments to the wireless network, which really is no different than any other network, SCA or MCA. Since we do primarily use a single channel throughout our instructional spaces, a single, identical, BSSID is seen by all clients within a Virtual Cell. Recently I was troubleshooting network performance issues on Chromebooks and found that they were constantly performing four way handshakes, several times per minute. Other devices in the area, including my Mac, did not experience the same problem. I decided to pull some pcaps to see what I could find. I ran Wireshark for a period of time then filtered out beacons (wlan.fc.type_subtype == 0x0008) and sorted them by lowest to highest RSSI. What I found next started to pull things together from several other recent troubleshooting sessions. I found that my beacons, all from the same BSSID, had valid FCS and were being received with RSSI into the high -80s (dBm). This only left room for an SNR in the single digits.
Because the SNR was so low on these distant beacons, unnecessary roams were being triggered on the Chromebooks even though there was ample signal available from much closer access points. From one second to another, a device could hear several beacons with the same BSSID, some with very poor SNR and others with very favorable SNR. For the last couple of years we have found a sweet spot with TX power and minimum supported rates which has worked nicely with the vast majority of the 80,000 devices we see at peak on a daily basis. Until the recent influx of Chromebooks (tens of thousands) in the past school year there has been no reason to make any adjustments.
RX-SOP?
Stay tuned for an upcoming post on changes that were made to the infrastructure to make these sensitive devices behave better.
Note – Lighthouse image was used with permission from Ranveig Marie Photography. Check out her stuff here: Instagram, Facebook, and Flickr.
This post will serve as part one of a multipart post concerning recent happenings involving client settings, client behavior, what was done to tune a network, and the results thereafter.
Chromebooks have been around for several years now and love them or hate them, they are here to stay. Chromebooks began as a low cost consumer device but have since crept their way into the enterprise, as well as established a strong foothold in the K12 vertical. What originally seemed like a cheap compute option, is now available in many different hardware options, many times offering “better than decent” performance for an affordable price. Don’t be fooled though, there are plenty of Chromebook devices out there that sport less than desirable performance but rep a very attractive price tag. This initial post explains my desire to get a better understanding of Chromebooks and their behavior on my beloved wireless network.
We currently have tens of thousands of Chromebooks in our Google Domain that are owned and managed by our school district. We recently reached over 80,000 concurrent wireless devices on our network and ChromeOS serves as a sizable percent of that number, steadily creeping towards surpassing iOS. Most of the time everything works great but when there are issues reported, you can bet there is a very good chance the device in question is a Chromebook. Known issues in the past have been cleared up by Google in relatively short order. Other times, issues linger on and off, coming and going like the ebb and flow of an ocean tide. In the recent past our district standardized on a specific manufacturer and model of Chromebook that would be purchased and supported by our department staff. Many people were brought together to make sure the device met and exceeded our expectation for an exceptional user experience. Keep in mind there are several thousand other Chromebook devices still on our network which are either legacy holdover or BYOD devices. Ironically, despite the vast differences, many of the Chromebooks exhibit similar behaviors. This post will go on to explain my thoughts as to why this may be.
To get a better understanding of Chromebook behavior on wireless networks, I decided to collect data on my own. This was done to see if I could correlate behaviors with known issues. I decided to create a simple form on my blog that asked for a few pieces of information easily obtained from a Chromebook with a few commands. The following information is what I was interested in:
Chromebook Full Model Name
ChromeOS Version
WLAN Adapter Model
Roam Threshold
Country
Overall, I was fairly underwhelmed with the response but I did get a handful of responses (thanks to those who contributed). Out of the responses I did receive there was one very interesting thing that stood out. The roaming threshold was EXACTLY the same no matter the manufacturer, ChromeOS version (recent or several years old), or WLAN hardware!
Why would 18 (roaming threshold (SNR)) be the threshold that was selected for ALL variations of Chromebooks? What this means is that if the Chromebook’s Signal to Noise Ratio (the difference between RSSI and noise floor) drops below 18, a roam event could occur because the device deems the signal to be less than optimal. Those of us who have been in the game for awhile know that the inconsistencies in end-point wireless hardware is the consistency. Wireless network hardware is not calibrated by the manufacturer. Wireless network hardware behavior will vary depending on manufacturer, device capabilities, DRIVER VERSION, etc. In fact, two wireless network adapters made by the same manufacturer using the exact same driver, placed in the exact same model device, will behave differently! Why would the very intelligent people at Google think that the same, non-adjustable, roaming threshold was best with so many different variables in play? There may be a good explanation, but so far I have been unable to find it. Other types of devices (ie. Windows) offer the ability to tune roaming behaviors within advanced driver settings. To date, ChromeOS does not allow adjustment to these types of settings.
As mentioned earlier, Chromebooks aren’t going anywhere. They are good devices and fit the K12 environment well due to many factors. As Chromebooks push further out of the consumer realm and into being an enterprise player, Google may need to adjust their one size fits all wireless settings model. The massive difference between home use and enterprise use should drive this change unless there is a good reason not to (I’d love to know, Google!).
Future posts will explain more discoveries and how I was able to tame these K12 beasts, with the understanding that bending a network for a single device type is often dangerous. I like to live dangerously…
As much as I hate to admit it, Fortinet is more than just “those guys who bought Meru.” In fact they are way more than just a wireless company. As we already know, Fortinet is best known in the industry as a security company with a rock solid reputation. The truth is, Fortinet is a true end to end solution, beginning with their FortiGate, moving through their switching, and into their wifi.
Fortinet’s wireless solution is broken into three distinct management platforms; FortiGate, FortiCloud, and controller (WLC). Check out one of my blog posts from earlier in the year where I broke down the differences here.
Until recently I had been mostly unfamiliar with anything but the controller solution which is built on the legacy Meru product. As most of you already know I have an affinity for the FortiRu product line which goes beyond the controversial Single Channel Architecture. The hardware is rock solid and it seems as though their code is a lot less buggy than what I hear from customers of other vendors. The cloud managed solution also seems to be a very viable solution as it chugs along in the lab at my house. This is of course a far cry from a true enterprise deployment but from what I know it seems to be reliable. Compared to other cloud managed competitors who have been around awhile, the one downside to Fortinet’s cloud managed solution is it could stand to be somewhat more intuitive during configuration. In Fortinet’s defense, it is hard to stay unique, not re-invent the wheel, and yet remain intuitive in an already saturated cloud managed playing field. The third management option is via a FortiGate. In all honestly, I have not tried this method before so I shouldn’t comment until I am more familiar. I have been graciously provided a FortiGate 81E as well as FortiSwitch 108E to check out but haven’t had the time to dive in yet.
Those who aren’t familiar with Fortinet should pay very close attention to their licensing platform. Aside from a select few products, THERE ISN’T ONE! You don’t need to sift through confusing price lists to find licensing for APs, firewall features, etc because there are no additional licenses required to operate the equipment you bought! If there is one thing that sticks in the craw of IT professionals these days, it is licensing that needs to be considered for what seems like every single feature/widget that comes with “Box X.” Not only do you have to think about the confusing and cumbersome licensing when you purchase “Box X,” you have to deal with it each time those licenses come up for renewal. Not with Fortinet! Reliable hardware, mostly stable code, and no licensing should move Fortinet onto your short list for consideration the next time you are evaluating equipment for your next purchase, and give you comfort in it’s potential to be a that true end to end solution.
Check out the following videos from our visit with Fortinet at MFD3 below.
Fortinet Company Introduction: From Security to Wireless with Chris Hinsz
Fortinet Portfolio Overview: Access Points and More with Chris Hinsz
Fortinet FortiGate as a Wireless Controller: Bringing the Security Fabric to the Edge with Koroush Saraf
Fortinet FortiGate Demonstration with Koroush Saraf
Fortinet Wireless Analytics: Birthed in Retail, Applicable to Everyone with Koroush Saraf
For those who lived through the Meru presentation a few years ago at Wireless Field Day 5, many were left with questions about Single Channel Architecture. This year Ted Fornoles gave a brief presentation on how the “special sauce” behind Single Channel Architecture actually works. Ted is one of the last remaining “Original Gangsta” Meru guys left within Fortinet. More time could have been spent here but overall the presentation was great. Although no questions were asked, there were still some skeptics in the room. I know it works, Fortinet knows that it works, but there are some out there who will always refuse to accept SCA as a viable, scalable option regardless of what is said. I stick to my original position on Fortinet’s publicity of Single Channel Architecture. If you aren’t already familiar, check out a blog post from earlier in the year here. I wish there was more talk about SCA from Fortinet’s camp as their silence gives off the appeal that they don’t have confidence in Single Channel Architecture going forward. Even though SCA has been around for several years now, and is getting somewhat long in the tooth without further advancement, there has to be something Fortinet can do to bolster the SCA solution and make it a heavy hitter in future wireless adaptations. Check out Ted’s video explaining Single Channel Architecture below.
Fortinet Virtual Cell & Single Channel Demystified with Ted Fornoles
A Very Special Thanks
As many of you know I use FortiRu gear to provide wifi to 100,000ish students and staff every day at Loudoun County Public Schools. Before I worked for LCPS, I installed and configured FortiRu wireless gear for a VAR. Pretty much my entire wifi career has been based around FortiRu wireless networks and more specifically Single Channel Architecture. Over the last several years working with FortiRu, I have had the chance to build many relationships with people who work(ed) for Meru/Fortinet. Three fellas of note within Fortinet SWAT (TAC) I work with on a regular basis are Kaushik, Harish, and Vikas. These guys are always available to help me and I often feel bad for interrupting their day by not going through the traditional methods of seeking help, such as submitting a ticket or calling support. They go out of their way to make sure I get the assistance I need as quickly as possible. I can’t say enough about these guys as they have truly made a positive impact on my wifi career. Without them, the success I have had would not have been possible. While at Mobility Field Day 3, I had the chance to meet Harish and Vikas in person. It was great to shake the hands of these terrific engineers and thank them in person for all that they do to support me. Again, thank you very much Harish, Kaushik, and Vikas. You guys are huge assets to Fortinet. I hope they know how great you guys are!
Me and Harish at MFD3
Another special word of thanks goes to Paul Lambert. I met Paul via Marcus Barman a couple years back. Paul has been an invaluable asset to me as I strive to provide fast, reliable wifi to 80,000 concurrent devices at peak every day. Paul is also one of the very few “Original Gangsta” Meru guys left at Fortinet. Paul spends most of his time on the security side of things these days but he is still one of the most knowledgeable people in the world (aside from Dr. B.) on Virtual Cell.
Since last week’s open letter to FortiRu, I came to the realization that there may be some confusion related to Fortinet’s AP product line. I decided to throw this post together so that it is easier to understand which APs are what, and which category they fall into. Keep in mind this is a high level overview to simply point out the different AP product lines.
Lets jump right into it…
Fortinet offers three different management options under it’s Enterprise Secure Wifi solution; FortiGate, Cloud, or dedicated controller, and five different AP lines that fall into one or more of those management options.
Excerpt from Fortinet Product Matrix
Access points that can be managed via the ForiGate or the Cloud are broken down here:
The FortiAP-S313C, which was given to attendees of the recent Wireless LAN Professionals Conference, falls into the preceding category. You can read a great blog post about the FortiAP-S313C AP written by my apple butter lovin’ buddy Lee Badman here. Please pay close attention to what Lee says concerning the licensing for these APs. A true game changer…
FortiAP-S313C
There may have been some confusion with the FortiAP-S313C stemming from the Fortinet presentation at WLPC. The FortiAP-S313C is NOT from the Meru pedigree of access points of which I will elaborate on next.
Access points that are managed from a dedicated controller are broken down here:
These APs ARE from the Meru lineage of access points which I affectionately refer to as FortiRu. These APs support Virtual Cell/Single Channel Architecture and are still commonly referred to as Meru access points. The APs in the Standard AP list are all carry over from the Meru purchase. The AP1020 and AP832 are models that we use in our school district and the models that I have the most experience with. Both the AP1020 and AP832 have been rock solid since their release to market. It is often confused that these standard APs are only capable of operating under Virtual Cell/Single Channel Architecture. All traditional Meru access points can operate in EITHER a multi channel OR Virtual Cell configuration.
You may have noticed that under each section there was a group of APs referred to as Universally Manageable. These APs can be managed using any of the three management solutions. Just like its’ Meru predecessor, the Universal AP can be used in EITHER a multi channel OR Virtual Cell configuration depending on the chosen management solution. I should have more to write on the “U” line of APs in the near future as we should be receiving some very soon.
FAP-U421EV
As previously stated, this blog post was intended to be a high level overview of the Fortinet/Meru (FortiRu) AP product line. Clicking on the links and reading through the material will give you a deeper explanation into each of the access points. As always, if there are any questions, comments, corrections, etc, please feel free to reach out to me.
You and I have been together for almost seven years. We have been many places together; local government buildings, soft drink distributors, beer distributors, small town banks, surgical tool manufacturing facilities, resorts, retirement communities, many private and public schools (too many to count), and Thomas Jefferson’s Monticello (Remember TJ’s attic?). We have had ups and downs but to be honest it has been mostly a positive relationship. I will admit, there was a time when I didn’t want to talk much about you and I, afraid of what others would think, but I eventually professed my affection for you publicly. Many out there were launching arrows at you and it became too difficult to stand by and watch you take hit after hit. You would get criticism, some just, but mostly un-just, from people that barely knew you or didn’t know you at all. I tried to jump in and take bullets on your behalf. People kept firing…
It has been nearly three years since we reached a milestone in our relationship. Some milestones are good, some not so good. I’m not sure which one is a good reflection of us yet. Some people probably wonder why I care so much about you. My answer would be that after you have been with something for so long, it becomes part of you. I feel like I have so much invested in you. You have been great for me and my career! There are many more fish in the sea and sometimes I think I might be better off somewhere else, but I always find my way back to you. You have been there since I decided to really focus my attention on wireless technologies. You have been there as my career blossomed and you have even give me opportunities that I may have not had otherwise.
My questions for you at this point in our relationship are this; where do you see us going? Do you think we have a future together? Not necessarily where everyone else is going (MCA), but with something that our relationship has a FOUNDATION on (SCA\Virtual Cell)? All those other choices out there seem so much alike. Some are built in the cloud, offer similar features, and solid reputations, but you… you are different! You have something special. You have something that others have tried but have failed at. That cute little thing you do (SCA\Virtual Cell) “may” be getting a little stale, but there are so many advantages and unrealized potential. I was hoping so badly that the milestone I spoke of above would bring a revitalized fire, and I hope that it still might! There have been GREAT improvements since that milestone but I just don’t see where you want to be yet. I am concerned for you. Do you want to be like everyone else or do you want to be unique and build on something great? How would Dr. Bharghavan, Srinath Sarang, Joseph Epstein, and Sung-Wook Han feel about the direction you are taking? I know you might not care what they think anymore, but they are the ones who brought you into this world.
People ask me about you all the time. They ask how you are doing and what you are going to do with your future. I can’t answer those questions. YOU HAVE TO ANSWER THOSE QUESTIONS!
I was present recently when you talked about some things that you have been going through. There were a lot of people around (Wireless LAN Professionals Conference 2018) and I was so excited to see you stand up and start talking. I thought you were getting ready to have a breakthrough and let everyone know how awesome you can be… but then I was let down just as quickly as it started. I’ll be honest, I was a little embarrassed. People around me noticed how excited I got when I saw you and then they noticed how quickly I was let down. You did a great job talking and the presentation was fine, but I couldn’t help but think it was a little fake. It wasn’t who you really are. I felt like you were just saying things you thought others wanted to hear. People told me afterwards that they would have rather heard you talk about WHO YOU REALLY ARE, but instead you were “like all the rest of them.” Love you or hate you, people just want you to BE YOU!
I’ve been places and heard those other fish in the sea speak about themselves with confidence. They keep wanting so badly to improve themselves, make themselves look good, and be accepted. I can’t help but notice that it seems like you don’t care about everything that makes you who you are. It seems you don’t remember where you came from, almost like you are trying to be someone else. The world doesn’t need followers. They (the others) all look the same to me. I love you for who you were, who you are, and for the potential that you have. Please don’t fall into the same line all of the others are in. You were about being different. Don’t lose sight of that!
There are so many people out there that know your potential. I know they might not always outwardly express their feelings, but they tell me. They tell me to fight for you and tell me to be patient… and I will, but I can’t do it by myself. I need your help. I need you to be confident in yourself. I need you to help me. It can’t be a one way road like it has been. Help me help you!
I can’t speak for the future and what others feel, but I can speak for myself. If I don’t see effort in this relationship, I don’t know that I can continue on. I will still work beside you and be your friend, but I can’t guarantee that I will have the fight in me anymore.
No matter what happens, I will always care about you. I just hope that you can figure out who you really are. You might already know but just haven’t told anyone yet, and that is ok, but just know that you are not helping anyone, including yourself.
I am writing this letter to you, not because I want to break up, but because I care about you and your future. I know you have great potential even though I don’t know exactly what the future looks like. There are so many people pulling for you, I promise. I hope my future has you in it.
A few weeks ago at Mobility Field Day 2 I was afforded the opportunity to listen to a presentation from the fine people of NETSCOUT (all capitals because that is how Kendall Hershey told me it should be). It was somewhat of a reunion for Brennan, Rob, and myself as we had established a relationship with NETSCOUT and their Social Media Manager, Kendall Hershey, at Cisco Live 2016 (shameless blog plug) . It was great to see her again as well as take a deeper dive into the NETSCOUT product line, especially the AirCheck G2.
Chris Hinsz, NETSCOUT Product Manager, was the main presenter and he gave a nice overview of the NETSCOUT product line, more specifically the beloved AirCheck G2. Many of us WLAN professionals use the AirCheck G2 at our day jobs and have come to love the simple yet powerful tool. Chris went on to offer a closer look into what makes the AirCheck G2 so great as well as some lesser known tips and tricks. Check out the presentation here.
Chris also gave us a quick demo of the Link-Live dashboard. The Link-Live dashboard is a clean, easy to read area where you can collect the information that your various NETSCOUT tools gather. I have had the opportunity to use the Link-Live dashboard with my LinkSprinters for some time and I appreciate the simple layout. I can quickly get the information I need with a quick glance. The dashboard organizes your tests into a table and allows you to make annotations, as well as create reports from the collected data. NETSCOUT has definitely spent some time refining the layout making it easy to get the information you need from your test equipment.
I have been drooling over the AirCheck G2 since it was released. We recently purchased one for work so I have only recently had the opportunity to use it on a regular basis. I love the features of the G2 especially the touchscreen and well laid out menus. The testing features are straight forward. The G2 is hands down the first device we go to when we get called out for wireless issues. The G2 saves us a significant amount of time troubleshooting. Before the G2, troubleshooting wireless issues took much longer. These handheld testers bring you to a solution much faster with the information that they supply. I personally own the G2’s predecessor, the **Fluke AirCheck. The original AirCheck is still very useful to me because it is able to extract a Meru/Fortinet/FortiRu AP ID out of a beacon’s information elements. This makes it very easy for me to identify an AP that is a member of a Virtual Cell (virtual BSSID). I still love my original AirCheck for this reason, but mostly because Kendall Hershey gave it to me. I am told that they will look into adding the FortiRu information element to the G2 in the near future.
Gratuitous NETSCOUT AirCheck G2 product placement by Brett (@crabby_fi)
The AirCheck G2 has been, and continues to be, a staple in the WLAN professional’s bag of tools. The NETSCOUT product line is built on a solid platform and is sure to deliver for the foreseeable future. Anyone who uses a G2 will let you know how much it enhances their troubleshooting experience. Issues that may have previously taken a few hours to figure out are easily diagnosed in a much shorter time period. NETSCOUT is another one of those companies who I hold near and dear to my heart because they listen to their customers. I am looking forward to what NETSCOUT has coming in the future.
**On July 14, 2015, the following products from Fluke Networks were merged with NETSCOUT Systems; Visual TruView™, OptiView® XG, OneTouch™, LinkSprinter™, LinkRunner™, AirMagnet™ and AirCheck™. Fluke Networks continue to retain the DTX CableAnalyzer™, Versiv Cabling Certification System, LinkWare™ Live and Telecom products.
It has been a couple weeks since I returned from WLPC in Phoenix. It was a great trip down to the southwest which included catching up with some good friends, listening to great presentations, learning a lot, and also presenting on a topic near and dear to my heart. I was able to put a lot more names to faces and have some great conversation.
As many of you know I presented on how well single channel architecture works for us. You can find the video below:
I have received mostly all positive feedback on the presentation. It was a great opportunity for me to speak in front of a large crowd and give examples of how we have Meru deployed across our school district. As well as it went, I feel like I need to explain a few things. I have come to the realization that a Ten Talk may have been the wrong platform to discuss the topic at hand. I understand that most people wanted more information about SCA rather than “it just works for us.” Ten minutes was simply not enough time to discuss all of this.
First, I understand that being a large network doesn’t equal wired\wireless networking done right. Even though this wasn’t conveyed in the presentation, I have received feedback indicating as such. Although we are very proud of how large and successful our network is, it doesn’t mean that large = successful. A lot of time and effort goes into making a wireless network work well for 60,000 users on average every day. We all know that architecture doesn’t matter if you don’t define, design, implement, and validate.
Second, I gave some confusing information. I realized shortly after the presentation that the stat showing that we “average ~12 connections per AP county wide” was very vague. Some of our access points have upwards of 100 clients on them at once for an extended period of time. Some of our access points have 10 clients total (maybe less) in an entire day. It doesn’t matter whether there is 30+ clients on an AP during every class or another AP that sits unused for the majority of the day. If an AP was placed in a location, it was done there with the intent that wifi could be needed at any point of an instructional day; planned or un-planned. The opinion that there are too many APs or not is irrelevant. Unless you have actually visited our schools and used the network, you won’t know. The proof is in the pudding so to say.
I know that most of us consider our wireless networks as mission critical. The vast majority of our schools don’t have desktop computers other than in administrative areas. Many of our schools don’t have computer labs, but employ several carts of mobile devices. I know that most of us want NUMBERS to back up the user experience, but most of the time I don’t have time to get this type of information. If reports of “wireless problems” are low or non-existent and teachers are able to complete their instruction using mobile devices and be successful, our mission is accomplished. I know a lot of you won’t be happy until you see NUMBERS and that is fine. I am going to try real hard to get some of those and put them out there. To be perfectly honest, I don’t know if some of you will believe it then, and that is fine too.
I understand many of the things that have been said as far as physics, airtime consumption, high density, etc. I don’t necessarily disagree with some of the opinions. You may be absolutely right! The thing that is somewhat discouraging is that there are a lot of opinions based on no experience or an experience that was several years ago. Don’t get me wrong, the first time I saw virtual port, even with my limited wireless knowledge at the time, I couldn’t believe it would work. A lot has been improved upon since then with virtual cell and especially the equipment. I’m not saying Meru will beat another vendor head to head every time, but I bet it will sometimes! Does it really matter how many jigabits we can ram through the air or is it more important for a user to have an experience where they don’t even notice the wifi? A user sits down, opens their laptop, completes a task, closes the laptop and moves on without even acknowledging the presence of wifi. The wifi just works. Please understand, I am not trying to discount numbers such as channel utilization, retries, available bandwidth, etc. Those are all very important things to consider in a wireless environment. We all look to those numbers first when trouble is initially reported. Those are the numbers that give us a baseline to begin troubleshooting. They are absolutely critical to a successful wireless deployment.
Obviously interest has been piqued since the presentation. It has been fun, most of the time, discussing various aspects concerning single channel vs multi channel environments. I have heard a handful of different people give a handful of different explanations on what Meru’s special sauce is, and they are all different. There isn’t a ton of information out there concerning the “magic” of Meru but if you are truly interested please watch any video by Dr. Bharghavan, founder of Meru networks. Many of these videos are a few years old, but still offer great information. To be honest, I need to re-watch most of them as I get tangled in the “it just works,” sometimes. Below are a handful of videos.
If you want to catch the videos later but still want to read the conclusion, please scroll down.
Meru Networks Wireless Virtualization Architecture – Part 1
Meru Networks Wireless Virtualization Architecture – Part 2
Meru Networks Wireless Virtualization Architecture – Part 3
Contention Management Schemes: Part 1 – Single / Multiple AP
Contention Management Schemes: Part 2 – Multiple APs
Maximizing Air Traffic – Part 1: Maximize Channel Reuse
Maximizing Air Traffic – Part 2: Simultaneous Transmissions
Leveraging Single Channel Architecture for Multiple Channels
A Little Dated But Still Good
Very High Density Wireless LAN Demonstration for BYOD: #1
Very High Density Wireless LAN Demonstration for BYOD: #2
Conclusion
For those of us who went to WLPC 2017, we heard more than one person mention that wireless networking can be done in more than one way. We also heard that less than ideal practices may be employed against our better wireless judgement due to other factors such as politics, aesthetics, etc. Sometimes I think we need to remember that just because someone does something different doesn’t mean that it is wrong. We also need to remember that just because we don’t like a technology it doesn’t mean it doesn’t fit someone’s need. I need to remind myself of this from time to time. We deploy wireless networks, in schools, mines, warehouses, large refrigerators, outdoors; you name it, we put wifi in all kinds of places. Our ultimate goal should be to use the knowledge we have to deploy a wireless network that gives a reliable experience to the greatest number of users.
We all find bugs in software from time to time and most of the time they are fixed in relatively short order…..maybe….
It is always an unsettling feeling when those bugs resurface in later versions of code after being “fixed.”
We recently received reports from a school that the wifi was “not working.” Early on in troubleshooting I could see that stations were associated to APs in the reported troublesome area. All the normal quick checks looked ok. There wasn’t high channel utilization, retries were low, the APs were not overwhelmed with clients, etc, etc. I touched base with the person who reported the trouble and they informed me that the issue affected all devices and that it seemed to be in one specific area; the library. I again checked the APs via the web ui and upon quick glance everything still looked ok. I logged into the controller CLI and issued the station database command to see what the clients were doing specifically on the library access points. And then it got cold……..
Station database showing the dreaded TX freeze
As you can see in the graphic above the TX packets are few or none at all. This problem occurred a couple years ago in later versions of System Director 6 code and was fixed with a newer release so I had seen the problem before. I took down all of the necessary information so that I could put a ticket in with TAC then rebooted the two APs in the library. Both of the APs affected were AP 832s and both were experiencing the same problem. All other APs (AP1020s) were working fine. After the reboot the APs returned back to working order.
I began putting my documentation together to submit a ticket when we received word that another school was having wireless issues. This school was having trouble in it’s library as well. Like most of our schools, all high density areas (for the most part) are serviced by AP832s. I logged into the CLI of that school’s controller and found the same TX freeze occurring. It became obvious to me at this point that the problem was likely affecting more than these two schools. I checked a few other controllers and found most AP832s were in a frozen TX state. I started to think what the common denominators could be. All the APs affected were AP832s. All the controllers were running the same code, System Director 8.1-2-0.
And there it was….
I glanced at the uptime of the AP832s on the controllers that I had up. All of the AP832s had been up for 99 days. I started going through other controllers and checking AP832s that had been up for 99 days and found them all to be in a TX frozen state as well. A few I found that hadn’t reached 99 days were still operating normally. Clearly there was a bug that reared it’s head on the 99th day of uptime.
So you are probably asking yourself why so many of the APs had reached the 99 day uptime mark at the same time. I was wondering the same thing at first and then it hit me. I had done a mass upgrade of controllers over the summer to System Director 8.1-2-0 which mostly put all of the controller/AP uptimes in sync.
We launched a preemptive strike and rebooted all AP 832s less a group of three APs at one middle school which hadn’t reached 99 days yet. We knew that if we rebooted all of the APs the issue would be resolved for another 98 days or until FortiRu provided us a fix. All of the Ap832s returned to working order.
At that point I assembled all of my documentation and submitted a ticket to TAC. We did some initial troubleshooting but they needed to have some APs in the frozen state to get the information they needed. We put the ticket on hold until our test group of three APs reached 99 days uptime. That occurred over this past weekend, November 20. When I returned to the office I checked the APs which were at an uptime of 102 days. All three were in a TX frozen state.
AP832s with uptime of 99 plus in a frozen TX state
As of right now, the issue is with TAC. It looks like we are the first to report the issue. They have all the information that they requested; logs, diags, etc so I should be hearing back from them soon. In the meantime, a quick reboot is just the ticket to thaw out the TX.
Here are a few commands I used to gather AP/radio data from within the AP CLI:
stadb display assigned
stadb display assigned -v <mac-addr>
stadb display rxq_info <client mac-address>
stadb display txq_info <client mac-address>
sys exec /wl -i radio0 msglevel err
sys exec /wl -i radio1 msglevel err
radio txqinfo radio0 (radio zero)
radio txqinfo radio1
radio txq radio0
radio txq radio1
radio display
radio show radio0 (radio zero) – radio specific parameters
radio show radio1 – radio specific parameters
radio stats radio0 (radio zero) – radio specific stats
radio stats radio1 – radio specific stats
dev cmd radio0 reset – resets radio without rebooting the AP
dev cmd radio1 reset – resets radio without rebooting the AP
good friends, listening to great presentations, learning a lot, and also presenting on a topic near and dear to my heart. I was able to put a lot more names to faces and have some great conversation.
