Tag: Ez-RF

  • From the Edge to the Air, Fortinet’s Full Stack End to End Solution

    As much as I hate to admit it, Fortinet is more than just “those guys who bought Meru.”  In fact they are way more than just a wireless company.  As we already know, Fortinet is best known in the industry as a security company with a rock solid reputation.  The truth is, Fortinet is a true end to end solution, beginning with their FortiGate, moving through their switching, and into their wifi.

    Fortinet’s wireless solution is  broken into three distinct management platforms; FortiGate, FortiCloud, and controller (WLC).  Check out one of my blog posts from earlier in the year where I broke down the differences here.

    Until recently I had been mostly unfamiliar with anything but the controller solution which is built on the legacy Meru product.  As most of you already know I have an affinity for the FortiRu product line which goes beyond the controversial Single Channel Architecture.  The hardware is rock solid and it seems as though their code is a lot less buggy than what I hear from customers of other vendors.  The cloud managed solution also seems to be a very viable solution as it chugs along in the lab at my house.  This is of course a far cry from a true enterprise deployment but from what I know it seems to be reliable. Compared to other cloud managed competitors who have been around awhile, the one downside to Fortinet’s cloud managed solution is it could stand to be somewhat more intuitive during configuration.  In Fortinet’s defense, it is hard to stay unique, not re-invent the wheel, and yet remain intuitive in an already saturated cloud managed playing field.  The third management option is via a FortiGate.  In all honestly, I have not tried this method before so I shouldn’t comment until I am more familiar.  I have been graciously provided a FortiGate 81E as well as FortiSwitch 108E to check out but haven’t had the time to dive in yet.

    Those who aren’t familiar with Fortinet should pay very close attention to their licensing platform.  Aside from a select few products, THERE ISN’T ONE!  You don’t need to sift through confusing price lists to find licensing for APs, firewall features, etc because there are no additional licenses required to operate the equipment you bought!  If there is one thing that sticks in the craw of IT professionals these days, it is licensing that needs to be considered for what seems like every single feature/widget that comes with “Box X.”  Not only do you have to think about the confusing and cumbersome licensing when you purchase “Box X,” you have to deal with it each time those licenses come up for renewal.  Not with Fortinet!  Reliable hardware, mostly stable code, and no licensing should move Fortinet onto your short list for consideration the next time you are evaluating equipment for your next purchase, and give you comfort in it’s potential to be a that true end to end solution.

    Check out the following videos from our visit with Fortinet at MFD3 below.

    Fortinet Company Introduction: From Security to Wireless with Chris Hinsz

    Fortinet Portfolio Overview: Access Points and More with Chris Hinsz

    Fortinet FortiGate as a Wireless Controller: Bringing the Security Fabric to the Edge with Koroush Saraf

    Fortinet FortiGate Demonstration with Koroush Saraf

    Fortinet Wireless Analytics: Birthed in Retail, Applicable to Everyone with Koroush Saraf

    For those who lived through the Meru presentation a few years ago at Wireless Field Day 5, many were left with questions about Single Channel Architecture.  This year Ted Fornoles gave a brief presentation on how the “special sauce” behind Single Channel Architecture actually works.  Ted is one of the last remaining “Original Gangsta” Meru guys left within Fortinet.  More time could have been spent here but overall the presentation was great.  Although no questions were asked, there were still some skeptics in the room.  I know it works, Fortinet knows that it works, but there are some out there who will always refuse to accept SCA as a viable, scalable option regardless of what is said.  I stick to my original position on Fortinet’s  publicity of Single Channel Architecture.  If you aren’t already familiar, check out a blog post from earlier in the year here.  I wish there was more talk about SCA from Fortinet’s camp as their silence gives off the appeal that they don’t have confidence in Single Channel Architecture going forward.  Even though SCA has been around for several years now, and is getting somewhat long in the tooth without further advancement, there has to be something Fortinet can do to bolster the SCA solution and make it a heavy hitter in future wireless adaptations.  Check out Ted’s video explaining Single Channel Architecture below.

    Fortinet Virtual Cell & Single Channel Demystified with Ted Fornoles

    A Very Special Thanks

    As many of you know I use FortiRu gear to provide wifi to 100,000ish students and staff every day at Loudoun County Public Schools.  Before I worked for LCPS, I installed and configured FortiRu wireless gear for a VAR.  Pretty much my entire wifi career has been based around FortiRu wireless networks and more specifically Single Channel Architecture.  Over the last several years working with FortiRu, I have had the chance to build many relationships with people who work(ed) for Meru/Fortinet.  Three fellas of note within Fortinet SWAT (TAC) I work with on a regular basis are Kaushik, Harish, and Vikas.  These guys are always available to help me and I often feel bad for interrupting their day by not going through the traditional methods of seeking help, such as submitting a ticket or calling support.  They go out of their way to make sure I get the assistance I need as quickly as possible.  I can’t say enough about these guys as they have truly made a positive impact on my wifi career.  Without them, the success I have had would not have been possible.  While at Mobility Field Day 3, I had the chance to meet Harish and Vikas in person.  It was great to shake the hands of these terrific engineers and thank them in person for all that they do to support me.  Again, thank you very much Harish, Kaushik, and Vikas.  You guys are huge assets to Fortinet.  I hope they know how great you guys are!

    Me and Harish at MFD3

    Another special word of thanks goes to Paul Lambert.  I met Paul via Marcus Barman a couple years back.  Paul has been an invaluable asset to me as I strive to provide fast, reliable wifi to 80,000 concurrent devices at peak every day.  Paul is also one of the very few “Original Gangsta” Meru guys left at Fortinet.  Paul spends most of his time on the security side of things these days but he is still one of the most knowledgeable people in the world (aside from Dr. B.) on Virtual Cell.

  • Frozen Fi – The Big TX Freeze

    frozenfi

    We all find bugs in software from time to time and most of the time they are fixed in relatively short order…..maybe….

    It is always an unsettling feeling when those bugs resurface in later versions of code after being “fixed.”

    We recently received reports from a school that the wifi was “not working.”  Early on in troubleshooting I could see that stations were associated to APs in the reported troublesome area.  All the normal quick checks looked ok.  There wasn’t high channel utilization, retries were low, the APs were not overwhelmed with clients, etc, etc.  I touched base with the person who reported the trouble and they informed me that the issue affected all devices and that it seemed to be in one specific area; the library.  I again checked the APs via the web ui and upon quick glance everything still looked ok.  I logged into the controller CLI and issued the station database command to see what the clients were doing specifically on the library access points.  And then it got cold……..

    txfreeze
    Station database showing the dreaded TX freeze

    As you can see in the graphic above the TX packets are few or none at all.  This problem occurred a couple years ago in later versions of System Director 6 code and was fixed with a newer release so I had seen the problem before.  I took down all of the necessary information so that I could put a ticket in with TAC then rebooted the two APs in the library.  Both of the APs affected were AP 832s and both were experiencing the same problem.  All other APs (AP1020s) were working fine.  After the reboot the APs returned back to working order.

    I began putting my documentation together to submit a ticket when we received word that another school was having wireless issues.  This school was having trouble in it’s library as well.  Like most of our schools, all high density areas (for the most part) are serviced by AP832s.  I logged into the CLI of that school’s controller and found the same TX freeze occurring.  It became obvious to me at this point that the problem was likely affecting more than these two schools.  I checked a few other controllers and found most AP832s were in a frozen TX state.  I started to think what the common denominators could be.  All the APs affected were AP832s.  All the controllers were running the same code, System Director 8.1-2-0.

    And there it was….

    I glanced at the uptime of the AP832s on the controllers that I had up.  All of the AP832s had been up for 99 days.  I started going through other controllers and checking AP832s that had been up for 99 days and found them all to be in a TX frozen state as well.  A few I found that hadn’t reached 99 days were still operating normally.  Clearly there was a bug that reared it’s head on the 99th day of uptime.

    So you are probably asking yourself why so many of the APs had reached the 99 day uptime mark at the same time.  I was wondering the same thing at first and then it hit me.  I had done a mass upgrade of controllers over the summer to System Director 8.1-2-0 which mostly put all of the controller/AP uptimes in sync.

    We launched a preemptive strike and rebooted all AP 832s less a group of three APs at one middle school which hadn’t reached 99 days yet.  We knew that if we rebooted all of the APs the issue would be resolved for another 98 days or until FortiRu provided us a fix.  All of the Ap832s returned to working order.

    At that point I assembled all of my documentation and submitted a ticket to TAC.  We did some initial troubleshooting but they needed to have some APs in the frozen state to get the information they needed.  We put the ticket on hold until our test group of three APs reached 99 days uptime.  That occurred over this past weekend, November 20.  When I returned to the office I checked the APs which were at an uptime of 102 days.  All three were in a TX frozen state.

    AP832s with uptime of 99 plus, in a frozen TX state
    AP832s with uptime of 99 plus in a frozen TX state

    As of right now, the issue is with TAC.  It looks like we are the first to report the issue.  They have all the information that they requested; logs, diags, etc so I should be hearing back from them soon.  In the meantime, a quick reboot is just the ticket to thaw out the TX.

    Here are a few commands I used to gather AP/radio data from within the AP CLI:

    stadb display assigned

    stadbdisplayassignedstadb display assigned -v <mac-addr>

    stadbdisplayassignedmacstadb display rxq_info <client mac-address>

    stadbrxqstadb display txq_info <client mac-address>

    stadbtxqsys exec /wl -i radio0 msglevel err

    sys exec /wl -i radio1 msglevel err

    radio txqinfo radio0 (radio zero)

    radiotxqinforadio0

     

     

     

     

     

     

     

     

     

     

     

     

     

     

    radio txqinfo radio1

    radiotxqinforadio1

     

     

     

     

     

     

     

     

     

     

     

     

     

     

    radio txq radio0

    radiotxqradio0radio txq radio1

    radiotxqradio1radio display

    radiodisplay

     

     

     

    radio show radio0 (radio zero) – radio specific parameters

    radio show radio1 – radio specific parameters

    radio stats radio0 (radio zero) – radio specific stats

    radio stats radio1 – radio specific stats

    dev cmd radio0 reset – resets radio without rebooting the AP

    dev cmd radio1 reset – resets radio without rebooting the AP

    sys exec cat  /proc/meminfo

    sysexeccatprocmeminfo