![\"frozenfi\"](\"https:\/\/badger-fi.com\/wp-content\/uploads\/2016\/11\/frozenfi.png?w=226\")
<\/p>\n<\/p>\n
We all find bugs in software from time to time and most of the time they are fixed in relatively short order…..maybe….<\/p>\n
It is always an unsettling feeling when those bugs resurface in later versions of code after being “fixed.”<\/p>\n
We recently received reports from a school that the wifi was “not working.”\u00a0 Early on in troubleshooting I could see that stations were associated to APs in the reported troublesome area.\u00a0 All the normal quick checks looked ok.\u00a0 There wasn’t high channel utilization, retries were low, the APs were not overwhelmed with clients, etc, etc.\u00a0 I touched base with the person who reported the trouble and they informed me that the issue affected all devices and that it seemed to be in one specific area; the library.\u00a0 I again checked the APs via the web ui and upon quick glance everything still looked ok.\u00a0 I logged into the controller CLI and issued the station database command to see what the clients were doing specifically on the library access points.\u00a0 And then it got cold……..<\/p>\n As you can see in the graphic above the TX packets are few or none at all.\u00a0 This problem occurred a couple years ago in later versions of System Director 6 code and was fixed with a newer release so I had seen the problem before.\u00a0 I took down all of the necessary information so that I could put a ticket in with TAC then rebooted the two APs in the library.\u00a0 Both of the APs affected were AP 832s and both were experiencing the same problem.\u00a0 All other APs (AP1020s) were working fine.\u00a0 After the reboot the APs returned back to working order.<\/p>\n I began putting my documentation together to submit a ticket when we received word that another school was having wireless issues.\u00a0 This school was having trouble in it’s library as well.\u00a0 Like most of our schools, all high density areas (for the most part) are serviced by AP832s.\u00a0 I logged into the CLI of that school’s controller and found the same TX freeze occurring.\u00a0 It became obvious to me at this point that the problem was likely affecting more than these two schools.\u00a0 I checked a few other controllers and found most AP832s were in a frozen TX state.\u00a0 I started to think what the common denominators could be.\u00a0 All the APs affected were AP832s.\u00a0 All the controllers were running the same code, System Director 8.1-2-0.<\/p>\n And there it was….<\/p>\n I glanced at the uptime of the AP832s on the controllers that I had up.\u00a0 All of the AP832s had been up for 99 days.\u00a0 I started going through other controllers and checking AP832s that had been up for 99 days and found them all to be in a TX frozen state as well.\u00a0 A few I found that hadn’t reached 99 days were still operating normally.\u00a0 Clearly there was a bug that reared it’s head on the 99th day of uptime.<\/p>\n So you are probably asking yourself why so many of the APs had reached the 99 day uptime mark at the same time.\u00a0 I was wondering the same thing at first and then it hit me.\u00a0 I had done a mass upgrade of controllers over the summer to System Director 8.1-2-0 which mostly put all of the controller\/AP uptimes in sync.<\/p>\n We launched a preemptive strike and rebooted all AP 832s less a group of three APs at one middle school which hadn’t reached 99 days yet.\u00a0 We knew that if we rebooted all of the APs the issue would be resolved for another 98 days or until FortiRu provided us a fix.\u00a0 All of the Ap832s returned to working order.<\/p>\n At that point I assembled all of my documentation and submitted a ticket to TAC.\u00a0 We did some initial troubleshooting but they needed to have some APs in the frozen state to get the information they needed.\u00a0 We put the ticket on hold until our test group of three APs reached 99 days uptime.\u00a0 That occurred over this past weekend, November 20.\u00a0 When I returned to the office I checked the APs which were at an uptime of 102 days.\u00a0 All three were in a TX frozen state.<\/p>\n As of right now, the issue is with TAC.\u00a0 It looks like we are the first to report the issue.\u00a0 They have all the information that they requested; logs, diags, etc so I should be hearing back from them soon.\u00a0 In the meantime, a quick reboot is just the ticket to thaw out the TX.<\/p>\n Here are a few commands I used to gather AP\/radio data from within the AP CLI:<\/p>\n stadb display assigned <\/strong><\/em><\/p>\n sys exec \/wl -i radio1 msglevel err<\/strong><\/em><\/p>\n radio txqinfo radio0 (radio zero) <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n radio txqinfo radio1<\/strong><\/em><\/p>\n \u00a0<\/strong><\/em><\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n radio txq radio0<\/strong><\/em><\/p>\n![\"txfreeze\"](\"https:\/\/badger-fi.com\/wp-content\/uploads\/2016\/11\/txfreeze.png\")
<\/a><\/a><\/a>stadb display assigned -v <mac-addr><\/strong><\/em><\/p>\n<\/a>stadb display rxq_info <client mac-address><\/strong><\/em><\/p>\n<\/a>stadb display txq_info <client mac-address><\/strong><\/em><\/p>\n<\/a>sys exec \/wl -i radio0 msglevel err<\/strong><\/em><\/p>\n
\n<\/strong><\/em><\/p>\n<\/a><\/p>\n<\/a><\/p>\n<\/a>radio txq radio1<\/strong><\/em><\/p>\n<\/a>radio display<\/strong><\/em><\/p>\n