Well back to school went pretty well for myself and those of us working with Blackboard at Mizzou. However one word did rear its ugly head over the start of the Fall semester. ActiveMQ. That program could frustrate the Dahil Llama. If you are not familiar with ActiveMQ, and if you are a Blackboard admin count yourself very lucky, here’s a quick breakdown. Every application server in a Blackboard Learn instance uses ActiveMQ. If you are running multiple servers like we do at Mizzou, one of those servers is the broker. Call him the CIO or chief of the application nodes. ActiveMQ allows each server to talk to one another and the broker helps to keep the peace and make sure every server knows how to talk to the others. It’s a great tool, but earlier this year when we would run our log rotation process the servers would not restart. This created some early mornings and sleepless nights. (See An “Active” Week for the Blackboard App Admin… blog post.) We thought we had the problem licked, but it came back and in full force. Again every morning at least one server would not come back properly and sometimes it would be several servers. Our production instance would never be fully down, but service would be degraded for users. Not something you want to wake up to find every morning.
So what was the issue? We were rotating our logs on every application server one at a time with a 15 minute window. This rotation of logs required us to stop and then start Blackboard back up again. We found that when the log rotation started on the broker server, all the other servers would try to update a table in the database. Each server would try and lock the table on the database. There were more locks on that table then Mizzou football players on a fumble. We would then have to call in our Oracle DBA Team to clear the locks and allow the servers to restart.
After our first incident, we collected a lot of data for Blackboard Support. We went through all of it and created a ticket. We quickly had our issue moved up to Tier 2 and 3. After multiple back and forths with the support team. Blackboard recommended us to go back to the original peer discovery mode (see the blog post mentioned earlier). We had changed from this mode after a recommendation from Blackboard Support a few months earlier. We planned to implement the change during a maintenance window. However, our users started to experience major issue with Blackboard one evening. I requested an immediate change to apply the change and it was approved. We did see some improvement, but the issue happened again. Blackboard Support recommended the addition of a new index to the table that all the application servers were trying to update. That was applied as well.
I’m not sure if that issue is fixed or if there will be at least a triology of these posts about ActiveMQ.
The ET@MO team also brought me some interesting issues as well. The music department was trying to print out some grade reports for students. When they tried to use the grade reporting tool, the page breaks were ignored by the printer. After some digging with Blackboard Support they found a patch for this issue. We tested it in QA and applied it this weekend.
We also experienced some database troubles over the first week of classes. Our Oracle Database team found that the issue was a specific query which we reported to Blackboard. We also got a patch to fix that issue and applied it this weekend. We also applied the security patch from Blackboard this weekend as well.
We also applied patches or made configuration changes for the following issues:
- When trying to see a student’s discussion board in the Performance Dashboard, a user sometimes gets and error.
- When trying to use the Batch Import/Export tool, a warning displays in the command line but the process completes successfully.
- When editing a question in an assessment/survey and leaving it for over 20 minutes, clicking on the submit button will give the user an error.
Well I’m sure I will find much more to deal with in the coming days.
Technically Yours,
Blackboard Guru
P.S. – Much thanks to Cory, Jory, and Pamela in Blackboard Support for the help on some of the cases I mentioned.
