Quality of Service

Any sufficiently advanced incompetence is indistinguishable from malice.

Archive for July, 2010

Microsoft Exchange Event ID 9646

Posted by qualityofservice on July 15, 2010

Ever see these?

EVENT #
345641
EVENT LOG
Application
EVENT TYPE
Error
OPCODE
Info
SOURCE
MSExchangeIS
CATEGORY
General
EVENT ID
9646
COMPUTERNAME
MAILBOXES
DATE / TIME
08/07/2010 8:54:38 PM

MESSAGE
Mapi session “/o=Org/ou=Exchange Administrative Group/cn=Recipients/cn=User ” exceeded the maximum of 32 objects of type “session”.

BINARY DATA: ……

In our case, these were caused by Riverbed Steelhead appliances running RiOS 6.0.2 (though downrevs will likely experience similar issues) with MAPI Transparent Pre-population enabled.  Users were accessing a VPN in one site, while their mailboxes were in another.  Intersite traffic passes through Riverbed Steelhead appliances in each site, subjecting the user’s Outlook client to Riverbed optimization.

When the client disconnected, the Riverbed appliance maintained the MAPI sessions opened by the Outlook client in order to prefetch mail/calendering/etc.  As the client reconnected/disconnected over the course of a day, this would cause build-up of MAPI session state in the mailbox server, eventually hitting the default limit of 32 sessions.  Subsequent attempts to load Outlook by the user would result in their being prompted for their domain username/password; extremely annoying for the user.

While this 32-session limit could be increased, or the stale connections could be pruned, we preferred a more permanent fix.  After the pre-pop feature was disabled, this problem went away.  This comes at the cost of disconnected clients having to download their new mail from a remote mailserver the next time they connect (as opposed to the Riverbed pre-fetching it for them and handing mail to them locally) — but if mail is a relatively low-priority app in your environment, this is probably more acceptable than users not being able to access it at all (or being prompted for domain credentials all the time).

I should note that in all likelihood, a compromise could have been reached by tweaking the pre-pop timers to time out sessions after a few hours instead of the default 96 hours. YMMV.

I’ll just leave this here in the event that Event ID 9646 gets googled or something.  Outlook clients are generally well-behaved; you shouldn’t see an error like this unless a) the Exchange mailbox server is holding connections open for no reason –not generally a steady-state condition– or b) something in the path is holding the connection to your Exchange mailbox server(s) open on behalf of the client.  Look for anything in the path that “touches” the application-layer and you’ve found a likely culprit.

For Riverbed customers with a valid support contract, this issue can be found as KB article #1053.

Posted in Miscellany | 2 Comments »

 
Follow

Get every new post delivered to your Inbox.