SQL Server Forums
Profile | Register | Active Topics | Members | Search | Forum FAQ
 
Register Now and get your question answered!
Username:
Password:
Save Password
Forgot your Password?

 All Forums
 SQL Server 2008 Forums
 High Availability (2008)
 Cluster Update KB2710487 CPU Increase
 New Topic  Reply to Topic
 Printer Friendly
Author Previous Topic Topic Next Topic  

DataHammer
Starting Member

USA
3 Posts

Posted - 02/01/2013 :  11:29:02  Show Profile  Reply with Quote
This is an interesting one.

Windows 2008r2SP1 EE
SQL 2008r2SP2CU1 EE


We have installed Cluster Service Update KB 2710487, which is supposed to fix inexplicable Cluster Service crashes with error 1359.

"The cluster service encountered an unexpected problem and will be shut down. The error code was '1359'"

The problem is that the Update increases CPU consumption a great deal. We've identified some interesting behavior:
• Install the patch on one Node, reboot the Node. CPU is normal (0) until you move a cluster group to that node
• With each Cluster Group Move a 1.25% increase in CPU is added to the Node, and it persists. Forever. Note that these are dedicated Nodes, and SQL Groups are complete inactive (no users, no databases)
• This CPU increase persists even after all Cluster Groups are moved off the Node
• 3 processes clearly account for the total CPU increase. When comparing these processes to non-updated hosts we found each of their CPU usage has clearly increased, and each are in constant use. On non-patched hosts all 3 processes have little CPU usage, and are in use infrequently. Here's some metrics:

PROCESSES ON A NON-UPDATED NODE
Process CPU Frequency
------------------ --------- ------------------------------
Clussvc 1 Frequent but not steady
lssas 0 infrequent
Wmiprvse 0 infrequent
---
1 Total Avg CPU

PROCESSES ON AN UPDATED NODE
Process CPU Frequency
------------------ --------- ------------------------------
Clussvc 10 Constant
lssas 2 Constant
Wmiprvse 1 Constant
---
13 Total Avg CPU

• When a workload is placed on the Cluster Node, CPU consumption of these 3 processes increases. The worst we're seeing is a 15% CPU increase on each node, where each Node is hosting 4 active SQL Groups. The inactive Node in this Cluster has a steady 5% CPU load. It is hosting 4 SQL Groups with activity.
• CPU usage patterns by these 3 processes on Updated Nodes are the same regardless of Node CPU and Memory resources

MS tried to argue that this was not a significant increase... riiiiiiiiight. So after days and days of data collection and arguing with support, they've finally agreed this is a bug, and it will be fixed.

Unfortunately this is as far as we've gotten. We have no root cause, or time frame for the fix. We badly need to fix this 1359 error, but we can't afford a 15% increase in CPU.

One thing we've found is that Clusters with Updated Nodes have overly verbose Heartbeat chatter:

Cluster Update Status Packet/Sec
--------------------------- ----------------
0 of 5 nodes Updated 2572
1 of 2 nodes Updated 5795
2 of 2 nodes Updated 12008
4 of 4 nodes Updated 18865

Have any of you seen this issue or have thoughts on HB chattiness being a root cause for CPU load increase?


Edited by - DataHammer on 02/01/2013 11:39:43

russell
Pyro-ma-ni-yak

USA
5071 Posts

Posted - 02/06/2013 :  12:54:51  Show Profile  Visit russell's Homepage  Reply with Quote
We installed CU3 for SP2 (http://support.microsoft.com/kb/2754552).

Fixed the event log chatter, and a non-yielding scheduler thread issue.

Not having the CPU issues you are seeing.
Go to Top of Page

DataHammer
Starting Member

USA
3 Posts

Posted - 02/06/2013 :  13:46:23  Show Profile  Reply with Quote
Thanks for the reply Russell, I will give that KB the once over.
Go to Top of Page

russell
Pyro-ma-ni-yak

USA
5071 Posts

Posted - 02/06/2013 :  17:07:03  Show Profile  Visit russell's Homepage  Reply with Quote
You're welcome. Hope it helps.
Go to Top of Page
  Previous Topic Topic Next Topic  
 New Topic  Reply to Topic
 Printer Friendly
Jump To:
SQL Server Forums © 2000-2009 SQLTeam Publishing, LLC Go To Top Of Page
This page was generated in 0.06 seconds. Powered By: Snitz Forums 2000