| Author |
Topic |
|
ws5926
Yak Posting Veteran
88 Posts |
 Posted - 2007-05-17 : 11:13:35
|
| The last two days my backup speed has went from about 20-30 MB/sec to 2-5 MB/sec. The backup job is the only thing running on the server at this time. 1. My server is running RAID 5. (i know, i know, goto RAID 10)2. There is plenty of free space on all my drives.3. Windows Automatic updates are off.4. There are no hard drives failing right now.5. All other drive access seems to be slow also.I have not defragged my drives in a while, but everything was running fast 2 days ago. I would expect that if it was too fragmented, then it would have slowly crept up on me.Thanks in advance for any responses to this post.Any idea on what may be causing my slow down?--------------------------------------------------------------------------------Live to ThrowThrow to LiveWill SummersLive to ThrowThrow to Live |
|
|
rmiao
Master Smack Fu Yak Hacker
7266 Posts |
Posted - 2007-05-17 : 16:32:10
|
| I'll still check for disk related (controller and so) error in event logs. |
 |
|
|
deepakontheweb
Starting Member
14 Posts |
Posted - 2007-05-23 : 06:33:05
|
| defrag can slow down db backup but not till that level, I am sure you must have got some conflict in RAID controller or disk.. as 'rmiao' mentioned.. i do suggest checking all events first to identify problemetic area.Deepak KumarSQL Server MVP, MCDBAdeepak.kumar@sqlknowledge.comDisclaimer: This post is provided as is with no rights & warranty for accuracy, for the sake of knowledge sharing only. |
|
|
|
ws5926
Yak Posting Veteran
88 Posts |
Posted - 2007-05-29 : 09:22:42
|
| The problem is back. No events in the event viewer. Started sometime yesterday.Live to ThrowThrow to Live |
|
|
|
ws5926
Yak Posting Veteran
88 Posts |
Posted - 2007-05-29 : 12:53:11
|
| Here's the only message that raises a flag for me:SQL Server has encountered 15 occurrence(s) of IO requests taking longer than 15 seconds to complete on file [E:\xxxx_Log.LDF] in database [xxxx] (7). The OS file handle is 0x000005A8. The offset of the latest long IO is: 0x000000feed5a00There are over 20 of these type within the last 24hrs. Before that there are none within the last week.Live to ThrowThrow to Live |
|
|
|
rmiao
Master Smack Fu Yak Hacker
7266 Posts |
Posted - 2007-05-29 : 13:56:57
|
| You need check disk i/o in perfmon. By the way, any scheduled db maintenance job during that time? Any db file expend? |
|
|
|
ws5926
Yak Posting Veteran
88 Posts |
Posted - 2007-05-29 : 14:42:10
|
| There are multiple backups running at the same time. This normally runs fine until yesterday. I have the perfomance monitor up and am monitoring all of the Physical Disk counters. What should I be looking for? What stats should raise a red flag?Live to ThrowThrow to Live |
|
|
|
rmiao
Master Smack Fu Yak Hacker
7266 Posts |
Posted - 2007-05-29 : 23:01:09
|
| Seems backup and db file growth happened at same time. Do you put backup file on separate disk array? |
|
|
|
ws5926
Yak Posting Veteran
88 Posts |
Posted - 2007-05-30 : 11:47:35
|
quote:
Originally posted by rmiao
Seems backup and db file growth happened at same time. Do you put backup file on separate disk array?
I didn't say anything about my databases growing in size. Yes, my backups are written to a seperate array. As I also originally said, there are plenty of space on all of my drives.Live to ThrowThrow to Live |
|
|
|
ws5926
Yak Posting Veteran
88 Posts |
Posted - 2007-05-30 : 11:57:24
|
| Just an update. I have been working with dell and they did their Dset scan and found no problems. They had me download and ISO image that contains updates for all the drivers on my server. I plan on applying it this weekend and then doing a low level scan of the hard drives. I cannot do this while in production for obvious reasons.I don't see how new drivers will fix this since the current drivers were working fine before this and I didn't change anything. This seems to be a 'blanket' fix because they don't see a problem. I'm going to make sure my backups have been moved off the machine before I run the updates!Live to ThrowThrow to Live |
|
|
|
ws5926
Yak Posting Veteran
88 Posts |
Posted - 2007-06-05 : 10:26:08
|
| Update : 1. updated drivers and new maintenance software from dell, problem still exists 2. drives seem to go into 'slow mode' when system is stressed for a long period 3. if system is not stressed for a long period, then drives don't go into 'slow mode'I have not had enough system down-time to run a full scan on all the hard drives. When I run the scan it puts the drives into 'slow-mode' and requires a reboot to clear, so I can't run the scan while in production. I will have to do one drive at a time this weekend and see how much I can get finished. I only have maintenance windows on the weekend, so I've only had one chance to run the tests since installing the new drivers and scanning software.When I say 'stressed for a long period', I mean a process that handle 10s of millions of rows that runs for at least 20-30 minutes. When I say 'slow-mode', that's my name for it. I don't have enough info on the problem to give it a better name. My other names for it aren't fit for this forum anyway. ha ha.Live to ThrowThrow to Live |
|
|
|
ws5926
Yak Posting Veteran
88 Posts |
Posted - 2007-06-27 : 09:26:23
|
| I hate to take a chance on jinxing my server, but I think we may have solved the problem.Turn out that the battery on the PERC 3 card that controls the array was slowly going dead. It would go into recharge mode and never come out because it wouldn't charge to full capacity. During the recharge it would operate without cacheing. Hence the slow response time.We replaced the PERC 3 card with a PERC 4 card on Monday and haven't went into 'slow-mode' since. Keeping my fingers crossed that it doesn't come back. We are also seeing about a 20-25% increase in speed with the new card.We will wait a couple of weeks and then replace the PERC 3 card that control the internal array to a PERC 4. The PERC 3 operates at 160 and the PERC 4 operates at 320, so we should see an increase in speed on the internal array also. That array holds the operating system, so I'm looking forward to that upgrade!Live to ThrowThrow to Live |
|
|
|
|
Hard drive suddenly running slow??