| Author |
Topic |
|
SamC
White Water Yakist
3467 Posts |
Posted - 2009-12-23 : 11:50:47
|
| We have a recurring execution plan problem with our stored procedures. Almost every day, a random stored procedure will drop the use of indexes and begin table scans. Timeouts follow. Recompiling the stored procedure solves the issue. The next day, some other random stored procedure will do the same requiring a recompile to resolve the timeouts.We reindex every night which rebuilds statistics in an effort to avoid this issue. It doesn't seem to be helping much of late.Strange thing about the execution plan going bad: when the plan goes bad, the stored procedure will execute promptly in an SSMS query window using the same username as the web page. However, execute the same stored proc from a web page and it will time out. Don't jump to the conclusion that the web page has some other problem disguising the issue: recompiling the stored procedure will correct the timeout on the web page.I wonder if anyone has seen something similar. Is a cause or solution known? |
|
|
russell
Pyro-ma-ni-yak
5072 Posts |
Posted - 2009-12-23 : 13:14:19
|
| are there a lot of inserts and/or updates?try updating stats with fullscanby the way, do stored procedures in question normally perform index seeks? |
 |
|
|
yosiasz
Master Smack Fu Yak Hacker
1635 Posts |
Posted - 2009-12-23 : 14:46:06
|
| how are you calling the stored procedures from your web page? adodb or what? are you closing any connections to the stored procedure properly, cleaning up after use of stored procedure? show us your web code please.<><><><><><><><><><><><><><><><><><><><><><><><><>If you don't have the passion to help people, you have no passion |
 |
|
|
SamC
White Water Yakist
3467 Posts |
Posted - 2009-12-23 : 15:01:13
|
quote: Originally posted by russell are there a lot of inserts and/or updates?try updating stats with fullscanby the way, do stored procedures in question normally perform index seeks?
Most or all of these stored procs are returning a short recordset, e.g., all the organization names within a company, or perhaps all the employees that have completed some task. I'd say that index seeks or lookups are common in these procs. The problem behavior seems to suggest that the procs begin to do table scans for some reason.Doesn't the nightly reindex accomplish a stat update with fullscan? |
 |
|
|
SamC
White Water Yakist
3467 Posts |
Posted - 2009-12-23 : 15:03:27
|
quote: Originally posted by yosiasz how are you calling the stored procedures from your web page? adodb or what? are you closing any connections to the stored procedure properly, cleaning up after use of stored procedure? show us your web code please.
We're using ado to call the procs from asp pages. Every page is good at closing the connection. I am reluctant to think that the problem is related to the web/asp/ado if recompiling the failing stored procedure corrects the issue. If restarting the website fixed the problem, then looking at the ado/asp would make sense. |
 |
|
|
tkizer
Almighty SQL Goddess
38200 Posts |
Posted - 2009-12-23 : 15:36:20
|
| If recompiling the stored procedure fixes the problem, then you've got a bad plan in cache. Bad plans are typically due to out of date statistics. You may need to run sp_updatestats in the middle of the day if this is a frequent occurrence, or perhaps just update the stats of the problematic table. Another culprit of a bad plan in cache is due to the use of IF logic in a stored procedure to decide which query to run. You can only have one plan in cache per stored procedure, so it is recommended that you break this IF logic into multiple stored procedures instead.Tara KizerMicrosoft MVP for Windows Server System - SQL Serverhttp://weblogs.sqlteam.com/tarad/Subscribe to my blog"Let's begin with the premise that everything you've done up until this point is wrong." |
 |
|
|
tkizer
Almighty SQL Goddess
38200 Posts |
|
|
yosiasz
Master Smack Fu Yak Hacker
1635 Posts |
Posted - 2009-12-23 : 15:40:46
|
| you never know :O) also any underlying views that might be problematic. this sounds indeed like a very strange problem. I would do a trace from web page onward. Is this a new problem that just suddenly appeare. Has something changed in your environment on web server and sql backend? new updates anything...are you calling the tables directly in your sproc or are you using views. if not try using views and see what happens. has underlying table changed but views haev not been refreshed?<><><><><><><><><><><><><><><><><><><><><><><><><>If you don't have the passion to help people, you have no passion |
 |
|
|
RiverReady
Starting Member
3 Posts |
Posted - 2009-12-23 : 15:57:29
|
| We had a very similar problem with poor performance for a report that was running via odbc and fine performance from SSMS and never found a true solution. However, in troubleshooting the problem, we were eventually able to recreate it in SSMS.Query Plans are cached based on more than just the query. I believe things like 'ansi padding', etc are taken into consideration.The defaults for SSMS are different than they are for odbc. When we manually overrode the settings to match the defaults for odbc, we were able to create the same problem within SSMS.There is a dmv available to get the cached execution plan for the 'slow query' and review it. We were able to compare this to the 'good' execution plan but we worked around our problem before identifying a root cause.In our case, we had a stored procedure that called several other stored procedures, we moved most of this process to a scheduled job with separate stored procedure calls at each job step.Note: When I find some of the links I had referenced when troubleshooting this before, I'll post them.One that was useful was 'What sql is currently executing?'http://www.sqlservercentral.com/articles/DMV/64425/We had a multi-step stored procedure, and this let us identify which sql statement within the nested stored procedure was having the problem. |
 |
|
|
RiverReady
Starting Member
3 Posts |
Posted - 2009-12-23 : 16:06:28
|
| This link is someone else that sounds like they had a similar problem to what you are having.http://www.windows-tech.info/15/8b8d1b0de852fe18.phpThis link may help in finding and researching cached query planshttp://www.mssqltips.com/tip.asp?tip=1829In our case, the 'difference' between the cached requests was related to Arithabort:Arithabort was set to zero in the slow query and 1 in the fast query.If we manually set Arithabort to zero within SSMS, we ended up with the same 'bad' query plan we received through odbc. |
 |
|
|
russell
Pyro-ma-ni-yak
5072 Posts |
Posted - 2009-12-23 : 16:25:05
|
| there's lots of replies here...please go back and read Tara's reply. it is the correct one.by the way, rebuilding indexes does in fact update stats but not with a fullscan. |
 |
|
|
tkizer
Almighty SQL Goddess
38200 Posts |
Posted - 2009-12-23 : 16:55:57
|
quote: Originally posted by yosiasz you never know :O) also any underlying views that might be problematic. this sounds indeed like a very strange problem. I would do a trace from web page onward. Is this a new problem that just suddenly appeare. Has something changed in your environment on web server and sql backend? new updates anything...are you calling the tables directly in your sproc or are you using views. if not try using views and see what happens. has underlying table changed but views haev not been refreshed?
Switching from a table to a view would not fix this problem. It's a bad plan in cache, plain and simple.Tara KizerMicrosoft MVP for Windows Server System - SQL Serverhttp://weblogs.sqlteam.com/tarad/Subscribe to my blog"Let's begin with the premise that everything you've done up until this point is wrong." |
 |
|
|
tkizer
Almighty SQL Goddess
38200 Posts |
Posted - 2009-12-23 : 16:57:59
|
quote: Originally posted by russell there's lots of replies here...please go back and read Tara's reply. it is the correct one.by the way, rebuilding indexes does in fact update stats but not with a fullscan.
I think I'm right too. I've encountered this issue numerous times in our production environments and have worked directly with Microsoft on them.Another reason for a bad plan in cache is parameter sniffing.Tara KizerMicrosoft MVP for Windows Server System - SQL Serverhttp://weblogs.sqlteam.com/tarad/Subscribe to my blog"Let's begin with the premise that everything you've done up until this point is wrong." |
 |
|
|
yosiasz
Master Smack Fu Yak Hacker
1635 Posts |
Posted - 2009-12-23 : 17:05:45
|
| yes go with Tara's point, I know she is right! I did not see her post otherwise I would not have commented at all after her.<><><><><><><><><><><><><><><><><><><><><><><><><>If you don't have the passion to help people, you have no passion |
 |
|
|
SamC
White Water Yakist
3467 Posts |
Posted - 2009-12-23 : 23:17:45
|
I thought Tara's comments would be spot on. We do use IF statements in many stored procs. I quickly took a look at the proc that failed today and... no IFs. I'll post the code here so you can see for yourselves.I suspect parameter sniffing could be related, but as I understand parameter sniffing, the proc should use a bad query plan "all the time". In fact, these procs can run fine for days or weeks. It's like pulling a random number out of a hat, I never know which one will fail next.I'll read over the other links posted.Here's the proc that failed today:/****** Object: StoredProcedure [dbo].[AD_Branches_3] Script Date: 12/23/2009 23:10:39 ******/SET ANSI_NULLS ONGOSET QUOTED_IDENTIFIER ONGOCREATE PROCEDURE [dbo].[AD_Branches_3]/************************************************************************************************* Purpose: Return recordset of Centers and Branches for a specific Clientexec dbo.AD_Branches_3 @CallerAdminID=17109,@CourseID=4532, @CenterID=0,@BranchID=0,@Visibility=0*************************************************************************************************/ @CallerAdminID INT , @CourseID INT , @CenterID INT = NULL , @BranchID INT = NULL , @Visibility BIT = 0 , -- Set to 1 for User Update @Wildcard INT = NULL ASDECLARE @Adminlevel INT, @Admincenterid INT, @Adminbranchid INT, @ClientID INT-- Get the controlling parameters for this AdministratorSELECT @ClientID = ClientID, @AdminLevel = AdminLevel, @AdminCenterID = CenterID, @AdminBranchID = BranchID FROM dbo.Admins A WHERE AdminID = @CallerAdminID-- SET THE SELECTED CENTERID TO NULL IF IT IS GLOBALSET @CenterID = CASE WHEN @CenterID = 0 THEN NULL ELSE @CenterID ENDSET @BranchID = CASE WHEN @BranchID = 0 THEN NULL ELSE @BranchID END-- RESTRICT VISIBILITY ACCORDING TO ADMINLEVELSET @AdminCenterID = CASE WHEN @AdminLevel <= 1 OR @Visibility = 1 THEN NULL ELSE @AdminCenterID ENDSET @AdminBranchID = CASE WHEN @AdminLevel <= 2 OR @Visibility = 1 THEN NULL ELSE @AdminBranchID END-- RETURN THE SCOPED RECORDSETSELECT B.BranchID, B.Branchname + ' (' + CAST(IsNull(A.TotalBranchID, 0) AS VARCHAR) + ')' As Branchname, B.CenterID, C.Centername FROM dbo.Branches B LEFT OUTER JOIN ( SELECT BranchID, COUNT(*) As TotalBranchID -- Count of assigned users for each BranchID FROM dbo.Users U LEFT OUTER JOIN dbo.CourseAssignments CA ON CA.CourseID = @CourseID AND CA.UserID = U.UserID AND CA.Assigned = 1 WHERE U.ClientID = @ClientID AND U.Deleted = 0 GROUP BY BranchID ) A ON A.BranchID = B.BranchID INNER JOIN dbo.Centers C ON C.CenterID = B.CenterID AND C.ClientID = B.ClientID WHERE B.ClientID = @ClientID AND (B.Inactive = 0 OR B.BranchID = @BranchID) -- Include SELECTED BranchID even if Inactive AND (@CenterID IS NULL OR B.CenterID = @CenterID) AND (@AdminBranchID IS NULL OR B.Branchname LIKE ( -- All branchnames wildcarded to match Admin's Branchname SELECT Branchname FROM dbo.Branches WHERE B.ClientID =@ClientID AND (@AdminCenterID IS NULL OR B.CenterID = @AdminCenterID) AND BranchID = @AdminBranchID)+'%') ORDER BY Centername, Branchname |
 |
|
|
ehorn
Master Smack Fu Yak Hacker
1632 Posts |
Posted - 2009-12-23 : 23:57:52
|
quote: Originally posted by SamC ...I suspect parameter sniffing could be related...
Hi Sam,I just read an article on "parameter sniffing" and thought it was worth linking to as the author posted a remedy (which he agreed was wierd - but worked in their situation).http://elegantcode.com/2008/05/17/sql-parameter-sniffing-and-what-to-do-about-it/FWIW, we use conditional where clauses quite often (similar to your sample) and get flaky plans every once in a while too. The following author discusses it mid-way down the article ("Static SQL"):http://www.sommarskog.se/dyn-search-2005.htmlGood luck with a resolution. |
 |
|
|
SamC
White Water Yakist
3467 Posts |
Posted - 2009-12-24 : 00:38:32
|
quote: Originally posted by ehornhttp://elegantcode.com/2008/05/17/sql-parameter-sniffing-and-what-to-do-about-it/
I had an ah-ha moment reading this. I forgot another symptom...When a stored procedure fails it usually fails for one client, not all of them. Exactly like the symptom reported in the link above.The solution proposed above (indirect parameter references) may be a way out, but there are a lot of stored procedures making this quite a job. It also doesn't answer all the questions.- Why does the stored procedure timeout (for specific clients) when invoked from the web but run correctly for all clients within SSMS? This unanswered question appears both in my experience and in the link quoted above. What is it about web access passing EXACTLY identical parameters that causes the proc to do table scans? Wouldn't a skewed query plan also fail in SSMS?
|
 |
|
|
ehorn
Master Smack Fu Yak Hacker
1632 Posts |
Posted - 2009-12-24 : 00:59:49
|
quote: Originally posted by SamC
quote: Originally posted by ehornhttp://elegantcode.com/2008/05/17/sql-parameter-sniffing-and-what-to-do-about-it/
...The solution proposed above (indirect parameter references) may be a way out, but there are a lot of stored procedures making this quite a job. It also doesn't answer all the questions.- Why does the stored procedure timeout (for specific clients) when invoked from the web but run correctly for all clients within SSMS? This unanswered question appears both in my experience and in the link quoted above. What is it about web access passing EXACTLY identical parameters that causes the proc to do table scans? Wouldn't a skewed query plan also fail in SSMS?
It is hard for me to rationalize the differences between Web and SMSS calls. What is different? (network, unoptimized input parameter combinations, etc..., on and on)-Can we eliminate any potential network issues?-Are you performing any web logging (i.e. the error and /or the db proc calls from the web pages)?-Can you investigate the parameter combinations which are causing the timeouts?-Do you have a way of logging the web page proc calls (i.e. the parameters) to test against the pre-compiled plans?-Are you calling the proc (in SMSS) with the same inputs params as the user(s) who is experiencing the timeouts?-Have you tried (OPTION) RECOMPILE? Is this a performance option which is feasible or can it be implemented for a period to see if this resolves the timeouts?As usual, more info leads to more questions... :)My best guess is that you have an execution plan which is not optimized for a certain combination of input params (and based on the latest plan (roll the dice), that might be like a dog chasing its tail). As Tara mentions, Stats and indexes should be examined, then if you still experience the issue and it is feasible, I might force a recompile and re-evaluate whether that resolves the timeouts, then look for an alternative (possible refactor) if performance is not acceptable. |
 |
|
|
RiverReady
Starting Member
3 Posts |
Posted - 2009-12-24 : 18:28:53
|
| It was this query that allowed me to see what settings for things like 'arithabort' were different between my SSMS session and my 'hung/slow' odbc session.After I knew what was different, I was able to enable those options with tsql commands like SET ARITHABORT ONand get the same bad query plan.SELECT SDES.session_id , SDES.status , SDES.context_info , SDES.cpu_time , memory_usage , total_scheduled_time , SDES.total_elapsed_time , endpoint_id , last_request_start_time , last_request_end_time , SDES.reads , SDES.writes , SDES.logical_reads , is_user_process , SDES.text_size , SDES.language , SDES.date_format , SDES.date_first , SDES.quoted_identifier , SDES.arithabort , SDES.ansi_null_dflt_on , SDES.ansi_defaults , SDES.ansi_warnings , SDES.ansi_padding , SDES.ansi_nulls , SDES.concat_null_yields_null , SDES.transaction_isolation_level , SDES.lock_timeout , SDES.deadlock_priority , SDES.row_count , SDES.prev_errorFROM sys.dm_exec_cached_plans SDECP INNER JOIN sys.dm_exec_requests SDER ON SDECP.plan_handle = SDER.plan_handle INNER JOIN sys.dm_exec_sessions SDES ON SDER.session_id = SDES.session_idWHERE SDES.login_name = 'DOMAIN\username' |
 |
|
|
|