Please start any new threads on our new
site at https://forums.sqlteam.com. We've got lots of great SQL Server
experts to answer whatever question you can come up with.
| Author |
Topic |
|
mike123
Master Smack Fu Yak Hacker
1462 Posts |
Posted - 2004-10-28 : 23:53:35
|
Hi,I've been having some problems with my sqlserver slowing down. I recently ran a trace, and as expected I knew what my slowest running queriy by far was. I did not know how drastic the increase in performance would be when I disabled it. My server usually sits between 80-100% CPU usage at peak times, after I disable this SPROC its down at an estimated 5-25%.My experience and knowledge of optimizing queries like this is pretty limited so I'm hoping some of the SQLTeam Guru's can help me out here I'm willing to do anything to speed this up to its greatest potential. There are 28 parameters, 2 of which are for paging, and 1 for orderby. This leaves 25 parameters to be searched on, however many searches may only pass 3 or 4 values. Since its getting so many parameters I was wondering if I could create multiple search SPROCS depending on the criteria that was being searched, and execute the appropriate one. Would this help? Is this ever done in practice? I don't care if I have to make 200 SPROCS to run this query faster, I need to give this thing the biggest boost I can. Am I barking up the wrong tree ? ANY tips on what I can do here is greatly appreciated. Thanks alot once againmike123 |
|
|
mike123
Master Smack Fu Yak Hacker
1462 Posts |
Posted - 2004-10-28 : 23:56:45
|
| [code]CREATE Procedure dbo.UserSearch @Page int, @RecsPerPage int, @GenderID tinyint = NULL, @NameOnline varchar(15) = NULL, --LIKE ? @sexualityID tinyint = NULL, @minAge tinyint = NULL, @maxAge tinyint = NULL, @PostalCode varchar(7) = NULL, --LIKE ? @City varchar(25) = NULL, @StateProvID tinyint = NULL, @CountryID tinyint = NULL, @statusID tinyint = NULL, @bodyTypeID tinyint = NULL, @hairColorID tinyint = NULL, @eyeColorID tinyint = NULL, @HeightFeet tinyint = NULL, @HeightInches tinyint = NULL, @educationID tinyint = NULL, @employmentID tinyint = NULL, @drinkID tinyint = NULL, @smokeID tinyint = NULL, @memberSinceDays tinyint = NULL, @qsVideo char(2) = NULL, @qsOnline char(2) = NULL, @qsSuperPhoto char(2) = NULL, @orderBy tinyint AS Set Nocount on Declare @Rows int Create Table #Paging ( --Declare @Paging Table ( RowID int IDENTITY, userID int, NameOnline varchar(15), age tinyint, statusID tinyint, Points int, Votes int, userNote varchar(35), GenderID tinyint, [date] smalldatetime, thumb_Count tinyint, mainPhotoID int, vidExist char, SuperPhoto char ) INSERT INTO #Paging ( userID, NameOnline, age, statusID, Points, Votes, userNote, GenderID, date, thumb_Count, mainPhotoID, vidExist, SuperPhoto) SELECT TOP 50 tblUserDetails.userID, tblUserDetails.nameOnline, age, statusID, points, votes, userNote, genderID, date, ( select count(userID) as tblthumb_count from tblthumbs where status = 1 AND tblUserDetails.userid = tblthumbs.userid ) as thumb_count, (select counterID FROM tblExtraPhotos where photoID = '0' AND tblUserDetails.userid = tblExtraPhotos.userid) as mainPhotoID, case when exists ( select userID from tblvideo where tblUserDetails.userid = tblvideo.userid AND active ='1' ) then 'Y' Else 'N' end as vidExist, case when exists ( select userID from tblExtraPhotos where tblUserDetails.userid = tblExtraPhotos.userID AND active ='1' AND photoID ='99' --select userID from tblThumbs where tblUserDetails.userid = tblThumbs.userid AND active ='1' AND thumbID ='99' ) then 'Y' Else 'N' end as SuperPhoto FROM tblUserDetails WHERE tblUserDetails.active = '1' and GenderID = isNull(@GenderID, GenderID) and NameOnline like '%' + isNull(@NameOnline, '') + '%' and sexualityID = isNull(@sexualityID , sexualityID) and ( age >= @minAge OR @minAge IS NULL ) and ( age <= @maxAge OR @maxAge IS NULL ) and PostalCode like '%' + isNull(@PostalCode, '') + '%' and City like '%' + isNull(@City, '') + '%' and StateProvID = isNull(@StateProvID, StateProvID) and CountryID = isNull(@CountryID, CountryID) and statusID = isNull(@statusID, statusID) and bodyTypeID = isNull(@bodyTypeID, bodyTypeID) and hairColorID = isNull(@hairColorID, hairColorID) and eyeColorID = isNull(@eyeColorID, eyeColorID) and HeightFeet = isNull(@HeightFeet, HeightFeet) and HeightInches = isNull(@HeightInches, HeightInches) and educationID = isNull(@educationID, educationID) and employmentID = isNull(@employmentID, employmentID) and drinkID = isNull(@drinkID, drinkID) and smokeID = isNull(@smokeID, smokeID) and ( @memberSinceDays is NULL OR DateDiff(d, date, GetDate()) <= @memberSinceDays ) and ( @qsVideo != 'on' OR EXISTS ( select userID from tblvideo where userId = tblUserDetails.userID and tblvideo.active ='1') ) and ( @qsOnline != 'on' OR EXISTS ( select userID from tblActive_Users where userId = tblUserDetails.userID) ) and ( @qsSuperPhoto != 'on' OR EXISTS ( select userID from tblExtraPhotos where userId = tblUserDetails.userID AND photoID ='99') ) ORDER BY Case @OrderBy WHEN 1 THEN date WHEN 3 THEN lastLoggedIn ELSE NULL END DESC, Case WHEN @OrderBy = 2 and votes > 0 THEN (points / votes) END DESC SET @Rows = @@Rowcount Declare @FirstRow int, @LastRow int, @TotalPages int SET @FirstRow = ((@Page - 1) * @RecsPerPage) + 1 SET @LastRow = @FirstRow + (@RecsPerPage - 1) SET @TotalPages = ceiling(Cast(@rows as decimal)/ @RecsPerPage ) SELECT *, @Rows TotalRows, @TotalPages TotalPages FROM #Paging WHERE rowid BETWEEN @FirstRow and @LastRow Drop Table #Paging Set Nocount offGO[/code] |
 |
|
|
spirit1
Cybernetic Yak Master
11752 Posts |
Posted - 2004-10-29 : 04:49:42
|
i had a similar problem:i got a 50% speed up by doing this for each parameter:...AND (statusID = @statusID or @statusID is null) AND....but the catch here is that the execution plan can change dramatically how you put it:i use simple trial and error and i got:...(col1 = @col1 or @col1 is null) AND(@col2 is null or col2 = @col2) AND(col2 = @col2 or @col2 is null) AND ...it depends on how many nulls are in the specific column.hope it helps a little.Go with the flow & have fun! Else fight the flow |
 |
|
|
mike123
Master Smack Fu Yak Hacker
1462 Posts |
Posted - 2004-10-29 : 16:22:03
|
| Ok that gives me something to try. Do you think I can get more than 50% increase on this ? Is 300% dreaming?What do you think of having multiple SPROCS?Thanks again mike123 |
 |
|
|
Kristen
Test
22859 Posts |
Posted - 2004-10-29 : 19:30:22
|
If you can live with the permissions issues you could use dynamic SQLDECLARE @strSQL varchar(8000)SELECT @strSQL = 'SELECT ...FROM tblUserDetails WHERE tblUserDetails.active = ''1'''IF @GenderID IS NOT NULL SELECT @strSQL = @strSQL + ' AND GenderID = @GenderID'IF @GenderID IS NOT NULL SELECT @strSQL = @strSQL + ' AND NameOnline like ''%'' + @NameOnline + ''%'' '...EXEC sp_excuteSQL @strSQL, @GenderID tinyint, @NameOnline varchar(15), ... This will mean that the frequently occuring combinations of parameters will be in the query plan, and will be reused.Note that its OK to have a stack of parameters to sp_excuteSQL - even if they are not referenced in @strSQL :-)If the dynamic SQL is a no-no then I've got a couple of suggestions:NameOnline like '%' + isNull(@NameOnline, '') + '%' this is a performance killer and will cause a table scan every time. If instead you do:IF @NameOnline IS NOT NULL THEN SELECT @NameOnline = '%' + @NameOnline + '%' then in your WHERE clause you can put(@NameOnline IS NULL OR NameOnline like @NameOnline)which may optimise better.As an alternative you could pull data into a temporary table using the more popular criteria - in a query that was tested to give good performance (all columns indexed etc.)And then progressively delete records [from the temporary table] based on the other criteria (you need to add columns to #Paging for all the criteria you want to test separately) using statements like:IF @NameOnline IS NOT NULL DELETE #Paging WHERE NameOnline NOT like '%' + @NameOnline + '%' Finally, beware that your current statementNameOnline like '%' + isNull(@NameOnline, '') + '%'will EXCLUDE all rows in the table where NameOnline IS NULL :-(Hopefully all your LIKE tests are on NOT NULL columns :-)Kristen |
 |
|
|
mike123
Master Smack Fu Yak Hacker
1462 Posts |
Posted - 2004-10-29 : 22:14:21
|
| Hi Kristen,thanks for the pointers, I can't live with dynamic-SQL so I'll try your second recommendation for now and see what happens.What do you think of the general idea of having different SPROCS for different search criteria resultion in different execution plans for each SPROC? This SPROC is killing me and I really need to find a way to boost performance ALOT not just a little. More opinions welcome if anyone else has thoughts.Thanks againmike123 |
 |
|
|
Kristen
Test
22859 Posts |
Posted - 2004-10-29 : 22:47:19
|
| "What do you think of the general idea of having different SPROCS"As an off-hand-remark I'd say you have too many parameters to pull that trick.Kristen |
 |
|
|
hgorijal
Constraint Violating Yak Guru
277 Posts |
Posted - 2004-10-30 : 00:25:52
|
try this, ..1. Remove these Subqueries in SELECT clause and use regular JOINS instead. ( select count(userID) as tblthumb_count from tblthumbs where status = 1 AND tblUserDetails.userid = tblthumbs.userid ) as thumb_count, (select counterID FROM tblExtraPhotos where photoID = '0' AND tblUserDetails.userid = tblExtraPhotos.userid) as mainPhotoID, case when exists ( select userID from tblvideo where tblUserDetails.userid = tblvideo.userid AND active ='1' ) then 'Y' Else 'N' end as vidExist, case when exists ( select userID from tblExtraPhotos where tblUserDetails.userid = tblExtraPhotos.userID AND active ='1' AND photoID ='99' --select userID from tblThumbs where tblUserDetails.userid = tblThumbs.userid AND active ='1' AND thumbID ='99' ) then 'Y' Else 'N' end as SuperPhoto 2. Split the Query into multiple queries...[code] ( @qsVideo != 'on' OR EXISTS ( select userID from tblvideo where userId = tblUserDetails.userID and tblvideo.active ='1') ) and ( @qsOnline != 'on' OR EXISTS ( select userID from tblActive_Users where userId = tblUserDetails.userID) ) and ( @qsSuperPhoto != 'on' OR EXISTS ( select userID from tblExtraPhotos where userId = tblUserDetails.userID AND photoID ='99') ) [code]Take the above code out of this query. Then individually check the variables (@qsVideo, @qsOnline...) and delete from #Paging if required.Also, seems to me that you are using this proc to browse, calling it one time for each page. If performance is really the highest priority, you might want to look into incorporating caching techniques, so that the first time you run, you load the data into a permanent or permanent-temporary(##) table and you read this table for the subsequent pages, instead of rerunning the entire proc.hope it helps...Hemanth GorijalaBI Architect / DBA...(yuhoo!!! and now, an "Yak Master")Exchange a Dollar, we still have ONE each.Exchange an Idea, we have TWO each. |
 |
|
|
mike123
Master Smack Fu Yak Hacker
1462 Posts |
Posted - 2004-10-30 : 01:47:53
|
quote: Originally posted by Kristen "What do you think of the general idea of having different SPROCS"As an off-hand-remark I'd say you have too many parameters to pull that trick.Kristen
Yeah I estimate it at abou 400 diff sprocs for each specific combination, although i haven't double checked my math. If I could get a 300% increase in speed I will gladly, and painfully write these 400 SPROCS :) just wondering if its the most ridiculous thing you've ever seen attempted lolmike123 |
 |
|
|
mike123
Master Smack Fu Yak Hacker
1462 Posts |
Posted - 2004-10-30 : 01:48:32
|
quote: Originally posted by hgorijal try this, ..1. Remove these Subqueries in SELECT clause and use regular JOINS instead. ( select count(userID) as tblthumb_count from tblthumbs where status = 1 AND tblUserDetails.userid = tblthumbs.userid ) as thumb_count, (select counterID FROM tblExtraPhotos where photoID = '0' AND tblUserDetails.userid = tblExtraPhotos.userid) as mainPhotoID, case when exists ( select userID from tblvideo where tblUserDetails.userid = tblvideo.userid AND active ='1' ) then 'Y' Else 'N' end as vidExist, case when exists ( select userID from tblExtraPhotos where tblUserDetails.userid = tblExtraPhotos.userID AND active ='1' AND photoID ='99' --select userID from tblThumbs where tblUserDetails.userid = tblThumbs.userid AND active ='1' AND thumbID ='99' ) then 'Y' Else 'N' end as SuperPhoto 2. Split the Query into multiple queries...[code] ( @qsVideo != 'on' OR EXISTS ( select userID from tblvideo where userId = tblUserDetails.userID and tblvideo.active ='1') ) and ( @qsOnline != 'on' OR EXISTS ( select userID from tblActive_Users where userId = tblUserDetails.userID) ) and ( @qsSuperPhoto != 'on' OR EXISTS ( select userID from tblExtraPhotos where userId = tblUserDetails.userID AND photoID ='99') ) [code]Take the above code out of this query. Then individually check the variables (@qsVideo, @qsOnline...) and delete from #Paging if required.Also, seems to me that you are using this proc to browse, calling it one time for each page. If performance is really the highest priority, you might want to look into incorporating caching techniques, so that the first time you run, you load the data into a permanent or permanent-temporary(##) table and you read this table for the subsequent pages, instead of rerunning the entire proc.hope it helps...Hemanth GorijalaBI Architect / DBA...(yuhoo!!! and now, an "Yak Master")Exchange a Dollar, we still have ONE each.Exchange an Idea, we have TWO each.
thanks hgorijal I'll play around with this and see what happens :)cheersmike123 |
 |
|
|
Kristen
Test
22859 Posts |
Posted - 2004-10-30 : 07:39:55
|
"If I could get a 300% increase in speed"I would think you should get a 10 fold improvement in speed, maybe more. At least for some of the combinations of queries.But I don't think you need to do loads of SProcst ... I think pulling a "first cut" dataset into a temporary table and progresively deleting is the best strategy ... unless you are going to pull gazillions of rows from which you then delete 90% I think my first approach would be to create a "log" table and have the existing SProc record the parameters it is given. Then you could use those "real world" examples to run some benchmarks on how many rows a "first cut" would grab, compared to the actual number in the final query.You might very well have a handful of SProcs for that job - "IF @minAge IS NOT NULL AND @maxAge IS NOT NULL EXEC SProc_FirstCut_A ELSE ..." so that your "first cut" is biased towards a criteria which is provided by the user AND you know has good query performance AND is likely to produce a modest-ish "first cut"I'll be interested to know how you get on pleaseKristen |
 |
|
|
spirit1
Cybernetic Yak Master
11752 Posts |
Posted - 2004-10-30 : 12:16:38
|
when i was speeding up my search (remember those joins kristen )i thought about an sproc but in the "we" decided we're going to build it directly in the asp and execute the query directly because the user could also chose the operator to use in data compare.so we had a bunch of search conditions enabled (about 25) and we built the query for those that were chosen to search for.so the more conditions were chosen the resultset was smaller so that was no problem.i used 2 conditions that returned more than 20K rows and optimized that.i guess i'm probably saying the same as Kristen optimize the hungriest SQL and put in the first IF ... ELSE ... than do the in the descending order of search conditions.Go with the flow & have fun! Else fight the flow |
 |
|
|
Kristen
Test
22859 Posts |
Posted - 2004-10-30 : 12:47:11
|
| Do you remember what sort of speed improvement you got? Maybe you told me at the time, at my age I forget! But its always nice to reminisce about the performance gains I had when I was younger - by a week or two!Kristen |
 |
|
|
spirit1
Cybernetic Yak Master
11752 Posts |
Posted - 2004-10-30 : 14:27:50
|
heh.... i got 65% speed increase in average...i also changed the asp page a bit and got it to load 10 times faster so i guess everyone's happy now . me too as i got a raise i did forgot to look up that bit slice thing you told me about. or was that just a joke on your part?Go with the flow & have fun! Else fight the flow |
 |
|
|
Kristen
Test
22859 Posts |
Posted - 2004-10-31 : 02:15:41
|
| http://www.cs.caltech.edu/courses/cs184/winter2003/lectures/Day3.pptKristen |
 |
|
|
Arnold Fribble
Yak-finder General
1961 Posts |
Posted - 2004-10-31 : 05:59:52
|
| Or rather more abstractly,http://www.informatik.uni-bonn.de/~ralf/publications/MPC2004.pdf |
 |
|
|
spirit1
Cybernetic Yak Master
11752 Posts |
Posted - 2004-10-31 : 07:02:09
|
damn... i guess you weren't joking... that's quite a reading... so does anyone acctually use this practically???Go with the flow & have fun! Else fight the flow |
 |
|
|
Kristen
Test
22859 Posts |
Posted - 2004-10-31 : 12:23:50
|
| I'm pretty sure they used to - having two parallel tasks doing 4-bits each meant the job was done in half the time, I suppose. And back then that was a neat trick. Now you can probably do the whole job in 1 cycle, they ahve so many acres of silicon to play with, so its probably no longer relevant. But hey! I don't do no sticking hardware, so who knows!Kristen |
 |
|
|
|
|
|
|
|