| Author |
Topic |
|
j4ydh
Starting Member
18 Posts |
Posted - 2010-08-20 : 04:32:41
|
| Hi,I recently had help from the SQL TEAM forum to create an insert sproc that only added items that didn't already exist.@name varchar (50),@DT smalldatetime,@cam int = 1ASINSERT INTO x_b([name] ,DT ,cam)SELECT @name, @DT, @camWHERE NOT EXISTS( SELECT * FROM x_b xb1 WITH (SERIALIZABLE) -- apply key range lock WHERE xb1.[name] = @name)My new concern is performance over time and how to prepare either my page code or addition to the sproc when the database items carry hundreds of thousands of entries.Two concerns:1) If a large amount of additions are being added at once.. best way to prepare?2) If a large amount of items already exist in the database.. the best way to prepare?Or am I worrying for no reason?Thanks in advanceJ |
|
|
jen
Master Smack Fu Yak Hacker
4110 Posts |
Posted - 2010-08-20 : 04:40:09
|
| 1. check the fragmentation levels and defrag if necessary2. backup your trans log (if not done already via job)3. run your script4 backup your trans log (if not done already via job)5. check the fragmentation levels and defrag if necessaryif condition (a) and you can afford to set the database recovery to bulk_logged, do it--------------------keeping it simple... |
 |
|
|
SwePeso
Patron Saint of Lost Yaks
30421 Posts |
Posted - 2010-08-20 : 04:49:04
|
Are you using SQL Server 2008?Then replace your insert with MERGE. N 56°04'39.26"E 12°55'05.63" |
 |
|
|
j4ydh
Starting Member
18 Posts |
Posted - 2010-08-20 : 04:55:37
|
| Hi,I am using SQL SERVER 2008, and am aware of MERGE but have not yet put it into practice. I believe this is new?j |
 |
|
|
SwePeso
Patron Saint of Lost Yaks
30421 Posts |
Posted - 2010-08-20 : 05:48:54
|
Yes, MERGE was introduced with SQL Server 2008 and has become one of my favorite commands.It let's you do INSERT, UPDATE, DELETE within same command.Here is code snippet from the datawarehouse I am desiging. Background is that we receive a daily file from a partner and the file consists of a complete lists of member who was current restrictions of some kind.MERGE Member.factMessageRestriction AS tgtUSING ( SELECT mmrCustomerID, mmrMessageTypeID, MAX(mmrLastKnownDate) AS mmrLastKnownDate FROM StageArea.MessageRestrictions GROUP BY mmrCustomerID, mmrMessageTypeID ) AS src ON src.mmrCustomerID = tgt.mmrCustomerID AND src.mmrMessageTypeID = tgt.mmrMessageTypeIDWHEN MATCHED AND tgt.mmrLastKnownDate < src.mmrLastKnownDate THEN UPDATE SET tgt.mmrLastKnownDate = src.mmrLastKnownDate, tgt.mmrStatus = 'Renewed'WHEN NOT MATCHED BY SOURCE THEN UPDATE SET tgt.mmrStatus = 'Removed'--WHEN NOT MATCHED BY SOURCE-- THEN DELETEWHEN NOT MATCHED BY TARGET THEN INSERT ( mmrCustomerID, mmrMessageTypeID, mmrLastKnownDate, mmrStatus ) VALUES ( src.mmrCustomerID, src.mmrMessageTypeID, src.mmrLastKnownDate, 'New' ); N 56°04'39.26"E 12°55'05.63" |
 |
|
|
Transact Charlie
Master Smack Fu Yak Hacker
3451 Posts |
Posted - 2010-08-20 : 09:17:15
|
| Hi peso,Slightly off topic but.Just started using MERGE for some incremental reports I've been writing (we just upgraded to 2008) (I'm impressed its very powerful)Do you have any performance related points on MERGE? anything I should be looking for?cheers,Charlie.Charlie===============================================================Msg 3903, Level 16, State 1, Line 1736The ROLLBACK TRANSACTION request has no corresponding BEGIN TRANSACTION |
 |
|
|
Kristen
Test
22859 Posts |
Posted - 2010-08-20 : 09:35:02
|
Very useful summary of its capabilities, thanks Peso.Unfortunately it requires building a new set of templates for our CRUD generator but one rainy day soon ... |
 |
|
|
Lumbago
Norsk Yak Master
3271 Posts |
Posted - 2010-08-20 : 09:45:10
|
| Are you on Enterprise version as well? Then partitioning could be your friend. When things get *really* large partitioning is one of the few good friends you have...- LumbagoMy blog (yes, I have a blog now! just not that much content yet) -> www.thefirstsql.com |
 |
|
|
SwePeso
Patron Saint of Lost Yaks
30421 Posts |
Posted - 2010-08-20 : 10:27:42
|
Two things1) Just the ordinary checks for locks and so on2) Make sure the bindings between Source and Target only has column references. Writing like the example below will get you in trouble.-- Will get you in troubleMERGE Member.factMessageRestriction AS tgtUSING ( SELECT mmrCustomerID, mmrMessageTypeID, MAX(mmrLastKnownDate) AS mmrLastKnownDate FROM StageArea.MessageRestrictions GROUP BY mmrCustomerID, mmrMessageTypeID ) AS src ON src.mmrCustomerID = tgt.mmrCustomerID AND src.mmrMessageTypeID = 1-- Proper workaroundMERGE Member.factMessageRestriction AS tgtUSING ( SELECT mmrCustomerID, mmrMessageTypeID, MAX(mmrLastKnownDate) AS mmrLastKnownDate FROM StageArea.MessageRestrictions WHERE mmrMessageTypeID = 1 GROUP BY mmrCustomerID, mmrMessageTypeID ) AS src ON src.mmrCustomerID = tgt.mmrCustomerID N 56°04'39.26"E 12°55'05.63" |
 |
|
|
Transact Charlie
Master Smack Fu Yak Hacker
3451 Posts |
Posted - 2010-08-20 : 10:44:55
|
Cool. Cheers Peso.I'm loving merge so far. Especially for a potentially expensive query that you'd otherwise have to dump to a temp table and check that before inserting / updating.Example:MERGE INTO #changeRows AS crUSING (SELECT pw.[ownerPersonIdentityID] AS [personIdentityID] , pw.[personId] AS [personId] , pw.[effective] AS [effective] , pw.[initiated] AS [initiated] , bens.[selectionID] AS [selectionID] , sc.[ID] AS [selectionComponentID]FROM #personWorkspace pw -- Limit rows employee in a valid payroll window JOIN #empPayrollMap epm ON epm.[personIdentityID] = pw.[ownerPersonIdentityId] AND epm.[effectiveFrom] <= pw.[effective] AND epm.[effectiveTo] >= pw.[effective] -- get the benefits effective at this time CROSS APPLY dbo.RPT_FN_selectionsForPersonIdentityID ( pw.[typ] , pw.[ownerPersonIdentityID] , pw.[effective] , pw.[initiated] , @reportToDate ) AS bens -- limit rows to benefits we are reporting on JOIN #reportBens rb ON rb.[benefitID] = bens.[benefitID] -- A personal Details change for employee effects all selectionComponentes JOIN selectionComponent sc ON sc.[selectionID] = bens.[selectionID] -- Join to person to base comparison on JOIN person p ON p.[ID] = pw.[personID] -- Join the previous relevent row in #personWorkspace JOIN #personWorkspace pw2 ON pw2.[personIdentityID] = pw.[personIdentityID] AND pw2.[reportLevel] = pw.[reportLevel] - 1 -- 2nd join to person to compare to JOIN person p2 ON p2.[ID] = pw2.[personID]WHERE -- Only lines that are effective after the from DATETIME pw.[effective] >= @reportFromDate -- Dynamic Substitution of checked columns here AND ( 1 = 0 OR ( p.[typ] = 'Employee' AND p.[address1] <> p2.[address1] ) ))AS pd ON pd.[personIdentityID] = cr.[personIdentityID] AND pd.[effective] = cr.[effective] AND pd.[selectionComponentID] = cr.[selectionComponentID]-- Found a match? Great -- update the info with new personID etcWHEN MATCHED THEN UPDATE SET cr.[initiated] = pd.[initiated] , cr.[personId] = pd.[personId] , cr.[changeType] = cr.[changeType] + ' / Personal Details change'-- No match? Insert a Personal Details Change RowWHEN NOT MATCHED THEN INSERT ( [personIdentityID] , [personID] , [effective] , [initiated] , [selectionID] , [selectionComponentID] , [previousSelectionID] , [previousSelectionComponentID] , [changeType] ) VALUES ( pd.[personIdentityID] , pd.[personId] , pd.[effective] , pd.[initiated] , pd.[selectionID] , pd.[selectionComponentID] , pd.[selectionID] , pd.[selectionComponentID] , 'Personal Details Change' ); Charlie===============================================================Msg 3903, Level 16, State 1, Line 1736The ROLLBACK TRANSACTION request has no corresponding BEGIN TRANSACTION |
 |
|
|
SwePeso
Patron Saint of Lost Yaks
30421 Posts |
Posted - 2010-08-20 : 10:53:30
|
That's a good example.However... Since it's me and SQL Server 2008.1) , cr.[changeType] += ' / Personal Details change' ( New syntax for compound operators)2) WHEN NOT MATCHED BY TARGET THEN ( I know BY TARGET is the default, but...) N 56°04'39.26"E 12°55'05.63" |
 |
|
|
Transact Charlie
Master Smack Fu Yak Hacker
3451 Posts |
Posted - 2010-08-20 : 11:02:34
|
quote: +=
Cool. but I'll pass. that looks like a very easy thing to mistype / misread.(unless there's any performance benefit?)quote: BY TARGET
Good idea -- makes the code more explicit.Cheers again Peso.Charlie===============================================================Msg 3903, Level 16, State 1, Line 1736The ROLLBACK TRANSACTION request has no corresponding BEGIN TRANSACTION |
 |
|
|
Lamprey
Master Smack Fu Yak Hacker
4614 Posts |
Posted - 2010-08-20 : 12:16:02
|
quote: Originally posted by j4ydh Hi,I recently had help from the SQL TEAM forum to create an insert sproc that only added items that didn't already exist.@name varchar (50),@DT smalldatetime,@cam int = 1ASINSERT INTO x_b([name] ,DT ,cam)SELECT @name, @DT, @camWHERE NOT EXISTS( SELECT * FROM x_b xb1 WITH (SERIALIZABLE) -- apply key range lock WHERE xb1.[name] = @name)My new concern is performance over time and how to prepare either my page code or addition to the sproc when the database items carry hundreds of thousands of entries.Two concerns:1) If a large amount of additions are being added at once.. best way to prepare?2) If a large amount of items already exist in the database.. the best way to prepare?Or am I worrying for no reason?Thanks in advanceJ
More than likely, since you are doing a singleton lookup on a single table, that the EXISTS will be the more performant. However, you'll need to test it for yourself. I can't imagine that a JOIN or MERGE will be any better in this case. |
 |
|
|
Noor Anwar
Starting Member
8 Posts |
Posted - 2010-08-23 : 02:45:23
|
| i m trying to create partition function and following error from sql2005 enterprise editionMsg 7736, Level 16, State 1 Server 'SERVERNAME\SQLEXPRESS', Line 4 Partition function can only be created in Enterprise edition of SQL Server. Only Enterprise edition of SQL Server supports partitioning. Signature |
 |
|
|
Noor Anwar
Starting Member
8 Posts |
Posted - 2010-08-23 : 02:53:54
|
| sir i m trying to create partition function and following error from sql2005 enterprise editionMsg 7736, Level 16, State 1 Server 'SERVERNAME\SQLEXPRESS', Line 4 Partition function can only be created in Enterprise edition of SQL Server. Only Enterprise edition of SQL Server supports partitioning. SignatureSignature |
 |
|
|
Lumbago
Norsk Yak Master
3271 Posts |
Posted - 2010-08-23 : 03:10:36
|
| Please start a new topic when you have a problem, don't hijack others threads. The problem is that you connected to an sql express instance, not the enterprise version.- LumbagoMy blog (yes, I have a blog now! just not that much content yet) -> www.thefirstsql.com |
 |
|
|
|