Please start any new threads on our new site at https://forums.sqlteam.com. We've got lots of great SQL Server experts to answer whatever question you can come up with.

 All Forums
 General SQL Server Forums
 New to SQL Server Programming
 Performance over time

Author  Topic 

j4ydh
Starting Member

18 Posts

Posted - 2010-08-20 : 04:32:41
Hi,

I recently had help from the SQL TEAM forum to create an insert sproc that only added items that didn't already exist.

@name varchar (50),
@DT smalldatetime,
@cam int = 1

AS

INSERT INTO x_b([name] ,DT ,cam)
SELECT @name, @DT, @cam
WHERE NOT EXISTS
(
SELECT *
FROM x_b xb1 WITH (SERIALIZABLE) -- apply key range lock
WHERE xb1.[name] = @name
)

My new concern is performance over time and how to prepare either my page code or addition to the sproc when the database items carry hundreds of thousands of entries.

Two concerns:

1) If a large amount of additions are being added at once.. best way to prepare?
2) If a large amount of items already exist in the database.. the best way to prepare?

Or am I worrying for no reason?

Thanks in advance
J

jen
Master Smack Fu Yak Hacker

4110 Posts

Posted - 2010-08-20 : 04:40:09
1. check the fragmentation levels and defrag if necessary
2. backup your trans log (if not done already via job)
3. run your script
4 backup your trans log (if not done already via job)
5. check the fragmentation levels and defrag if necessary

if condition (a) and you can afford to set the database recovery to bulk_logged, do it

--------------------
keeping it simple...
Go to Top of Page

SwePeso
Patron Saint of Lost Yaks

30421 Posts

Posted - 2010-08-20 : 04:49:04
Are you using SQL Server 2008?
Then replace your insert with MERGE.



N 56°04'39.26"
E 12°55'05.63"
Go to Top of Page

j4ydh
Starting Member

18 Posts

Posted - 2010-08-20 : 04:55:37
Hi,
I am using SQL SERVER 2008, and am aware of MERGE but have not yet put it into practice. I believe this is new?

j
Go to Top of Page

SwePeso
Patron Saint of Lost Yaks

30421 Posts

Posted - 2010-08-20 : 05:48:54
Yes, MERGE was introduced with SQL Server 2008 and has become one of my favorite commands.
It let's you do INSERT, UPDATE, DELETE within same command.

Here is code snippet from the datawarehouse I am desiging. Background is that we receive a daily file from a partner and the file consists of a complete lists of member who was current restrictions of some kind.
MERGE	Member.factMessageRestriction AS tgt
USING (
SELECT mmrCustomerID,
mmrMessageTypeID,
MAX(mmrLastKnownDate) AS mmrLastKnownDate
FROM StageArea.MessageRestrictions
GROUP BY mmrCustomerID,
mmrMessageTypeID
) AS src ON src.mmrCustomerID = tgt.mmrCustomerID
AND src.mmrMessageTypeID = tgt.mmrMessageTypeID
WHEN MATCHED AND tgt.mmrLastKnownDate < src.mmrLastKnownDate
THEN UPDATE
SET tgt.mmrLastKnownDate = src.mmrLastKnownDate,
tgt.mmrStatus = 'Renewed'
WHEN NOT MATCHED BY SOURCE
THEN UPDATE
SET tgt.mmrStatus = 'Removed'
--WHEN NOT MATCHED BY SOURCE
-- THEN DELETE

WHEN NOT MATCHED BY TARGET
THEN INSERT (
mmrCustomerID,
mmrMessageTypeID,
mmrLastKnownDate,
mmrStatus
)
VALUES (
src.mmrCustomerID,
src.mmrMessageTypeID,
src.mmrLastKnownDate,
'New'
);



N 56°04'39.26"
E 12°55'05.63"
Go to Top of Page

Transact Charlie
Master Smack Fu Yak Hacker

3451 Posts

Posted - 2010-08-20 : 09:17:15
Hi peso,

Slightly off topic but.

Just started using MERGE for some incremental reports I've been writing (we just upgraded to 2008) (I'm impressed its very powerful)

Do you have any performance related points on MERGE? anything I should be looking for?

cheers,
Charlie.


Charlie
===============================================================
Msg 3903, Level 16, State 1, Line 1736
The ROLLBACK TRANSACTION request has no corresponding BEGIN TRANSACTION
Go to Top of Page

Kristen
Test

22859 Posts

Posted - 2010-08-20 : 09:35:02
Very useful summary of its capabilities, thanks Peso.

Unfortunately it requires building a new set of templates for our CRUD generator but one rainy day soon ...
Go to Top of Page

Lumbago
Norsk Yak Master

3271 Posts

Posted - 2010-08-20 : 09:45:10
Are you on Enterprise version as well? Then partitioning could be your friend. When things get *really* large partitioning is one of the few good friends you have...

- Lumbago

My blog (yes, I have a blog now! just not that much content yet)
-> www.thefirstsql.com
Go to Top of Page

SwePeso
Patron Saint of Lost Yaks

30421 Posts

Posted - 2010-08-20 : 10:27:42
Two things

1) Just the ordinary checks for locks and so on
2) Make sure the bindings between Source and Target only has column references. Writing like the example below will get you in trouble.
-- Will get you in trouble
MERGE Member.factMessageRestriction AS tgt
USING (
SELECT mmrCustomerID,
mmrMessageTypeID,
MAX(mmrLastKnownDate) AS mmrLastKnownDate
FROM StageArea.MessageRestrictions
GROUP BY mmrCustomerID,
mmrMessageTypeID
) AS src ON src.mmrCustomerID = tgt.mmrCustomerID
AND src.mmrMessageTypeID = 1

-- Proper workaround
MERGE Member.factMessageRestriction AS tgt
USING (
SELECT mmrCustomerID,
mmrMessageTypeID,
MAX(mmrLastKnownDate) AS mmrLastKnownDate
FROM StageArea.MessageRestrictions
WHERE mmrMessageTypeID = 1
GROUP BY mmrCustomerID,
mmrMessageTypeID
) AS src ON src.mmrCustomerID = tgt.mmrCustomerID





N 56°04'39.26"
E 12°55'05.63"
Go to Top of Page

Transact Charlie
Master Smack Fu Yak Hacker

3451 Posts

Posted - 2010-08-20 : 10:44:55
Cool. Cheers Peso.

I'm loving merge so far. Especially for a potentially expensive query that you'd otherwise have to dump to a temp table and check that before inserting / updating.

Example:


MERGE INTO #changeRows AS cr
USING (
SELECT
pw.[ownerPersonIdentityID] AS [personIdentityID]
, pw.[personId] AS [personId]
, pw.[effective] AS [effective]
, pw.[initiated] AS [initiated]
, bens.[selectionID] AS [selectionID]
, sc.[ID] AS [selectionComponentID]
FROM
#personWorkspace pw

-- Limit rows employee in a valid payroll window
JOIN #empPayrollMap epm ON
epm.[personIdentityID] = pw.[ownerPersonIdentityId]
AND epm.[effectiveFrom] <= pw.[effective]
AND epm.[effectiveTo] >= pw.[effective]

-- get the benefits effective at this time
CROSS APPLY dbo.RPT_FN_selectionsForPersonIdentityID (
pw.[typ]
, pw.[ownerPersonIdentityID]
, pw.[effective]
, pw.[initiated]
, @reportToDate
)
AS bens

-- limit rows to benefits we are reporting on
JOIN #reportBens rb ON rb.[benefitID] = bens.[benefitID]

-- A personal Details change for employee effects all selectionComponentes
JOIN selectionComponent sc ON sc.[selectionID] = bens.[selectionID]

-- Join to person to base comparison on
JOIN person p ON p.[ID] = pw.[personID]

-- Join the previous relevent row in #personWorkspace
JOIN #personWorkspace pw2 ON
pw2.[personIdentityID] = pw.[personIdentityID]
AND pw2.[reportLevel] = pw.[reportLevel] - 1

-- 2nd join to person to compare to
JOIN person p2 ON p2.[ID] = pw2.[personID]
WHERE
-- Only lines that are effective after the from DATETIME
pw.[effective] >= @reportFromDate

-- Dynamic Substitution of checked columns here
AND (
1 = 0
OR ( p.[typ] = 'Employee' AND p.[address1] <> p2.[address1] )
)
)
AS pd ON
pd.[personIdentityID] = cr.[personIdentityID]
AND pd.[effective] = cr.[effective]
AND pd.[selectionComponentID] = cr.[selectionComponentID]

-- Found a match? Great -- update the info with new personID etc
WHEN MATCHED THEN
UPDATE
SET
cr.[initiated] = pd.[initiated]
, cr.[personId] = pd.[personId]
, cr.[changeType] = cr.[changeType] + ' / Personal Details change'

-- No match? Insert a Personal Details Change Row
WHEN NOT MATCHED THEN
INSERT (
[personIdentityID]
, [personID]
, [effective]
, [initiated]
, [selectionID]
, [selectionComponentID]
, [previousSelectionID]
, [previousSelectionComponentID]
, [changeType]
)
VALUES (
pd.[personIdentityID]
, pd.[personId]
, pd.[effective]
, pd.[initiated]
, pd.[selectionID]
, pd.[selectionComponentID]
, pd.[selectionID]
, pd.[selectionComponentID]
, 'Personal Details Change'
);



Charlie
===============================================================
Msg 3903, Level 16, State 1, Line 1736
The ROLLBACK TRANSACTION request has no corresponding BEGIN TRANSACTION
Go to Top of Page

SwePeso
Patron Saint of Lost Yaks

30421 Posts

Posted - 2010-08-20 : 10:53:30
That's a good example.
However... Since it's me and SQL Server 2008.

1) , cr.[changeType] += ' / Personal Details change' ( New syntax for compound operators)
2) WHEN NOT MATCHED BY TARGET THEN ( I know BY TARGET is the default, but...)


N 56°04'39.26"
E 12°55'05.63"
Go to Top of Page

Transact Charlie
Master Smack Fu Yak Hacker

3451 Posts

Posted - 2010-08-20 : 11:02:34
quote:

+=

Cool. but I'll pass. that looks like a very easy thing to mistype / misread.
(unless there's any performance benefit?)

quote:

BY TARGET


Good idea -- makes the code more explicit.

Cheers again Peso.


Charlie
===============================================================
Msg 3903, Level 16, State 1, Line 1736
The ROLLBACK TRANSACTION request has no corresponding BEGIN TRANSACTION
Go to Top of Page

Lamprey
Master Smack Fu Yak Hacker

4614 Posts

Posted - 2010-08-20 : 12:16:02
quote:
Originally posted by j4ydh

Hi,

I recently had help from the SQL TEAM forum to create an insert sproc that only added items that didn't already exist.

@name varchar (50),
@DT smalldatetime,
@cam int = 1

AS

INSERT INTO x_b([name] ,DT ,cam)
SELECT @name, @DT, @cam
WHERE NOT EXISTS
(
SELECT *
FROM x_b xb1 WITH (SERIALIZABLE) -- apply key range lock
WHERE xb1.[name] = @name
)

My new concern is performance over time and how to prepare either my page code or addition to the sproc when the database items carry hundreds of thousands of entries.

Two concerns:

1) If a large amount of additions are being added at once.. best way to prepare?
2) If a large amount of items already exist in the database.. the best way to prepare?

Or am I worrying for no reason?

Thanks in advance
J


More than likely, since you are doing a singleton lookup on a single table, that the EXISTS will be the more performant. However, you'll need to test it for yourself. I can't imagine that a JOIN or MERGE will be any better in this case.
Go to Top of Page

Noor Anwar
Starting Member

8 Posts

Posted - 2010-08-23 : 02:45:23
i m trying to create partition function and following error from sql2005 enterprise edition
Msg 7736, Level 16, State 1
Server 'SERVERNAME\SQLEXPRESS', Line 4
Partition function can only be created in Enterprise edition of SQL Server. Only Enterprise edition
of SQL Server supports partitioning.


Signature
Go to Top of Page

Noor Anwar
Starting Member

8 Posts

Posted - 2010-08-23 : 02:53:54
sir i m trying to create partition function and following error from sql2005 enterprise edition
Msg 7736, Level 16, State 1
Server 'SERVERNAME\SQLEXPRESS', Line 4
Partition function can only be created in Enterprise edition of SQL Server. Only Enterprise edition
of SQL Server supports partitioning.


Signature

Signature
Go to Top of Page

Lumbago
Norsk Yak Master

3271 Posts

Posted - 2010-08-23 : 03:10:36
Please start a new topic when you have a problem, don't hijack others threads. The problem is that you connected to an sql express instance, not the enterprise version.

- Lumbago

My blog (yes, I have a blog now! just not that much content yet)
-> www.thefirstsql.com
Go to Top of Page
   

- Advertisement -