Please start any new threads on our new site at https://forums.sqlteam.com. We've got lots of great SQL Server experts to answer whatever question you can come up with.

 All Forums
 SQL Server 2005 Forums
 Transact-SQL (2005)
 ignore duplicate records while isnerting

Author  Topic 

tsw_mik
Starting Member

11 Posts

Posted - 2009-05-26 : 07:19:18
I'm trying to create a script that would copy some rows from one table to another, but the destination one has a unique check on one of the columns, so I simply want to ignore the duplicate rows, how can I write a query to do that?
I tried setting the IGNORE_DUP_KEY to ON before insert and back to OFF after it, but then I get "Incorrect syntax near '('." (even though the code I'm using was auto-generated and should be ok).
Below is the code I was trying to use:



ALTER TABLE [dbo].[tblUsers] ADD CONSTRAINT [IX_tblUsers] UNIQUE NONCLUSTERED
(
[Ein] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = ON, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]

/*INSERT here*/

ALTER TABLE [dbo].[tblUsers] ADD CONSTRAINT [IX_tblUsers] UNIQUE NONCLUSTERED
(
[Ein] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]

tsw_mik
Starting Member

11 Posts

Posted - 2009-05-26 : 07:27:26
My INSERT looks like this:

INSERT INTO tblUsers (Ein, FirstName, LastName)
(SELECT ein, first, last
FROM OLD_tblExtraPeople)

I was also thinking I could use something like:

INSERT INTO tblUsers (Ein, FirstName, LastName)
(SELECT ein, first, last
FROM OLD_tblExtraPeople
--WHERE ein IS UNIQUE
)

but is there anything like "WHERE ein IS UNIQUE" I could use?
Go to Top of Page

Transact Charlie
Master Smack Fu Yak Hacker

3451 Posts

Posted - 2009-05-26 : 07:37:03
can't you just use DISTINCT?


INSERT INTO tblUsers (Ein, FirstName, LastName)
SELECT DISTINCT
ein
, first
, last
FROM
OLD_tblExtraPeople





Charlie
===============================================================
Msg 3903, Level 16, State 1, Line 1736
The ROLLBACK TRANSACTION request has no corresponding BEGIN TRANSACTION
Go to Top of Page

tsw_mik
Starting Member

11 Posts

Posted - 2009-05-26 : 07:39:04
DISTINCT gives me unique rows and I need rows with only one column unique


e.g. From this table:

FirstName, LastName, EIN
qwe, qwe, 123
asd, asd, 124
zxc, zxc, 124
qwe, qwe, 125

I only need to get:

FirstName, LastName, EIN
qwe, qwe, 123
asd, asd, 124
qwe, qwe, 125
Go to Top of Page

Transact Charlie
Master Smack Fu Yak Hacker

3451 Posts

Posted - 2009-05-26 : 07:39:50
And if you don't want to insert any ein that already exists then:

INSERT INTO tblUsers (Ein, FirstName, LastName)
SELECT DISTINCT
o.ein
, o.first
, o.last
FROM
OLD_tblExtraPeople o
LEFT JOIN tblUser t ON t.[ein] = o.[ein]
WHERE
o.[ein] IS NULL



Charlie
===============================================================
Msg 3903, Level 16, State 1, Line 1736
The ROLLBACK TRANSACTION request has no corresponding BEGIN TRANSACTION
Go to Top of Page

Transact Charlie
Master Smack Fu Yak Hacker

3451 Posts

Posted - 2009-05-26 : 07:42:09
quote:
DISTINCT gives me unique rows and I need rows with only one column unique


OK -- so you have two different people with the same EIN?

Which would you like to choose for an individual EIN.

I think it's time you gave some sample data.......


Charlie
===============================================================
Msg 3903, Level 16, State 1, Line 1736
The ROLLBACK TRANSACTION request has no corresponding BEGIN TRANSACTION
Go to Top of Page

tsw_mik
Starting Member

11 Posts

Posted - 2009-05-26 : 07:49:03
Yes, there are some user entries with the same EIN - in that case I only want to copy the first instance and ignore the next ones

Please see the example I added to my last reply.
Go to Top of Page

tsw_mik
Starting Member

11 Posts

Posted - 2009-05-26 : 07:50:32
quote:
Originally posted by Transact Charlie

And if you don't want to insert any ein that already exists then:

INSERT INTO tblUsers (Ein, FirstName, LastName)
SELECT DISTINCT
o.ein
, o.first
, o.last
FROM
OLD_tblExtraPeople o
LEFT JOIN tblUser t ON t.[ein] = o.[ein]
WHERE
o.[ein] IS NULL



Charlie
===============================================================
Msg 3903, Level 16, State 1, Line 1736
The ROLLBACK TRANSACTION request has no corresponding BEGIN TRANSACTION




This query returns no rows.
Go to Top of Page

Transact Charlie
Master Smack Fu Yak Hacker

3451 Posts

Posted - 2009-05-26 : 07:53:43
quote:

Yes, there are some user entries with the same EIN - in that case I only want to copy the first instance and ignore the next ones

Please see the example I added to my last reply.


In your example there's nothing to indicate which would be inserted first given a set of data. If you only want the first name alphabeticaaly that's fine.

Give me a few minutes.




Charlie
===============================================================
Msg 3903, Level 16, State 1, Line 1736
The ROLLBACK TRANSACTION request has no corresponding BEGIN TRANSACTION
Go to Top of Page

tsw_mik
Starting Member

11 Posts

Posted - 2009-05-26 : 08:01:15
The order of inserting doesn't really matter as long as we only get unique EINs in the destination table.

I assume that the source table will not be sorted, so the first instance of EIN would be inserted, and next ones would be ignored, but we may as well do some sorting on regDate column.
Go to Top of Page

Transact Charlie
Master Smack Fu Yak Hacker

3451 Posts

Posted - 2009-05-26 : 08:03:37
How about thia:

1) Will insert only EIN's that don't already exist.
2) Chooses the first name based on surname, firstname order


DECLARE @foo TABLE (
[ein] INT UNIQUE
, [firstname] NVARCHAR(255)
, [surname] NVARCHAR(255)
)

DECLARE @bar TABLE (
[ein] INT
, [firstname] NVARCHAR(255)
, [surname] NVARCHAR(255)
)

-- Populate @foo
INSERT @foo ([ein], [firstname], [surname])
SELECT 1, 'Fred', 'Flintstone'
UNION SELECT 2, 'Barney', 'Rubble'
UNION SELECT 4, 'Jonus', 'Venture'

INSERT @bar ([ein], [firstname], [surname])
SELECT 1, 'Scooby', 'Doo'
UNION ALL SELECT 1, 'SCOOBy', 'DOO'
UNION ALL SELECT 3, 'Brock', 'Samson'
UNION ALL SELECT 3, 'b', 'Samson'
UNION ALL SELECT 3, 'Samson,', 'Brock'

SELECT 'FOO', * FROM @foo
SELECT 'BAR', * FROM @bar

INSERT @foo ([ein], [firstname], [surname])
SELECT
a.[ein]
, a.[firstname]
, a.[surname]
FROM
(
SELECT
ROW_NUMBER() OVER(PARTITION BY [ein] ORDER BY [surname], [firstname]) AS [pos]
, [ein] AS [ein]
, [firstname] AS [firstname]
, [surname] AS [surname]
FROM
@bar
)
a
LEFT JOIN @foo f ON f.[ein] = a.[ein]
WHERE
a.[pos] = 1
AND f.[ein] IS NULL

SELECT 'FOO AFTER INSERT', * FROM @foo



Charlie
===============================================================
Msg 3903, Level 16, State 1, Line 1736
The ROLLBACK TRANSACTION request has no corresponding BEGIN TRANSACTION
Go to Top of Page

tsw_mik
Starting Member

11 Posts

Posted - 2009-05-26 : 08:11:13
I'm getting the error:
'ROW_NUMBER' is not a recognized function name.
Go to Top of Page

tsw_mik
Starting Member

11 Posts

Posted - 2009-05-26 : 08:13:44
It's SQL Server 2000
Go to Top of Page

Transact Charlie
Master Smack Fu Yak Hacker

3451 Posts

Posted - 2009-05-26 : 08:17:20
why did you post in a sql 2005 forum them?





Charlie
===============================================================
Msg 3903, Level 16, State 1, Line 1736
The ROLLBACK TRANSACTION request has no corresponding BEGIN TRANSACTION
Go to Top of Page

tsw_mik
Starting Member

11 Posts

Posted - 2009-05-26 : 08:18:47
Heh, sorry. I just noticed it's 2000 this minute.
Go to Top of Page

Transact Charlie
Master Smack Fu Yak Hacker

3451 Posts

Posted - 2009-05-26 : 08:21:39
quote:

The order of inserting doesn't really matter as long as we only get unique EINs in the destination table.

I assume that the source table will not be sorted, so the first instance of EIN would be inserted, and next ones would be ignored, but we may as well do some sorting on regDate column.



OK -- so there is a regData column? would have been ahndy to know that for your posted data set.

Before I suggest anything else:

DO YOU HAVE A PRIMARY KEY ON THE OLD_tblExtraPeople table that you haven't told us about (like the date column)?
If you do this will be easy.


Charlie
===============================================================
Msg 3903, Level 16, State 1, Line 1736
The ROLLBACK TRANSACTION request has no corresponding BEGIN TRANSACTION
Go to Top of Page

tsw_mik
Starting Member

11 Posts

Posted - 2009-05-26 : 08:25:58
the old table has a personId column, which is a PK
also - the new table is empty, we're migriting the users from old table to it
also the old table has a regDate column (not nullable)
Go to Top of Page

Transact Charlie
Master Smack Fu Yak Hacker

3451 Posts

Posted - 2009-05-26 : 08:36:48
OK -- with personID is all you need!

2000 code



DECLARE @foo TABLE (
[ein] INT UNIQUE
, [firstname] NVARCHAR(255)
, [surname] NVARCHAR(255)
)

DECLARE @bar TABLE (
[personId] INT IDENTITY(1,1)
, [ein] INT
, [firstname] NVARCHAR(255)
, [surname] NVARCHAR(255)
, [regDate] DATETIME
)

-- Populate @foo
INSERT @foo ([ein], [firstname], [surname])
SELECT 1, 'Fred', 'Flintstone'
UNION SELECT 2, 'Barney', 'Rubble'
UNION SELECT 4, 'Jonus', 'Venture'

INSERT @bar ([ein], [firstname], [surname], [regDate])
SELECT 1, 'Scooby', 'Doo', '20090101'
UNION ALL SELECT 1, 'SCOOBy', 'DOO', '20090101'
UNION ALL SELECT 3, 'Brock', 'Samson', '20090101'
UNION ALL SELECT 3, 'b', 'Samson', '20090401'
UNION ALL SELECT 3, 'Samson,', 'Brock', '20090401'
UNION ALL SELECT 10, '', '', '20090401'

SELECT 'FOO', * FROM @foo
SELECT 'BAR', * FROM @bar


INSERT @foo ([ein], [firstname], [surname])
SELECT
b.[ein]
, b.[firstname]
, b.[surname]
FROM
@bar b

JOIN (
SELECT
[ein] AS [ein]
, MAX([personId]) AS [personId]
FROM
@bar
GROUP BY
[ein]
)
maxPId ON maxPId.[personId] = b.[personId]

LEFT JOIN @foo f ON f.[ein] = b.[ein]
WHERE
f.[ein] IS NULL

SELECT 'FOO AFTER INSERT', * FROM @foo




Charlie
===============================================================
Msg 3903, Level 16, State 1, Line 1736
The ROLLBACK TRANSACTION request has no corresponding BEGIN TRANSACTION
Go to Top of Page

tsw_mik
Starting Member

11 Posts

Posted - 2009-05-26 : 08:40:47
Excellent. Thanks a lot. That's all I need.
Go to Top of Page

Transact Charlie
Master Smack Fu Yak Hacker

3451 Posts

Posted - 2009-05-26 : 08:42:40
you won't need the jion back to the parent you are inserting into if it's empty -- it won't take up much if any performance though if you leave the join in.


Charlie
===============================================================
Msg 3903, Level 16, State 1, Line 1736
The ROLLBACK TRANSACTION request has no corresponding BEGIN TRANSACTION
Go to Top of Page
   

- Advertisement -