SQL Server Forums
Profile | Register | Active Topics | Members | Search | Forum FAQ
 
Register Now and get your question answered!
Username:
Password:
Save Password
Forgot your Password?

 All Forums
 SQL Server 2008 Forums
 SSIS and Import/Export (2008)
 Incremental load for million of rows
 New Topic  Reply to Topic
 Printer Friendly
Author Previous Topic Topic Next Topic  

shilpash
Posting Yak Master

103 Posts

Posted - 06/19/2013 :  09:50:09  Show Profile  Reply with Quote
Hi,

I am facing this problem.I have a source db from ado.net which has million of rows.It takes lot of time for initial load as well and now I want to perform incremental load but it does not have date or time stamp column and also no columns which increments in a sequential order.I don't want to do lookup for million of rows.Is there any other suggestion?

Thanks in advance

James K
Flowing Fount of Yak Knowledge

3719 Posts

Posted - 06/19/2013 :  10:26:22  Show Profile  Reply with Quote
Does the source table have a primary key? If there is, you can use that, and how efficient it is will depend on whether the PK is ever-increasing (which you said it is not), how random the inserts are etc. Regardless, that would be your best option.

However, if the table is a heap, I don't see any easy way to solve the problem. It is akin to a having a basket filled with white balls, and each day a few more white balls are added and you are asked top pick out the new balls that were added.

If the table is a heap and you are able to modify the source table and add an identity column to it, that will solve the problem. But you may not have access to it, and even if you do, when you alter the source table, that may cause all types of other issues that rely on that table having a fixed schema.
Go to Top of Page

shilpash
Posting Yak Master

103 Posts

Posted - 06/19/2013 :  10:41:47  Show Profile  Reply with Quote
Yes.Table has a pk and inserts are done frequently but when inserts are performed,the pk value could be anything,its not in a sequential manner like pk last value was 100 and its not going to be 101 for new rows.
Go to Top of Page

James K
Flowing Fount of Yak Knowledge

3719 Posts

Posted - 06/19/2013 :  10:57:12  Show Profile  Reply with Quote
Even if the PK is not sequential, given the scenario, that (or another unique index) is the only thing you can rely on. You could try to optimize its use - for example, by creating a bookkeeping table that has only the primary key column that would hold all the already exported primary keys and joining with that table to determine what has already been exported.
Go to Top of Page
  Previous Topic Topic Next Topic  
 New Topic  Reply to Topic
 Printer Friendly
Jump To:
SQL Server Forums © 2000-2009 SQLTeam Publishing, LLC Go To Top Of Page
This page was generated in 0.05 seconds. Powered By: Snitz Forums 2000