SQL Server Forums
Profile | Register | Active Topics | Members | Search | Forum FAQ
 
Register Now and get your question answered!
Username:
Password:
Save Password
Forgot your Password?

 All Forums
 SQL Server 2005 Forums
 SSIS and Import/Export (2005)
 SSIS
 New Topic  Reply to Topic
 Printer Friendly
Author Previous Topic Topic Next Topic  

whitmoj
Yak Posting Veteran

United Kingdom
68 Posts

Posted - 10/18/2012 :  05:52:01  Show Profile  Reply with Quote
Hi Not sure if this can be done in ssis. I currently have an ssis solution that will go to a folder and process all flat files of a certain type when its done I have it saving the file name to a table in sql. What I am after is a step at the start that will check the table in sql and see if the file as already been processed and if so finish and move to the next package, and if not process the file. Any help

Whitmoj
If I have inspired one person today then my job is done.

theboyholty
Posting Yak Master

United Kingdom
226 Posts

Posted - 12/19/2012 :  06:16:56  Show Profile  Visit theboyholty's Homepage  Reply with Quote
One approach to this could be to load each table into SQL and put it a stage table (truncated before each file is imported), then run a separate routine to import the data from the stage table to the main table. At this stage you could do a check to see if the file has been imported previously, and if it has then do nothing.
Its probably not the most elegant solution but it should do the job.

---------------------------------------------------------------------------------
http://www.mannyroadend.co.uk A Bury FC supporters website and forum
Go to Top of Page

nigelrivett
Flowing Fount of Yak Knowledge

United Kingdom
3383 Posts

Posted - 12/19/2012 :  06:52:19  Show Profile  Visit nigelrivett's Homepage  Reply with Quote
Two approaches - I use both
Have a control table which contains the file name and status.
One task looks at the folder and populates with the file names.
Another task then gets filenames from that table to load and updates the status.
That means that to reload a file you just need to update the status in the table - especially useful for testing.
Separating the tasks means that you can poll more frequently with the first task and keep a record of when the files arrive.

Another option is to move/rename the file when it is processed.

As I said I always do both. Process all files in a folder using the control table and move to an archive folder when complete.

The control table and status can also be used to split the processing into multiple small steps which is especially handy for troubleshooting and restarting processes that have failed.

==========================================
Cursors are useful if you don't know sql.
SSIS can be used in a similar way.
Beer is not cold and it isn't fizzy.
Go to Top of Page

visakh16
Very Important crosS Applying yaK Herder

India
52325 Posts

Posted - 12/19/2012 :  07:46:25  Show Profile  Reply with Quote
quote:
Originally posted by whitmoj

Hi Not sure if this can be done in ssis. I currently have an ssis solution that will go to a folder and process all flat files of a certain type when its done I have it saving the file name to a table in sql. What I am after is a step at the start that will check the table in sql and see if the file as already been processed and if so finish and move to the next package, and if not process the file. Any help

Whitmoj
If I have inspired one person today then my job is done.


Have a procedure which takes filename as parameter and returns a boolean result to indicate file exists or not.Populate a boolean variable created in SSIS with boolean result.
Use precedence constraint with expression and constraint option and add condition @FileExists == false and connect it to process and subsequent steps. this will make sure it ignores existing files

see similar logic here

http://visakhm.blogspot.in/2012/05/package-to-implement-daily-processing.html

only difference is to check for existence of file instead of pattern match in above example

------------------------------------------------------------------------------------------------------
SQL Server MVP
http://visakhm.blogspot.com/

Go to Top of Page
  Previous Topic Topic Next Topic  
 New Topic  Reply to Topic
 Printer Friendly
Jump To:
SQL Server Forums © 2000-2009 SQLTeam Publishing, LLC Go To Top Of Page
This page was generated in 0.03 seconds. Powered By: Snitz Forums 2000