Please start any new threads on our new site at https://forums.sqlteam.com. We've got lots of great SQL Server experts to answer whatever question you can come up with.

 All Forums
 SQL Server 2005 Forums
 SSIS and Import/Export (2005)
 SSIS

Author  Topic 

whitmoj
Yak Posting Veteran

68 Posts

Posted - 2012-10-18 : 05:52:01
Hi Not sure if this can be done in ssis. I currently have an ssis solution that will go to a folder and process all flat files of a certain type when its done I have it saving the file name to a table in sql. What I am after is a step at the start that will check the table in sql and see if the file as already been processed and if so finish and move to the next package, and if not process the file. Any help

Whitmoj
If I have inspired one person today then my job is done.

theboyholty
Posting Yak Master

226 Posts

Posted - 2012-12-19 : 06:16:56
One approach to this could be to load each table into SQL and put it a stage table (truncated before each file is imported), then run a separate routine to import the data from the stage table to the main table. At this stage you could do a check to see if the file has been imported previously, and if it has then do nothing.
Its probably not the most elegant solution but it should do the job.

---------------------------------------------------------------------------------
http://www.mannyroadend.co.uk A Bury FC supporters website and forum
Go to Top of Page

nigelrivett
Master Smack Fu Yak Hacker

3385 Posts

Posted - 2012-12-19 : 06:52:19
Two approaches - I use both
Have a control table which contains the file name and status.
One task looks at the folder and populates with the file names.
Another task then gets filenames from that table to load and updates the status.
That means that to reload a file you just need to update the status in the table - especially useful for testing.
Separating the tasks means that you can poll more frequently with the first task and keep a record of when the files arrive.

Another option is to move/rename the file when it is processed.

As I said I always do both. Process all files in a folder using the control table and move to an archive folder when complete.

The control table and status can also be used to split the processing into multiple small steps which is especially handy for troubleshooting and restarting processes that have failed.

==========================================
Cursors are useful if you don't know sql.
SSIS can be used in a similar way.
Beer is not cold and it isn't fizzy.
Go to Top of Page

visakh16
Very Important crosS Applying yaK Herder

52326 Posts

Posted - 2012-12-19 : 07:46:25
quote:
Originally posted by whitmoj

Hi Not sure if this can be done in ssis. I currently have an ssis solution that will go to a folder and process all flat files of a certain type when its done I have it saving the file name to a table in sql. What I am after is a step at the start that will check the table in sql and see if the file as already been processed and if so finish and move to the next package, and if not process the file. Any help

Whitmoj
If I have inspired one person today then my job is done.


Have a procedure which takes filename as parameter and returns a boolean result to indicate file exists or not.Populate a boolean variable created in SSIS with boolean result.
Use precedence constraint with expression and constraint option and add condition @FileExists == false and connect it to process and subsequent steps. this will make sure it ignores existing files

see similar logic here

http://visakhm.blogspot.in/2012/05/package-to-implement-daily-processing.html

only difference is to check for existence of file instead of pattern match in above example

------------------------------------------------------------------------------------------------------
SQL Server MVP
http://visakhm.blogspot.com/

Go to Top of Page
   

- Advertisement -