Please start any new threads on our new
site at https://forums.sqlteam.com. We've got lots of great SQL Server
experts to answer whatever question you can come up with.
| Author |
Topic |
|
Zathras
Starting Member
28 Posts |
Posted - 2005-02-21 : 12:09:15
|
The only way to access data from a certain design database is to write PML code that send the data to my SQL Server. The PML function basically opens a connection, send 300,000+ INSERT statements one-by-one, then executes a SP to tell my SQL Server that it has finished exporting. This SP then moves the data from the one set of tables to the "real" tables so that end users can see the complete data. This process of exporting takes about 8 to 9 hours per project becuase it is done in PML code (no choice).Sometimes that export stalls due to say the exporting computer crashes. To check for this, the SQL Server runs a SP every 20 minutes looking for projects that hasn't written data in over 20 minutes and doesn't have the finish marker. They way I check for data is that every single INSERT has a TimeStamp DateTime field as the very last field and the data is inserted as a GETDATE().My question is, is there any other way I can get the last time data was written for a certain project (connection?) other than having a separate TimeStamp DateTime field. I'm trying to speed up the exports since they take so long. Not having 300,000+ GETDATE() function calls seems to be a good place to start.EDIT: The field is a datetime field (not the TimeStamp datatype) |
|
|
spirit1
Cybernetic Yak Master
11752 Posts |
Posted - 2005-02-21 : 12:28:10
|
nope don't think there is.could you change the datetime to maybe integer that increments for every project?EDIT:just for the fun of it i ran this on my server which is pentium 4 1.6 GHz and it took 4 minutes 48 secondsdeclare @i intset @i = 0while @i<500000begin print(getdate()) if @i%10000=0 begin print(@i) end set @i = @i + 1end so i guess you're not loosing that much Go with the flow & have fun! Else fight the flow |
 |
|
|
rockmoose
SQL Natt Alfen
3279 Posts |
Posted - 2005-02-21 : 15:23:30
|
| PML ?The problem is the 300,000+ INSERTS in aloop. getdate() or no getdate()...If PML could write them to a file instead, then You could BULK INSERT the file.Imagine the performance boost !!!rockmoose |
 |
|
|
eyechart
Master Smack Fu Yak Hacker
3575 Posts |
Posted - 2005-02-21 : 15:46:58
|
| Just reading your post (and obviously not knowing anything about your system) it would seem that a simple DTS package could be used instead of the PML code. Btw, what is PML?you could even schedule the execution of the DTS package using sqlagent. Performance would be magnitudes faster.Keep in mind that if you are depending on having the datetimes be unique, using a bulk insert method (also used by DTS) might cause you more problems. Bulk insert is so fast that you may actually have records inserted that have duplicate timestamps.-ec |
 |
|
|
Xerxes
Aged Yak Warrior
666 Posts |
Posted - 2005-02-21 : 15:56:50
|
Eyechart, That's what I do....use DTS for Timestamping and Userstamping. Semper fi, Xerxes, USMC(Ret.)--------------------------------------------------Once a Marine ALWAYS a Marine! |
 |
|
|
Michael Valentine Jones
Yak DBA Kernel (pronounced Colonel)
7020 Posts |
Posted - 2005-02-21 : 16:35:07
|
Writting the data to a file and using BULK INSERT would be a lot faster.If you can't do that, you might try cutting down the number of inserts by inserting multiple rows on each insert:insert into table_1 (col1,col2,col3,col4)select 1,2,3,4 union allselect 12,2,3,4 union allselect 132,3,4 union allselect 14,2,3,4 union allselect 15,2,3,4 union all...etc... This will decrease the number of transactions if you insert say 100 rows at a time, and reduce the number of trips from your application to the database.quote: Originally posted by Zathras The only way to access data from a certain design database is to write PML code that send the data to my SQL Server. The PML function basically opens a connection, send 300,000+ INSERT statements one-by-one, then executes a SP to tell my SQL Server that it has finished exporting. This SP then moves the data from the one set of tables to the "real" tables so that end users can see the complete data. This process of exporting takes about 8 to 9 hours per project becuase it is done in PML code (no choice).Sometimes that export stalls due to say the exporting computer crashes. To check for this, the SQL Server runs a SP every 20 minutes looking for projects that hasn't written data in over 20 minutes and doesn't have the finish marker. They way I check for data is that every single INSERT has a TimeStamp DateTime field as the very last field and the data is inserted as a GETDATE().My question is, is there any other way I can get the last time data was written for a certain project (connection?) other than having a separate TimeStamp DateTime field. I'm trying to speed up the exports since they take so long. Not having 300,000+ GETDATE() function calls seems to be a good place to start.EDIT: The field is a datetime field (not the TimeStamp datatype)
Codo Ergo Sum |
 |
|
|
Zathras
Starting Member
28 Posts |
Posted - 2005-02-21 : 18:11:58
|
| PML is a text language (like LISP is). It is required becasue it is the only language supported by the propritary database used by Aveva's PDMS. www.aveva.comNo choice in the PML because there is no other way to communicate with the database, it sucks bigtime.Someone guessed correctly that there is a looping mechanism going on. I get the raw data of the item (steel beam, pipe flange, pump, etc) and upload that item, then move on to the next item. It totals to 13 different tables. I'll try the export to a text file then upload to see if that improves the speed. |
 |
|
|
|
|
|
|
|