Please start any new threads on our new site at https://forums.sqlteam.com. We've got lots of great SQL Server experts to answer whatever question you can come up with.

 All Forums
 SQL Server 2000 Forums
 Transact-SQL (2000)
 A way to replace timestamping records?

Author  Topic 

Zathras
Starting Member

28 Posts

Posted - 2005-02-21 : 12:09:15
The only way to access data from a certain design database is to write PML code that send the data to my SQL Server. The PML function basically opens a connection, send 300,000+ INSERT statements one-by-one, then executes a SP to tell my SQL Server that it has finished exporting. This SP then moves the data from the one set of tables to the "real" tables so that end users can see the complete data. This process of exporting takes about 8 to 9 hours per project becuase it is done in PML code (no choice).

Sometimes that export stalls due to say the exporting computer crashes. To check for this, the SQL Server runs a SP every 20 minutes looking for projects that hasn't written data in over 20 minutes and doesn't have the finish marker. They way I check for data is that every single INSERT has a TimeStamp DateTime field as the very last field and the data is inserted as a GETDATE().

My question is, is there any other way I can get the last time data was written for a certain project (connection?) other than having a separate TimeStamp DateTime field. I'm trying to speed up the exports since they take so long. Not having 300,000+ GETDATE() function calls seems to be a good place to start.

EDIT: The field is a datetime field (not the TimeStamp datatype)

spirit1
Cybernetic Yak Master

11752 Posts

Posted - 2005-02-21 : 12:28:10
nope don't think there is.
could you change the datetime to maybe integer that increments for every project?

EDIT:
just for the fun of it i ran this on my server which is pentium 4 1.6 GHz and it took 4 minutes 48 seconds

declare @i int
set @i = 0
while @i<500000
begin
print(getdate())
if @i%10000=0
begin
print(@i)
end
set @i = @i + 1
end


so i guess you're not loosing that much

Go with the flow & have fun! Else fight the flow
Go to Top of Page

rockmoose
SQL Natt Alfen

3279 Posts

Posted - 2005-02-21 : 15:23:30
PML ?
The problem is the 300,000+ INSERTS in aloop. getdate() or no getdate()...

If PML could write them to a file instead, then You could BULK INSERT the file.
Imagine the performance boost !!!

rockmoose
Go to Top of Page

eyechart
Master Smack Fu Yak Hacker

3575 Posts

Posted - 2005-02-21 : 15:46:58
Just reading your post (and obviously not knowing anything about your system) it would seem that a simple DTS package could be used instead of the PML code. Btw, what is PML?

you could even schedule the execution of the DTS package using sqlagent. Performance would be magnitudes faster.

Keep in mind that if you are depending on having the datetimes be unique, using a bulk insert method (also used by DTS) might cause you more problems. Bulk insert is so fast that you may actually have records inserted that have duplicate timestamps.


-ec
Go to Top of Page

Xerxes
Aged Yak Warrior

666 Posts

Posted - 2005-02-21 : 15:56:50
Eyechart,

That's what I do....use DTS for Timestamping and Userstamping.

Semper fi, Xerxes, USMC(Ret.)
--------------------------------------------------
Once a Marine ALWAYS a Marine!
Go to Top of Page

Michael Valentine Jones
Yak DBA Kernel (pronounced Colonel)

7020 Posts

Posted - 2005-02-21 : 16:35:07
Writting the data to a file and using BULK INSERT would be a lot faster.

If you can't do that, you might try cutting down the number of inserts by inserting multiple rows on each insert:

insert into table_1 (col1,col2,col3,col4)
select 1,2,3,4 union all
select 12,2,3,4 union all
select 132,3,4 union all
select 14,2,3,4 union all
select 15,2,3,4 union all
...etc...

This will decrease the number of transactions if you insert say 100 rows at a time, and reduce the number of trips from your application to the database.

quote:
Originally posted by Zathras

The only way to access data from a certain design database is to write PML code that send the data to my SQL Server. The PML function basically opens a connection, send 300,000+ INSERT statements one-by-one, then executes a SP to tell my SQL Server that it has finished exporting. This SP then moves the data from the one set of tables to the "real" tables so that end users can see the complete data. This process of exporting takes about 8 to 9 hours per project becuase it is done in PML code (no choice).

Sometimes that export stalls due to say the exporting computer crashes. To check for this, the SQL Server runs a SP every 20 minutes looking for projects that hasn't written data in over 20 minutes and doesn't have the finish marker. They way I check for data is that every single INSERT has a TimeStamp DateTime field as the very last field and the data is inserted as a GETDATE().

My question is, is there any other way I can get the last time data was written for a certain project (connection?) other than having a separate TimeStamp DateTime field. I'm trying to speed up the exports since they take so long. Not having 300,000+ GETDATE() function calls seems to be a good place to start.

EDIT: The field is a datetime field (not the TimeStamp datatype)



Codo Ergo Sum
Go to Top of Page

Zathras
Starting Member

28 Posts

Posted - 2005-02-21 : 18:11:58
PML is a text language (like LISP is). It is required becasue it is the only language supported by the propritary database used by Aveva's PDMS. www.aveva.com

No choice in the PML because there is no other way to communicate with the database, it sucks bigtime.

Someone guessed correctly that there is a looping mechanism going on. I get the raw data of the item (steel beam, pipe flange, pump, etc) and upload that item, then move on to the next item. It totals to 13 different tables.

I'll try the export to a text file then upload to see if that improves the speed.
Go to Top of Page
   

- Advertisement -