SQL Server Forums
Profile | Register | Active Topics | Members | Search | Forum FAQ
 
Register Now and get your question answered!
Username:
Password:
Save Password
Forgot your Password?

 All Forums
 SQL Server 2005 Forums
 Other SQL Server Topics (2005)
 Find Missing Data
 New Topic  Reply to Topic
 Printer Friendly
Author Previous Topic Topic Next Topic  

kenoki007
Starting Member

3 Posts

Posted - 05/01/2012 :  20:31:04  Show Profile  Reply with Quote
i have a table with 3 columns Fieldunitserial, Average and timestamp

the data stores something like this

88053 71.6478805541992 2012-05-01 01:40:59.377
88053 70.8151550292969 2012-05-01 01:41:59.510
88053 70.8151550292969 2012-05-01 01:42:59.657
88053 70.8151550292969 2012-05-01 01:43:59.727
88053 70.8151550292969 2012-05-01 01:44:59.940
88053 72.4079208374023 2012-05-01 01:46:00.097
88053 72.1670303344727 2012-05-01 01:47:00.310
88053 73.0599746704102 2012-05-01 01:48:00.420

the data for 01:45:00 is missing in 1 day i got 1440 registers how can i find the data like 1:45:00 with a query, and after find it how can i insert data to fullfill the gaps. thanks in advance

visakh16
Very Important crosS Applying yaK Herder

India
47099 Posts

Posted - 05/01/2012 :  23:18:22  Show Profile  Reply with Quote
is there a master table from which you populate values? or is there a formula based on which values vary?

------------------------------------------------------------------------------------------------------
SQL Server MVP
http://visakhm.blogspot.com/

Go to Top of Page

kenoki007
Starting Member

3 Posts

Posted - 05/01/2012 :  23:35:53  Show Profile  Reply with Quote
the data come from an OPC datalogger, but i will look the gaps from a day before, i mean tomorrow i check the today's data for gaps.
Go to Top of Page

RL
Starting Member

USA
15 Posts

Posted - 05/07/2012 :  00:36:59  Show Profile  Reply with Quote
(NOTE: I'm answering this several days after the original post because it raises an interesting question.)

In the data provided there actually isn't a row missing for 1:45, its timestamp is 1:44:59.727. Log entries prior to 1:46 all have timestamps less than one second before the target time, then it switched to less than one second after. To account for this, the SQL should allow for a "gap" of 1 or more seconds before and after the target time.

Suggested approach:

1. Create a permanent table, daily_minutes, containing one row for each minute in a day (60 * 24 = 1440 rows).
2. Write a query that returns missing rows by comparing log timestamp to daily_minutes (allowing before/after gap).

NOTE: INSERT of missing rows is more complicated, you have to figure out what "average" to use, and deal with things like multiple consecutive missing rows.

-----------------
-- STEP 1: CREATE and load table daily_minutes (only need to do this once)
-----------------
-- DROP table if it already exists
IF EXISTS (SELECT 1 FROM information_schema.tables
            WHERE table_name = 'daily_minutes')
  DROP TABLE daily_minutes;

CREATE TABLE daily_minutes
(minute_num INT, minute_ts TIME);

DECLARE @minute_num INT, @minute_ts TIME, @NTot INT;
 SELECT @minute_num  = 1, 
        @minute_ts = '00:00:00', -- start at 12AM
        @NTot = (60 * 24);       -- 1440 minutes in a day

-- Loop to INSERT a row for each minute
WHILE (@minute_num <= @NTot)
BEGIN
  INSERT INTO daily_minutes SELECT @minute_num, @minute_ts;
  SELECT @minute_num = @minute_num + 1;  -- increment by 1
  SELECT @minute_ts = DATEADD(minute, 1, @minute_ts);  -- increment by 1 minute
END;

-----------------
-- STEP 2: SQL to identify missing rows
-- (NOTE: Log table name = log_table, columns = serial, advance, log_ts)
-----------------
DECLARE 
  @log_day  DATETIME,  -- the day of the log rows
  @gap      INT;       -- allowed gap before/after each minute

SELECT
  @log_day = CONVERT(CHAR(10), getdate()-1, 120), -- yesterday 12AM
  @gap = 1; -- 1 second before or after


-- CTE includes rows for target minutes that DO exist in the log table
-- (DATEADDs allow for the before/after gap)
;WITH CTE AS
(
  SELECT DM.minute_num, DM.minute_ts, LT.serial, LT.average, LT.log_ts 
    FROM daily_minutes DM INNER JOIN log_table LT
         ON LT.[log_ts] BETWEEN DATEADD(s, -@gap, @log_day + DM.minute_ts)
                            AND DATEADD(s, +@gap, @log_day + DM.minute_ts)
)
-- Query returns rows for minutes that are NOT in CTE
SELECT (@log_day + DM.minute_ts) AS missed_target_ts,
       DM.minute_num             AS DM_minute_num,
       DM.minute_ts              AS DM_minute_ts
  FROM daily_minutes DM
 WHERE DM.minute_num 
       BETWEEN (SELECT MIN(minute_num) FROM CTE)
           AND (SELECT MAX(minute_num) FROM CTE)
   AND NOT EXISTS
       (SELECT 1 FROM CTE
         WHERE minute_num = DM.minute_num);

Go to Top of Page
  Previous Topic Topic Next Topic  
 New Topic  Reply to Topic
 Printer Friendly
Jump To:
SQL Server Forums © 2000-2009 SQLTeam Publishing, LLC Go To Top Of Page
This page was generated in 0.06 seconds. Powered By: Snitz Forums 2000