| Author |
Topic |
|
Clas
Starting Member
33 Posts |
Posted - 2008-12-04 : 08:45:49
|
| Hi.I want to fill missing value (=0) with previous "not missing" value (>0).Each membergroup first date is the startdate. The value must be in changedate order / membergroupIf startdate have a missing value(0), then the value is missing (=0). After first real value (>0)there should not appear any missing values in the membergroup.Result can be an update on #source or an new table #result.There are about 250.000 rows , 11.000 groups and 20 valuetypes.I have SQL-server 2008.create table #source(rowid int,changedate datetime,membergroup int,valueA int,valueB int,valueC int)INSERT INTO #source (rowid,changedate, membergroup, valueA,valueB,valueC) SELECT 1, 1945-01-01, 1, 1, 1,0 UNION ALL SELECT 11,1946-02-05, 1, 2, 1, 3 UNION ALL SELECT 32,1946-03-15, 1,0,0, 2 UNION ALL SELECT 9, 1946-12-12, 1,0, 1, 2 UNION ALL SELECT 8, 1947-05-04, 1, 1,0, 1 UNION ALL SELECT 12,1947-07-05, 1, 2,0, 0 UNION ALL SELECT 22,1945-01-13, 2,0, 2, 2 UNION ALL SELECT 21,1945-12-15, 2, 2,0,0 UNION ALL SELECT 34,1945-05-07, 2, 3, 1,0 SELECT * FROM #source ORDER BY membergroup, changedateValueA:membergroup 10 - startvalue --> result value = row value = 00 - value = 0 --> result value = previous result value = 02 - value > 0 --> result value = row value = 21 - value > 0 --> result value = row value = 12 - value > 0 --> result value = row value = 2membergroup 20 - startvalue --> result value = row value = 03 - value > 0 --> result value = row value = 3ValueBmembergroup 11 - startvalue --> result value = row value = 10 - value = 0 --> result value = previous result value = 10 - value = 0 --> result value = previous result value = 10 - value = 0 --> result value = previous result value = 11 - value > 0 --> result value = row value = 1 1 - value > 0 --> result value = row value, = 1 0 - value = 0 --> result value = previous result value = 1membergroup 22 - startvalue --> result value = row value = 21 - value > 0 --> result value = row value = 1 Thanks and RegardsClas |
|
|
jholovacs
Posting Yak Master
163 Posts |
Posted - 2008-12-04 : 08:48:32
|
I don't think you can do this in a set-based operation. Possibly a loop? SELECT TOP 1 w.[name]FROM dbo.women wINNER JOIN dbo.inlaws i ON i.inlaw_id = w.parent_idWHERE i.net_worth > 10000000 AND i.status IN ('dead', 'dying') AND w.husband_id IS NULLORDER BY w.hotness_factor DESC |
 |
|
|
SwePeso
Patron Saint of Lost Yaks
30421 Posts |
Posted - 2008-12-04 : 08:51:49
|
You explanation doesn't make sense and doesn't match provided sample data.Please post your final expected output.DECLARE @Sample TABLE ( rowid int, changedate datetime, membergroup int, valueA int, valueB int, valueC int )INSERT @SampleSELECT 1, '1945-01-01', 1, 1, 1, 0 UNION ALLSELECT 11, '1946-02-05', 1, 2, 1, 3 UNION ALLSELECT 32, '1946-03-15', 1, 0, 0, 2 UNION ALLSELECT 9, '1946-12-12', 1, 0, 1, 2 UNION ALLSELECT 8, '1947-05-04', 1, 1, 0, 1 UNION ALLSELECT 12, '1947-07-05', 1, 2, 0, 0 UNION ALLSELECT 22, '1945-01-13', 2, 0, 2, 2 UNION ALLSELECT 21, '1945-12-15', 2, 2, 0, 0 UNION ALLSELECT 34, '1945-05-07', 2, 3, 1, 0SELECT *FROM @SampleORDER BY membergroup, changedate E 12°55'05.63"N 56°04'39.26" |
 |
|
|
SwePeso
Patron Saint of Lost Yaks
30421 Posts |
Posted - 2008-12-04 : 08:53:18
|
Sample data is this according to the ORDER BYrowid changedate membergroup valueA valueB valueC1 1945-01-01 1 1 1 011 1946-02-05 1 2 1 332 1946-03-15 1 0 0 29 1946-12-12 1 0 1 28 1947-05-04 1 1 0 112 1947-07-05 1 2 0 022 1945-01-13 2 0 2 234 1945-05-07 2 3 1 021 1945-12-15 2 2 0 0 Is your intention to return this result?rowid changedate membergroup valueA valueB valueC1 1945-01-01 1 1 1 011 1946-02-05 1 2 1 332 1946-03-15 1 02 01 29 1946-12-12 1 02 1 28 1947-05-04 1 1 01 112 1947-07-05 1 2 01 0122 1945-01-13 2 0 2 234 1945-05-07 2 3 1 0221 1945-12-15 2 2 01 02 E 12°55'05.63"N 56°04'39.26" |
 |
|
|
visakh16
Very Important crosS Applying yaK Herder
52326 Posts |
Posted - 2008-12-04 : 08:58:55
|
| [code]UPDATE sSET s.valueA=COALESCE(a.valueA,s.valueA),s.valueB=COALESCE(b.valueB,s.valueB),s.valueC=COALESCE(c.valueC,s.valueC)FROM #source sOUTER APPLY (SELECT TOP 1 valueA FROM #source WHERE membergroup=s.membergroup AND changedate<s.changedate AND valueA >0 ORDER BY changedate DESC) aOUTER APPLY (SELECT TOP 1 valueB FROM #source WHERE membergroup=s.membergroup AND changedate<s.changedate AND valueB >0 ORDER BY changedate DESC) bOUTER APPLY (SELECT TOP 1 valueC FROM #source WHERE membergroup=s.membergroup AND changedate<s.changedate AND valueC >0 ORDER BY changedate DESC) c[/code] |
 |
|
|
Clas
Starting Member
33 Posts |
Posted - 2008-12-04 : 09:02:04
|
| INPUT9 1905-04-07 00:00:00.000 1 0 1 232 1905-04-13 00:00:00.000 1 0 0 212 1905-04-20 00:00:00.000 1 2 0 08 1905-04-23 00:00:00.000 1 1 0 111 1905-04-24 00:00:00.000 1 2 1 31 1905-04-28 00:00:00.000 1 1 1 021 1905-04-03 00:00:00.000 2 2 0 022 1905-04-16 00:00:00.000 2 0 2 234 1905-04-18 00:00:00.000 2 3 1 0RESULT9 1905-04-07 00:00:00.000 1 0 1 232 1905-04-13 00:00:00.000 1 0 1 212 1905-04-20 00:00:00.000 1 2 1 28 1905-04-23 00:00:00.000 1 1 1 111 1905-04-24 00:00:00.000 1 2 1 31 1905-04-28 00:00:00.000 1 1 1 321 1905-04-03 00:00:00.000 2 2 0 022 1905-04-16 00:00:00.000 2 2 2 234 1905-04-18 00:00:00.000 2 3 1 2 |
 |
|
|
visakh16
Very Important crosS Applying yaK Herder
52326 Posts |
Posted - 2008-12-04 : 09:04:35
|
quote: Originally posted by Clas INPUT9 1905-04-07 00:00:00.000 1 0 1 232 1905-04-13 00:00:00.000 1 0 0 212 1905-04-20 00:00:00.000 1 2 0 08 1905-04-23 00:00:00.000 1 1 0 111 1905-04-24 00:00:00.000 1 2 1 31 1905-04-28 00:00:00.000 1 1 1 021 1905-04-03 00:00:00.000 2 2 0 022 1905-04-16 00:00:00.000 2 0 2 234 1905-04-18 00:00:00.000 2 3 1 0RESULT9 1905-04-07 00:00:00.000 1 0 1 232 1905-04-13 00:00:00.000 1 0 1 212 1905-04-20 00:00:00.000 1 2 1 28 1905-04-23 00:00:00.000 1 1 1 111 1905-04-24 00:00:00.000 1 2 1 31 1905-04-28 00:00:00.000 1 1 1 321 1905-04-03 00:00:00.000 2 2 0 022 1905-04-16 00:00:00.000 2 2 2 234 1905-04-18 00:00:00.000 2 3 1 2
i think my suggestion will work for you seeing sample data |
 |
|
|
SwePeso
Patron Saint of Lost Yaks
30421 Posts |
Posted - 2008-12-04 : 09:11:44
|
Visakh, I wouldn't run that on 2.5 lakh records and 11000 groups.Try thisDECLARE @Sample TABLE ( rowid int, changedate datetime, membergroup int, valueA int, valueB int, valueC int, PRIMARY KEY CLUSTERED ( membergroup, changedate ) )INSERT @SampleSELECT 1, '1945-01-01', 1, 1, 1, 0 UNION ALLSELECT 11, '1946-02-05', 1, 2, 1, 3 UNION ALLSELECT 32, '1946-03-15', 1, 0, 0, 2 UNION ALLSELECT 9, '1946-12-12', 1, 0, 1, 2 UNION ALLSELECT 8, '1947-05-04', 1, 1, 0, 1 UNION ALLSELECT 12, '1947-07-05', 1, 2, 0, 0 UNION ALLSELECT 22, '1945-01-13', 2, 0, 2, 2 UNION ALLSELECT 21, '1945-12-15', 2, 2, 0, 0 UNION ALLSELECT 34, '1945-05-07', 2, 3, 1, 0-- OriginalSELECT *FROM @SampleORDER BY membergroup, changedate -- Prepare lightning fast updateDECLARE @membergroup int, @a INT, @b int, @c intSELECT top 1 @a = valuea, @b = valueb, @c = valuec, @membergroup = membergroupfrom @sampleORDER BY membergroup, changedateupdate @sampleset valuea = COALESCE(NULLIF(valuea, 0), @a), valueb = COALESCE(NULLIF(valueb, 0), @b), valuec = COALESCE(NULLIF(valuec, 0), @c), @a = case when membergroup = @membergroup then coalesce(nullif(valuea, 0), @a) else valuea end, @b = case when membergroup = @membergroup then coalesce(nullif(valueb, 0), @b) else valueb end, @c = case when membergroup = @membergroup then coalesce(nullif(valuec, 0), @c) else valuec end, @membergroup = membergroup-- AfterSELECT *FROM @SampleORDER BY membergroup, changedate The trick here is that you must have a clustered index over membergroup and changedate. E 12°55'05.63"N 56°04'39.26" |
 |
|
|
Clas
Starting Member
33 Posts |
Posted - 2008-12-04 : 09:43:16
|
| It is close...INPUT ValueC9 1905-04-07 00:00:00.000 232 1905-04-13 00:00:00.000 212 1905-04-20 00:00:00.000 08 1905-04-23 00:00:00.000 111 1905-04-24 00:00:00.000 31 1905-04-28 00:00:00.000 0OUTPUT9 1905-04-07 00:00:00.000 232 1905-04-13 00:00:00.000 212 1905-04-20 00:00:00.000 28 1905-04-23 00:00:00.000 211 1905-04-24 00:00:00.000 11 1905-04-28 00:00:00.000 3Row 12 valueC INPUT 0 --> 2 is OKRow 8 valueC INPUT 1 --> 2 , but it should have the input value 1. INPUT valueC > 0 --> result value = INPUT row valueResult valueC:9 1905-04-07 232 1905-04-13 212 1905-04-20 28 1905-04-23 111 1905-04-24 31 1905-04-28 3 |
 |
|
|
SwePeso
Patron Saint of Lost Yaks
30421 Posts |
Posted - 2008-12-04 : 09:46:04
|
For which suggestion? Mine or Visakh's? E 12°55'05.63"N 56°04'39.26" |
 |
|
|
Clas
Starting Member
33 Posts |
Posted - 2008-12-04 : 09:51:49
|
| My previos reply was to Visakh.I think it work with last solution from Peso.Give me 5 minutes to try ..... |
 |
|
|
visakh16
Very Important crosS Applying yaK Herder
52326 Posts |
Posted - 2008-12-04 : 09:57:59
|
quote: Originally posted by Clas My previos reply was to Visakh.I think it work with last solution from Peso.Give me 5 minutes to try .....
yup...i see that.......i had update otherway around. it should have beenUPDATE sSET s.valueA=COALESCE(NULLIF(s.valueA,0),a.valueA,0),s.valueB=COALESCE(NULLIF(s.valueB,0),b.valueB,0),s.valueC=COALESCE(NULLIF(s.valueC,0),c.valueC,0)FROM #source sOUTER APPLY (SELECT TOP 1 valueA FROM #source WHERE membergroup=s.membergroup AND changedate<s.changedate AND valueA >0 ORDER BY changedate DESC) aOUTER APPLY (SELECT TOP 1 valueB FROM #source WHERE membergroup=s.membergroup AND changedate<s.changedate AND valueB >0 ORDER BY changedate DESC) bOUTER APPLY (SELECT TOP 1 valueC FROM #source WHERE membergroup=s.membergroup AND changedate<s.changedate AND valueC >0 ORDER BY changedate DESC) c |
 |
|
|
jholovacs
Posting Yak Master
163 Posts |
Posted - 2008-12-04 : 10:02:25
|
Hmm... I've never played with APPLY and now I'm wondering if I'm missing something important. Performance-wise, how would this query fare against a table variable-based loop? SELECT TOP 1 w.[name]FROM dbo.women wINNER JOIN dbo.inlaws i ON i.inlaw_id = w.parent_idWHERE i.net_worth > 10000000 AND i.status IN ('dead', 'dying') AND w.husband_id IS NULLORDER BY w.hotness_factor DESC |
 |
|
|
SwePeso
Patron Saint of Lost Yaks
30421 Posts |
Posted - 2008-12-04 : 10:06:20
|
Clas, timetest both suggestions and please report back here with times. E 12°55'05.63"N 56°04'39.26" |
 |
|
|
Clas
Starting Member
33 Posts |
Posted - 2008-12-04 : 10:35:22
|
| Peso, i have test with 24000 rows and 7600 groups, time: 00:00:01 on a desktop computer with sql-server.Visakh, i think it works fine, it was the same time, 00:00:01.I need more rows and group to timetest.It is evening, so I am coming back tomorrow...... |
 |
|
|
visakh16
Very Important crosS Applying yaK Herder
52326 Posts |
Posted - 2008-12-04 : 11:04:23
|
quote: Originally posted by Clas Peso, i have test with 24000 rows and 7600 groups, time: 00:00:01 on a desktop computer with sql-server.Visakh, i think it works fine, it was the same time, 00:00:01.I need more rows and group to timetest.It is evening, so I am coming back tomorrow......
Ok..ThanksLet us know how tests go |
 |
|
|
Clas
Starting Member
33 Posts |
Posted - 2008-12-05 : 06:46:35
|
| 133148 rows, 7695 groups:SQL without Clustred index;Number of Values: 3 Time:00:00:00:03Number of Values: 8 Time:00:00:00:10Number of Values:16 Time:00:00:00:21SQL with Clustred index;Number of Values: 3 Time:00:00:00:02Number of Values: 8 Time:00:00:00:03Number of Values:16 Time:00:00:00:03Thanks for all help !!--------------------UPDATE sSET s.valueA=COALESCE(NULLIF(s.valueA,0),a.valueA,0),s.valueB=COALESCE(NULLIF(s.valueB,0),b.valueB,0),s.valueC=COALESCE(NULLIF(s.valueC,0),c.valueC,0),s.valueD=COALESCE(NULLIF(s.valueD,0),d.valueD,0),s.valueE=COALESCE(NULLIF(s.valueE,0),e.valueE,0),s.valueF=COALESCE(NULLIF(s.valueF,0),f.valueF,0),s.valueG=COALESCE(NULLIF(s.valueG,0),g.valueG,0),s.valueH=COALESCE(NULLIF(s.valueH,0),h.valueH,0),s.valueI=COALESCE(NULLIF(s.valueI,0),i.valueI,0),s.valueJ=COALESCE(NULLIF(s.valueJ,0),j.valueJ,0),s.valueK=COALESCE(NULLIF(s.valueK,0),k.valueK,0),s.valueL=COALESCE(NULLIF(s.valueL,0),l.valueL,0),s.valueM=COALESCE(NULLIF(s.valueM,0),m.valueM,0),s.valueN=COALESCE(NULLIF(s.valueN,0),n.valueN,0),s.valueO=COALESCE(NULLIF(s.valueO,0),o.valueO,0),s.valueP=COALESCE(NULLIF(s.valueP,0),p.valueP,0)FROM #Sample sOUTER APPLY (SELECT TOP 1 valueA FROM #Sample WHERE membergroup=s.membergroup AND changedate<s.changedate AND valueA >0 ORDER BY changedate DESC) aOUTER APPLY (SELECT TOP 1 valueB FROM #Sample WHERE membergroup=s.membergroup AND changedate<s.changedate AND valueB >0 ORDER BY changedate DESC) bOUTER APPLY (SELECT TOP 1 valueC FROM #Sample WHERE membergroup=s.membergroup AND changedate<s.changedate AND valueC >0 ORDER BY changedate DESC) cOUTER APPLY (SELECT TOP 1 valueD FROM #Sample WHERE membergroup=s.membergroup AND changedate<s.changedate AND valueD >0 ORDER BY changedate DESC) d OUTER APPLY (SELECT TOP 1 valueE FROM #Sample WHERE membergroup=s.membergroup AND changedate<s.changedate AND valueE >0 ORDER BY changedate DESC) e OUTER APPLY (SELECT TOP 1 valueF FROM #Sample WHERE membergroup=s.membergroup AND changedate<s.changedate AND valueF >0 ORDER BY changedate DESC) f OUTER APPLY (SELECT TOP 1 valueG FROM #Sample WHERE membergroup=s.membergroup AND changedate<s.changedate AND valueG >0 ORDER BY changedate DESC) g OUTER APPLY (SELECT TOP 1 valueH FROM #Sample WHERE membergroup=s.membergroup AND changedate<s.changedate AND valueH >0 ORDER BY changedate DESC) h OUTER APPLY (SELECT TOP 1 valueI FROM #Sample WHERE membergroup=s.membergroup AND changedate<s.changedate AND valueI >0 ORDER BY changedate DESC) i OUTER APPLY (SELECT TOP 1 valueJ FROM #Sample WHERE membergroup=s.membergroup AND changedate<s.changedate AND valueJ >0 ORDER BY changedate DESC) j OUTER APPLY (SELECT TOP 1 valueK FROM #Sample WHERE membergroup=s.membergroup AND changedate<s.changedate AND valueK >0 ORDER BY changedate DESC) k OUTER APPLY (SELECT TOP 1 valueL FROM #Sample WHERE membergroup=s.membergroup AND changedate<s.changedate AND valueL >0 ORDER BY changedate DESC) l OUTER APPLY (SELECT TOP 1 valueM FROM #Sample WHERE membergroup=s.membergroup AND changedate<s.changedate AND valueM >0 ORDER BY changedate DESC) m OUTER APPLY (SELECT TOP 1 valueN FROM #Sample WHERE membergroup=s.membergroup AND changedate<s.changedate AND valueN >0 ORDER BY changedate DESC) n OUTER APPLY (SELECT TOP 1 valueO FROM #Sample WHERE membergroup=s.membergroup AND changedate<s.changedate AND valueO >0 ORDER BY changedate DESC) o OUTER APPLY (SELECT TOP 1 valueP FROM #Sample WHERE membergroup=s.membergroup AND changedate<s.changedate AND valueP >0 ORDER BY changedate DESC) p ---------------------------------------------DECLARE @membergroup int, @a INT, @b int, @c decimal, @d int, @e int, @f int, @g int, @h int, @i int, @j int, @k int, @l int, @m int, @n int, @o int, @p intSELECT top 1 @a = valuea, @b = valueb, @c = valuec, @d = valued, @e = valuee, @f = valuef, @g = valueg, @h = valueh, @i = valuei, @j = valuej, @k = valuek, @l = valuel, @m = valuem, @n = valuen, @o = valueo, @p = valuep, @membergroup = membergroupfrom #SampleORDER BY membergroup, changedateupdate #Sampleset valuea = COALESCE(NULLIF(valuea, 0), @a), valueb = COALESCE(NULLIF(valueb, 0), @b), valuec = COALESCE(NULLIF(valuec, 0), @c), valued = COALESCE(NULLIF(valued, 0), @d), valuee = COALESCE(NULLIF(valuee, 0), @e), valuef = COALESCE(NULLIF(valuef, 0), @f), valueg = COALESCE(NULLIF(valueg, 0), @g), valueh = COALESCE(NULLIF(valueh, 0), @h), valuei = COALESCE(NULLIF(valuei, 0), @i), valuej = COALESCE(NULLIF(valuej, 0), @j), valuek = COALESCE(NULLIF(valuek, 0), @k), valuel = COALESCE(NULLIF(valuel, 0), @l), valuem = COALESCE(NULLIF(valuem, 0), @m), valuen = COALESCE(NULLIF(valuen, 0), @n), valueo = COALESCE(NULLIF(valueo, 0), @o), valuep = COALESCE(NULLIF(valuep, 0), @p), @a = case when membergroup = @membergroup then coalesce(nullif(valuea, 0), @a) else valuea end, @b = case when membergroup = @membergroup then coalesce(nullif(valueb, 0), @b) else valueb end, @c = case when membergroup = @membergroup then coalesce(nullif(valuec, 0), @c) else valuec end, @d = case when membergroup = @membergroup then coalesce(nullif(valued, 0), @d) else valued end, @e = case when membergroup = @membergroup then coalesce(nullif(valuee, 0), @e) else valuee end, @f = case when membergroup = @membergroup then coalesce(nullif(valuef, 0), @f) else valuef end, @g = case when membergroup = @membergroup then coalesce(nullif(valueg, 0), @g) else valueg end, @h = case when membergroup = @membergroup then coalesce(nullif(valueh, 0), @h) else valueh end, @i = case when membergroup = @membergroup then coalesce(nullif(valuei, 0), @i) else valuei end, @j = case when membergroup = @membergroup then coalesce(nullif(valuej, 0), @j) else valuej end, @k = case when membergroup = @membergroup then coalesce(nullif(valuek, 0), @k) else valuek end, @l = case when membergroup = @membergroup then coalesce(nullif(valuel, 0), @l) else valuel end, @m = case when membergroup = @membergroup then coalesce(nullif(valuem, 0), @m) else valuem end, @n = case when membergroup = @membergroup then coalesce(nullif(valuen, 0), @n) else valuen end, @o = case when membergroup = @membergroup then coalesce(nullif(valueo, 0), @o) else valueo end, @p = case when membergroup = @membergroup then coalesce(nullif(valuep, 0), @p) else valuep end, @membergroup = membergroup |
 |
|
|
SwePeso
Patron Saint of Lost Yaks
30421 Posts |
Posted - 2008-12-05 : 06:54:33
|
Try with your original 250,000 records too. E 12°55'05.63"N 56°04'39.26" |
 |
|
|
Clas
Starting Member
33 Posts |
Posted - 2008-12-05 : 09:10:44
|
| Both sql use the same source, #Sample (with PRIMARY KEY CLUSTERED)Now I have a join to individuals when I insert rows in #Sample.Membergroup = IndividualsRecords: 515 332V; 00:00:28P; 00:00:07Records: 6 034 476 Groups (individuals): 45045V; 00:06:28P; 00:03:38 #Sample without PRIMARY KEY CLUSTEREDV; First and second run, I cancelled query after 15 minutesP; First run 00:01:26, Second run 00:01:25There is a very big diffrence if I use primary key with Visakh16 sql.(Combination with OUTER APPLY ?)In Peso sql it was faster WITHOUT Primary Key ?????? |
 |
|
|
SwePeso
Patron Saint of Lost Yaks
30421 Posts |
Posted - 2008-12-05 : 09:12:38
|
My code will NOT work correctly if the CLUSTERED INDEX is missing. E 12°55'05.63"N 56°04'39.26" |
 |
|
|
Clas
Starting Member
33 Posts |
Posted - 2008-12-05 : 09:17:24
|
| I am going to use CLUSTERED INDEX......:) |
 |
|
|
Next Page
|
|
|