Please start any new threads on our new site at https://forums.sqlteam.com. We've got lots of great SQL Server experts to answer whatever question you can come up with.

 All Forums
 SQL Server 2005 Forums
 Transact-SQL (2005)
 Capped running totals

Author  Topic 

jimf
Master Smack Fu Yak Hacker

2875 Posts

Posted - 2010-04-12 : 10:43:47
Hi all,
I am working on a getting a running total so as to get the incremental amount necessary to keep that running total under a certain amount - to cap the actual running total. This query produces the correct result, but it seems a bit cumbersome to me. The actual underlying data set could have as many as 30,000 records, so was wondering if somebody has a more elegant solution?

Thanks,

Jim


DECLARE @cap int
SET @cap = 100000

DECLARE @table TABLE (ayq int,cyq int, paid int)
INSERT INTO @table
SELECT 200604, 200604, 2125 UNION ALL
SELECT 200604, 200701, 305 UNION ALL
SELECT 200604, 200702, 0 UNION ALL
SELECT 200604, 200703, 0 UNION ALL
SELECT 200604, 200704, 0 UNION ALL
SELECT 200604, 200802, 0 UNION ALL
SELECT 200604, 200903, 0 UNION ALL
SELECT 200604, 200904, 107000


SELECT t2.ayq,t2.cyq,t2.paid
,t2.totpaid
,[IncrementalPaid] = CASE WHEN @cap - t2.totpaid < 0 THEN @cap - (t2.TotPaid - t2.paid) ELSE t2.paid END
FROM
(
select t1.ayq,t1.cyq, t1.paid
,[TotPaid] = sum(t2.paid)
--,@cap - sum(t2.paid)
from @table t1
inner join
@table t2
on
t1.ayq = t2.ayq
and t1.cyq >= t2.cyq

group by t1.ayq,t1.cyq, t1.paid
) t2

RESULT SET:

ayq cyq paid totpaid IncrementalPaid
200604 200604 2125 2125 2125
200604 200701 305 2430 305
200604 200702 0 2430 0
200604 200703 0 2430 0
200604 200704 0 2430 0
200604 200802 0 2430 0
200604 200903 0 2430 0
200604 200904 107000 109430 97570



Everyday I learn something that somebody else already knew

DBA in the making
Aged Yak Warrior

638 Posts

Posted - 2010-04-12 : 11:38:36
This could be one of the few occasions where using a cursor gives better results. I think a cursor would be quicker, because you're method isn't keeping a single running total, it's, running an aggregate query for every row in the table, which is pretty expensive on long tables. Try this:
DECLARE @ayq INT
DECLARE @cyq INT
DECLARE @paid INT
DECLARE @ayqLast INT --SET @ayqLast = 0
DECLARE @totpaid INT

DECLARE @tableOut TABLE (ayq int,cyq int, paid int,
totpaid INT) --, [IncrementalPaid] INT)


DECLARE cur CURSOR FORWARD_ONLY FOR
SELECT ayq, cyq, paid
FROM @Table
ORDER BY ayq, cyq

OPEN cur
FETCH NEXT FROM cur INTO @ayq, @cyq, @paid
WHILE @@Fetch_Status = 0
BEGIN
IF @ayqLast = @ayq
BEGIN
-- The ayq value is the same as the previous record.
SET @totpaid = @totpaid + @paid
END
ELSE
BEGIN
-- The ayq value has changed.
SET @totpaid = @paid
END

INSERT INTO @tableOut
VALUES (@ayq, @cyq, @paid, @totpaid)

SET @ayqLast = @ayq

FETCH NEXT FROM cur INTO @ayq, @cyq, @paid
END

CLOSE cur
DEALLOCATE cur

SELECT ayq,cyq,paid
,totpaid
,[IncrementalPaid] = CASE WHEN @cap - totpaid < 0 THEN @cap - (TotPaid - paid) ELSE paid END

FROM @tableOut


------------------------------------------------------------------------------------
Any and all code contained within this post comes with a 100% money back guarantee.
Go to Top of Page

waterduck
Aged Yak Warrior

982 Posts

Posted - 2010-04-12 : 11:55:42
;with cte as(
select ROW_NUMBER() over (order by ayq, cyq) num, *
from @table
)
select a.ayq, a.cyq, a.paid, incre, a.paid-isnull(b.paid, 0)
from cte a left join cte b on a.ayq=b.ayq and a.num=b.num+1
cross apply
(select SUM(paid)incre from cte c where c.cyq<=a.cyq and c.ayq=a.ayq)c


Hope can help...but advise to wait pros with confirmation...
Go to Top of Page

DBA in the making
Aged Yak Warrior

638 Posts

Posted - 2010-04-12 : 12:11:15
quote:
Originally posted by waterduck

;with cte as(
select ROW_NUMBER() over (order by ayq, cyq) num, *
from @table
)
select a.ayq, a.cyq, a.paid, incre, a.paid-isnull(b.paid, 0)
from cte a left join cte b on a.ayq=b.ayq and a.num=b.num+1
cross apply
(select SUM(paid)incre from cte c where c.cyq<=a.cyq and c.ayq=a.ayq)c


That doesn't give the same results as the original (In fact it fails to take the cap into account at all), and it still has the same issue as the original, it runs an aggregate query for each row returned.

------------------------------------------------------------------------------------
Any and all code contained within this post comes with a 100% money back guarantee.
Go to Top of Page

waterduck
Aged Yak Warrior

982 Posts

Posted - 2010-04-12 : 12:15:39
so sorry...i was just trying to help


Hope can help...but advise to wait pros with confirmation...
Go to Top of Page

jimf
Master Smack Fu Yak Hacker

2875 Posts

Posted - 2010-04-12 : 12:22:15
Thanks for the input guys. Long time no see waterduck, I was wondering where you've been. Unfortunately your answer doesn't quite give me what I'm looking for.
DBAITM, you may be right, a cursor may be what is needed here. I do need the running total row-by-row, as I need to know when the amount goes over a certain limit (suppied by the user). I also found that my query doesn't work either. If I take a @cap of 2000 it illustrates the point that I also need to take into account that after that cap is reached, every incremental thereafter is 0. I really don't want to introduce another CASE statement, but that may be the best solution. I'll take a few tests with my way vs. cursors and see which one runs best and will post back when I get an answer.

Thanks,

Jim

Everyday I learn something that somebody else already knew
Go to Top of Page

visakh16
Very Important crosS Applying yaK Herder

52326 Posts

Posted - 2010-04-12 : 13:31:20
[code]DECLARE @cap int
SET @cap = 100000

SELECT t.ayq,
t.cyq,
t.paid,
t.paid + COALESCE(t1.paid,0) AS totpaid,
CASE WHEN @cap - (t.paid + COALESCE(t1.paid,0)) < 0 THEN @cap - COALESCE(t1.paid,0) ELSE t.paid END AS IncrementalPaid
FROM YourTable t
OUTER APPLY (SELECT SUM(paid) AS prevpaid
FROM Table
WHERE ayq=t.ayq
AND cyq < t. cyq
)t1
[/code]

------------------------------------------------------------------------------------------------------
SQL Server MVP
http://visakhm.blogspot.com/

Go to Top of Page

madhivanan
Premature Yak Congratulator

22864 Posts

Posted - 2010-04-13 : 03:24:58
Also see if you can make use of Quiky update technique
Refer method 4
http://beyondrelational.com/blogs/madhivanan/archive/2009/06/10/quirky-update-in-sql-server.aspx

Madhivanan

Failing to plan is Planning to fail
Go to Top of Page

DBA in the making
Aged Yak Warrior

638 Posts

Posted - 2010-04-13 : 10:13:43
I did some testing on this. The server I used is running inside a VM on my laptop, so it's not the fastest.

For data, I populates the @Table with 100,000 rows. 10,000 ayq values, each with 10 cyq values, all with random paid values.

I ran a test on the method I posted, which used a cursor (Method 1), against the method posted by visakh16, which used repeated SUMs (Method 2). With the table declared as is, here were the run times.

Method 1:- 7 seconds
Method 2:- 15 minutes 46 seconds


Then I added a clustered index to the table declaration on ayq, cyq, and the results changed dramatically.

Method 1:- 9 seconds
Method 2:- 3 seconds


Then I used a temp table instead of a table variable, and with the same clustered index, I got the same results. However, when I remove the index, I expected Method 2 to slow down again, but it didn't. Here are the result for that:

Method 1:- 7 seconds
Method 2:- 4 seconds


I ran these tests several times, and the results were very consistent. So for method 2, the clustered index had very little effect on the temp table, but a massive effect on the table variable. And for method 1, it decreased performance.

------------------------------------------------------------------------------------
Any and all code contained within this post comes with a 100% money back guarantee.
Go to Top of Page

jimf
Master Smack Fu Yak Hacker

2875 Posts

Posted - 2010-04-13 : 14:19:15
Very interesting. My method also ran surprisingly fast, and with fewer reads. I had to tweak the heck out of it since I didn't consider the situation where the total can fall back under the cap. For payments that's never the case, once the money's spent it's spent, but I also need to apply this to incurred losses -- how much we think we will have to pay -- and this can go back under the cap, for example, we think we have to pay out 150K, bu then the case settles for 75K. Here is my convulution for a dataset of 30,000 records. It runs in 500 ms and and about 500 reads. I put a clustered index on a temp table over policynumber,claimnumber,ayq,cyq.
I'm not convinced it's going to work in all case yet, there always seems to be another quirk in the data to deal with.

I'm still trying to wrap my head around yours and Visakh's solutions. I'm more of a Neanderthal programmer!

Thanks again
Jim


select t2.policynumber,t2.claimnumber
, t2.ayq,t2.cyq
,t2.paid,t2.totpaid,t2.incurred,t2.totincurred
,[IncrementalPaid] =
CASE WHEN
CASE WHEN @cap - t2.totpaid < 0
THEN @cap - (t2.TotPaid - t2.paid)
ELSE t2.paid
END > 0
THEN CASE WHEN @cap - t2.totpaid < 0 THEN @cap - (t2.TotPaid - t2.paid) ELSE t2.paid END
ELSE 0
END
,[IncrementalIncurred] =
CASE WHEN t2.totincurred < @cap and t2.incurred < 0 then t2.incurred else
CASE WHEN -- when you're over the cap
CASE WHEN @cap - t2.totincurred < 0
THEN @cap - (t2.totincurred - t2.incurred)
ELSE t2.incurred
END > 0
THEN
CASE WHEN @cap - t2.totincurred < 0
THEN @cap - (t2.totincurred - t2.incurred)
ELSE t2.incurred
END
ELSE --If you went back under the cap due to take down
CASE
WHEN t2.totincurred < @cap and t2.incurred < 0
THEN t2.totincurred - @cap
ELSE 0
END
END

end

from
(
select t1.policynumber,t1.claimnumber
,t1.ayq,t1.cyq, t1.paid ,t1.incurred--,t1.incexp,t1.expenses
,[TotPaid] = sum(t2.paid)
,[TotIncurred] = sum(t2.incurred)
from #t1 t1
inner join
#t1 t2
on
t1.policynumber = t2.policynumber
and t1.claimnumber = t2.claimnumber
and t1.ayq = t2.ayq
and t1.cyq >= t2.cyq
-- where t1.policynumber = 'aka000002' and t1.claimnumber = 214125
group by t1.policynumber,t1.claimnumber, t1.ayq,t1.cyq, t1.paid,t1.incurred
) t2



order by policynumber,claimnumber,cyq
END


Everyday I learn something that somebody else already knew
Go to Top of Page

Lamprey
Master Smack Fu Yak Hacker

4614 Posts

Posted - 2010-04-13 : 14:25:39
Here is a longer article on the Quirky Update:
http://www.sqlservercentral.com/articles/T-SQL/68467/
Also, be aware that the Quirky Update method not guaranteed to work on future versions of SQL. Although, it probably will.

And here is another one on how to denormalize and constrain your table for running totals:
http://sqlblog.com/blogs/alexander_kuznetsov/archive/2009/01/23/denormalizing-to-enforce-business-rules-running-totals.aspx




Go to Top of Page

SwePeso
Patron Saint of Lost Yaks

30421 Posts

Posted - 2010-04-13 : 14:42:56
Here is another approach very well dissected and explained.
http://www.simple-talk.com/sql/performance/writing-efficient-sql-set-based-speed-phreakery/



N 56°04'39.26"
E 12°55'05.63"
Go to Top of Page
   

- Advertisement -