| Author |
Topic |
|
fan2005
Yak Posting Veteran
85 Posts |
Posted - 2011-01-28 : 13:48:08
|
Is there any Faster solution fro this query when there is 70milion records?SELECTdistincttransactionID = identity(int, 1,1), C.abrnchcod,C.Cfcifno, C.tarikh,C.account,b.row as AccountID, C.Tbdptype , s.gardesh into TransactionDetail FROM dbo.Gardesh Cleft outer JOIN (select C.account, C.tarikh,SUM (amount )as gardesh from dbo.Gardesh C GROUP BY C.tarikh, C.account )S ON C.account = S.account and C.tarikh = S.tarikhleft outer join dbo.Allacounts b on C.account =b.account order by C.account,C.tarikh ------------table Gardesh-----id amount tarikh account abrnchcod cfcifno Tbdptype1 50000.00 2010/01/05 1-11-1 1 11 1 2 -3000.00 2010/02/05 1-11-1 1 11 1 3 1000.00 2010/09/09 1-11-1 1 11 1 4 65000.00 2010/05/01 1-11-2 1 11 2 5 6000.00 2010/09/08 1-12-2 1 11 2 6 100000.00 2010/04/12 2-12-1 2 12 1 7 -1000.00 2010/09/21 2-12-1 2 12 1 64 1000.00 2010/01/05 1-11-1 1 11 1 65 2000.00 2010/12/06 1-11-1 1 11 1 66 1000.00 2010/12/23 1-11-1 1 11 1 --------AllAccountsrow account101 1-11-1 102 1-11-2 103 1-12-2 104 2-12-1 -----------------------------------------------After this select into I Have a runing total part uses cursoralter table TransactionDetail add balance decimal(20,0)alter table TransactionDetail add preBalance decimal(20,0) CREATE CLUSTERED INDEX IXC_Transaction_AccountID_Tarikh_TransactionID ON dbo.TransactionDetail (AccountID, tarikh, TransactionID) SET NOCOUNT ONDECLARE @Amount decimal(20,0)DECLARE @CurAccountID bigINTDECLARE @PrevAccountID bigINTDECLARE @AccountRunningTotal decimal(20,0)DECLARE @PreAccountRunningTotal decimal(20,0)DECLARE curRunningTotal CURSOR LOCAL FORWARD_ONLY FOR SELECT AccountID, gardesh FROM dbo.TransactionDetail ORDER BY AccountID, tarikh, TransactionID OPEN curRunningTotal FETCH NEXT FROM curRunningTotal INTO @CurAccountID, @Amount WHILE @@FETCH_STATUS = 0 BEGIN SELECT @AccountRunningTotal = CASE WHEN @CurAccountID = @PrevAccountID THEN @AccountRunningTotal + @Amount ELSE @Amount END, @PreAccountRunningTotal = CASE WHEN @CurAccountID = @PrevAccountID THEN @AccountRunningTotal - @Amount ELSE 0 END, @PrevAccountID = @CurAccountID UPDATE dbo.TransactionDetail SET balance = @AccountRunningTotal, Prebalance = @PreAccountRunningTotal WHERE CURRENT OF curRunningTotal FETCH NEXT FROM curRunningTotal INTO @CurAccountID, @Amount END Its slow in huge recordsthanks for any faster solution |
|
|
robvolk
Most Valuable Yak
15732 Posts |
Posted - 2011-01-28 : 14:24:09
|
| Why are you storing running totals in a table? |
 |
|
|
fan2005
Yak Posting Veteran
85 Posts |
Posted - 2011-01-28 : 14:42:59
|
quote: Originally posted by robvolk Why are you storing running totals in a table?
it should be in temptable i need to store it for further reports |
 |
|
|
robvolk
Most Valuable Yak
15732 Posts |
Posted - 2011-01-28 : 14:47:42
|
| What are you using as your reporting layer? Reporting Services? Crystal Reports? Something else?Do you really need to report on all 70 million rows? How often is your summary table generated? (once a day, once a week, every time the report runs, etc.)The key point in improving performance here is reducing the amount of work and data needed. |
 |
|
|
fan2005
Yak Posting Veteran
85 Posts |
Posted - 2011-01-28 : 14:53:32
|
| different users work on this server who need to work with balanceso whenever we want this info we insert new datathen for example we want the customers whith average balance over xin y months we select the part of data we wantwhat do you think is better to do?is it ok, every one needs this table run a SP to build it?is it Ok with query itself? |
 |
|
|
robvolk
Most Valuable Yak
15732 Posts |
Posted - 2011-01-28 : 15:08:15
|
| I hate to say this but you didn't answer any of my questions. Please read them again and respond with specifics. As far as what is better to do, I can't say without a lot more information. But I can say what NOT to do:- Don't summarize 70 million rows every time you need to generate a report.- If the data is historical and rarely changes, don't recreate the same summaries on it. That means don't create a new temp table each time. Summarize it once and store it permanently.- If this is intended as an ad-hoc reporting tool, don't use a summary table at all. Your users are not going to need all this data, so if they already have a specific customer, date range, or other criteria that limits the data they need, query the original table(s) with those criteria and summarize as needed.- Don't do running totals in SQL Server, use a reporting tool to do it. The same applies to the other totals/sums. |
 |
|
|
fan2005
Yak Posting Veteran
85 Posts |
Posted - 2011-01-28 : 15:20:51
|
quote: Originally posted by robvolk I hate to say this but you didn't answer any of my questions. Please read them again and respond with specifics. As far as what is better to do, I can't say without a lot more information. But I can say what NOT to do:- Don't summarize 70 million rows every time you need to generate a report.- If the data is historical and rarely changes, don't recreate the same summaries on it. That means don't create a new temp table each time. Summarize it once and store it permanently.- If this is intended as an ad-hoc reporting tool, don't use a summary table at all. Your users are not going to need all this data, so if they already have a specific customer, date range, or other criteria that limits the data they need, query the original table(s) with those criteria and summarize as needed.- Don't do running totals in SQL Server, use a reporting tool to do it. The same applies to the other totals/sums.
there is no reporting toolsall we have is sql serverthis is a bank account so data is changing all times , but we dont have data online because it is on a oracle server that we can just select (we use the results of views )so we bring data to sql every time we need .and yes i need all data when i make a report . the origenal data has no balance it has only incoming and outgoing transactions.I basically am looking for an added calculated column which sums all the previous transactions plus the current one to give me a running total. |
 |
|
|
robvolk
Most Valuable Yak
15732 Posts |
Posted - 2011-01-28 : 15:59:59
|
For the running total, get rid of the cursor entirely, replace it with this:DECLARE @PrevAccountID bigINT = 0DECLARE @AccountRunningTotal decimal(20,0) = 0UPDATE dbo.TransactionDetail SET @AccountRunningTotal = balance = CASE WHEN AccountID = @PrevAccountID THEN @AccountRunningTotal + gardesh ELSE gardesh END, @PrevAccountID=AccountID This technique relies on non-standard behavior of the UPDATE statement, it's documented in Books Online. The UPDATE processes the data in the physical order of the table, since you've created a clustered index in that same order it should work correctly. Be advised this behavior may not be consistent on all SQL Server versions or may change in a future release. There's an article with another example here: http://www.sqlteam.com/article/creating-a-sequential-record-number-fieldI also suggest reading up on the OVER clause of the aggreate functions, in your case SUM(). You can probably rewrite the original query using OVER to get the AccountID, tarikh sums without joining to a subquery, it should run faster.Regarding the preBalance column, make it a computed column instead:alter table TransactionDetail add preBalance AS Balance - gardesh No point in storing that value when it can be calculated on the fly. |
 |
|
|
fan2005
Yak Posting Veteran
85 Posts |
Posted - 2011-01-28 : 16:13:52
|
| smartthank you very much |
 |
|
|
fan2005
Yak Posting Veteran
85 Posts |
Posted - 2011-01-28 : 16:33:47
|
OVER clause of the aggreate sum()select distinct s.account ,row as accountID,s.tarikh ,s.gardeshfrom (SELECT distinct b.account,b.amount, b.tarikh , sum(b.amount)OVER (PARTITION BY b.account,b.tarikh ) as gardesh FROM gardesh b)s inner join allacounts a on a.account=s.account order by s.account ,s.tarikh account accountID tarikh gardesh1-11-1 101 2010/01/05 51000.001-11-1 101 2010/02/05 -3000.001-11-1 101 2010/09/09 1000.001-11-1 101 2010/12/06 2000.001-11-1 101 2010/12/23 1000.001-11-2 102 2010/05/01 65000.001-12-2 103 2010/09/08 6000.002-12-1 104 2010/04/12 100000.002-12-1 104 2010/09/21 -1000.00 |
 |
|
|
robvolk
Most Valuable Yak
15732 Posts |
Posted - 2011-01-28 : 16:46:03
|
| What constitutes a duplicate? The totals match what you originally posted.If OVER doesn't give you the results you want then use the original query. |
 |
|
|
fan2005
Yak Posting Veteran
85 Posts |
Posted - 2011-01-28 : 16:52:35
|
| another question does it work in sql server 2000 too? |
 |
|
|
fan2005
Yak Posting Veteran
85 Posts |
Posted - 2011-01-28 : 16:54:33
|
quote: Originally posted by robvolk What constitutes a duplicate? The totals match what you originally posted.If OVER doesn't give you the results you want then use the original query.
I made a mistake then I edited the post no duplication .that's greate |
 |
|
|
robvolk
Most Valuable Yak
15732 Posts |
Posted - 2011-01-28 : 17:03:40
|
| No, the OVER clause was added in SQL Server 2005, you'll have to use the original query for SQL Server 2000. |
 |
|
|
|