Please start any new threads on our new site at https://forums.sqlteam.com. We've got lots of great SQL Server experts to answer whatever question you can come up with.

 All Forums
 SQL Server 2008 Forums
 Transact-SQL (2008)
 FASTER SOLUTION?

Author  Topic 

fan2005
Yak Posting Veteran

85 Posts

Posted - 2011-01-28 : 13:48:08
Is there any Faster solution fro this query when there is 70milion records?


SELECT
distinct
transactionID = identity(int, 1,1),
C.abrnchcod,C.Cfcifno, C.tarikh,C.account,b.row as AccountID,
C.Tbdptype , s.gardesh
into TransactionDetail
FROM
dbo.Gardesh C
left outer JOIN
(select C.account, C.tarikh,SUM (amount )as gardesh
from
dbo.Gardesh C

GROUP BY
C.tarikh,
C.account
)S
ON C.account = S.account
and C.tarikh = S.tarikh
left outer join

dbo.Allacounts b
on C.account =b.account

order by C.account,C.tarikh


------------table Gardesh-----
id	amount	tarikh	account	abrnchcod	cfcifno	Tbdptype
1 50000.00 2010/01/05 1-11-1 1 11 1
2 -3000.00 2010/02/05 1-11-1 1 11 1
3 1000.00 2010/09/09 1-11-1 1 11 1
4 65000.00 2010/05/01 1-11-2 1 11 2
5 6000.00 2010/09/08 1-12-2 1 11 2
6 100000.00 2010/04/12 2-12-1 2 12 1
7 -1000.00 2010/09/21 2-12-1 2 12 1
64 1000.00 2010/01/05 1-11-1 1 11 1
65 2000.00 2010/12/06 1-11-1 1 11 1
66 1000.00 2010/12/23 1-11-1 1 11 1


--------AllAccounts
row	account
101 1-11-1
102 1-11-2
103 1-12-2
104 2-12-1



-----------------------------------------------
After this select into I Have a runing total part uses cursor

alter table TransactionDetail add balance decimal(20,0)
alter table TransactionDetail add preBalance decimal(20,0)

CREATE CLUSTERED INDEX IXC_Transaction_AccountID_Tarikh_TransactionID
ON dbo.TransactionDetail (AccountID, tarikh, TransactionID)

SET NOCOUNT ON

DECLARE @Amount decimal(20,0)
DECLARE @CurAccountID bigINT

DECLARE @PrevAccountID bigINT
DECLARE @AccountRunningTotal decimal(20,0)
DECLARE @PreAccountRunningTotal decimal(20,0)

DECLARE curRunningTotal CURSOR LOCAL FORWARD_ONLY
FOR
SELECT AccountID, gardesh
FROM dbo.TransactionDetail
ORDER BY AccountID, tarikh, TransactionID
OPEN curRunningTotal

FETCH NEXT FROM curRunningTotal
INTO @CurAccountID, @Amount


WHILE @@FETCH_STATUS = 0
BEGIN


SELECT
@AccountRunningTotal = CASE
WHEN @CurAccountID = @PrevAccountID
THEN @AccountRunningTotal + @Amount
ELSE @Amount
END,

@PreAccountRunningTotal = CASE
WHEN @CurAccountID = @PrevAccountID
THEN @AccountRunningTotal - @Amount

ELSE 0
END,
@PrevAccountID = @CurAccountID
UPDATE dbo.TransactionDetail
SET balance = @AccountRunningTotal,
Prebalance = @PreAccountRunningTotal
WHERE CURRENT OF curRunningTotal

FETCH NEXT FROM curRunningTotal
INTO @CurAccountID, @Amount

END


Its slow in huge records

thanks for any faster solution

robvolk
Most Valuable Yak

15732 Posts

Posted - 2011-01-28 : 14:24:09
Why are you storing running totals in a table?
Go to Top of Page

fan2005
Yak Posting Veteran

85 Posts

Posted - 2011-01-28 : 14:42:59
quote:
Originally posted by robvolk

Why are you storing running totals in a table?



it should be in temptable
i need to store it for further reports
Go to Top of Page

robvolk
Most Valuable Yak

15732 Posts

Posted - 2011-01-28 : 14:47:42
What are you using as your reporting layer? Reporting Services? Crystal Reports? Something else?

Do you really need to report on all 70 million rows? How often is your summary table generated? (once a day, once a week, every time the report runs, etc.)

The key point in improving performance here is reducing the amount of work and data needed.

Go to Top of Page

fan2005
Yak Posting Veteran

85 Posts

Posted - 2011-01-28 : 14:53:32
different users work on this server who need to work with balance

so whenever we want this info we insert new data

then for example we want the customers whith average balance over x
in y months we select the part of data we want


what do you think is better to do?is it ok, every one needs this table run a SP to build it?
is it Ok with query itself?
Go to Top of Page

robvolk
Most Valuable Yak

15732 Posts

Posted - 2011-01-28 : 15:08:15
I hate to say this but you didn't answer any of my questions. Please read them again and respond with specifics.

As far as what is better to do, I can't say without a lot more information. But I can say what NOT to do:

- Don't summarize 70 million rows every time you need to generate a report.
- If the data is historical and rarely changes, don't recreate the same summaries on it. That means don't create a new temp table each time. Summarize it once and store it permanently.
- If this is intended as an ad-hoc reporting tool, don't use a summary table at all. Your users are not going to need all this data, so if they already have a specific customer, date range, or other criteria that limits the data they need, query the original table(s) with those criteria and summarize as needed.
- Don't do running totals in SQL Server, use a reporting tool to do it. The same applies to the other totals/sums.
Go to Top of Page

fan2005
Yak Posting Veteran

85 Posts

Posted - 2011-01-28 : 15:20:51
quote:
Originally posted by robvolk

I hate to say this but you didn't answer any of my questions. Please read them again and respond with specifics.

As far as what is better to do, I can't say without a lot more information. But I can say what NOT to do:

- Don't summarize 70 million rows every time you need to generate a report.
- If the data is historical and rarely changes, don't recreate the same summaries on it. That means don't create a new temp table each time. Summarize it once and store it permanently.
- If this is intended as an ad-hoc reporting tool, don't use a summary table at all. Your users are not going to need all this data, so if they already have a specific customer, date range, or other criteria that limits the data they need, query the original table(s) with those criteria and summarize as needed.
- Don't do running totals in SQL Server, use a reporting tool to do it. The same applies to the other totals/sums.



there is no reporting tools
all we have is sql server
this is a bank account so data is changing all times , but we dont have data online because it is on a oracle server that we can just select (we use the results of views )so we bring data to sql every time we need .
and yes i need all data when i make a report . the origenal data has no balance it has only incoming and outgoing transactions.
I basically am looking for an added calculated column which sums all the previous transactions plus the current one to give me a running total.


Go to Top of Page

robvolk
Most Valuable Yak

15732 Posts

Posted - 2011-01-28 : 15:59:59
For the running total, get rid of the cursor entirely, replace it with this:
DECLARE @PrevAccountID       bigINT = 0
DECLARE @AccountRunningTotal decimal(20,0) = 0

UPDATE dbo.TransactionDetail SET
@AccountRunningTotal = balance = CASE
WHEN AccountID = @PrevAccountID THEN @AccountRunningTotal + gardesh
ELSE gardesh END,
@PrevAccountID=AccountID
This technique relies on non-standard behavior of the UPDATE statement, it's documented in Books Online. The UPDATE processes the data in the physical order of the table, since you've created a clustered index in that same order it should work correctly. Be advised this behavior may not be consistent on all SQL Server versions or may change in a future release. There's an article with another example here: http://www.sqlteam.com/article/creating-a-sequential-record-number-field

I also suggest reading up on the OVER clause of the aggreate functions, in your case SUM(). You can probably rewrite the original query using OVER to get the AccountID, tarikh sums without joining to a subquery, it should run faster.

Regarding the preBalance column, make it a computed column instead:
alter table TransactionDetail add preBalance AS Balance - gardesh
No point in storing that value when it can be calculated on the fly.
Go to Top of Page

fan2005
Yak Posting Veteran

85 Posts

Posted - 2011-01-28 : 16:13:52
smart
thank you very much
Go to Top of Page

fan2005
Yak Posting Veteran

85 Posts

Posted - 2011-01-28 : 16:33:47
OVER clause of the aggreate sum()
select distinct s.account ,row as accountID,s.tarikh ,s.gardesh
from
(SELECT distinct b.account,b.amount, b.tarikh , sum(b.amount)OVER (PARTITION BY b.account,b.tarikh ) as gardesh FROM gardesh b)s
inner join
allacounts a on a.account=s.account
order by s.account ,s.tarikh


account accountID tarikh gardesh
1-11-1 101 2010/01/05 51000.00
1-11-1 101 2010/02/05 -3000.00
1-11-1 101 2010/09/09 1000.00
1-11-1 101 2010/12/06 2000.00
1-11-1 101 2010/12/23 1000.00
1-11-2 102 2010/05/01 65000.00
1-12-2 103 2010/09/08 6000.00
2-12-1 104 2010/04/12 100000.00
2-12-1 104 2010/09/21 -1000.00
Go to Top of Page

robvolk
Most Valuable Yak

15732 Posts

Posted - 2011-01-28 : 16:46:03
What constitutes a duplicate? The totals match what you originally posted.

If OVER doesn't give you the results you want then use the original query.
Go to Top of Page

fan2005
Yak Posting Veteran

85 Posts

Posted - 2011-01-28 : 16:52:35
another question does it work in sql server 2000 too?
Go to Top of Page

fan2005
Yak Posting Veteran

85 Posts

Posted - 2011-01-28 : 16:54:33
quote:
Originally posted by robvolk

What constitutes a duplicate? The totals match what you originally posted.

If OVER doesn't give you the results you want then use the original query.



I made a mistake then I edited the post no duplication .that's greate
Go to Top of Page

robvolk
Most Valuable Yak

15732 Posts

Posted - 2011-01-28 : 17:03:40
No, the OVER clause was added in SQL Server 2005, you'll have to use the original query for SQL Server 2000.
Go to Top of Page
   

- Advertisement -