| Author |
Topic |
|
mclusker
Starting Member
6 Posts |
 Posted - 2009-07-12 : 02:41:44
|
| Hello Everyone,I need to manipulate 43 million records. I will be querying off this data with joins. So the data can exponentially grow. What is the most efficient approach to handle this amount of data? Thank you ahead of time. MattMatt |
|
|
SwePeso
Patron Saint of Lost Yaks
30421 Posts |
Posted - 2009-07-12 : 02:50:01
|
It depends on what your definition of "manipulate" is...DELETE? UPDATE? INSERT?
N 56°04'39.26"E 12°55'05.63" |
 |
|
|
khtan
In (Som, Ni, Yak)
17689 Posts |
Posted - 2009-07-12 : 02:55:35
|
quote:
I will be querying off this data with joins
It will depends on what is the query doing. What are the indexes on the related tables.
KH[spoiler]Time is always against us[/spoiler] |
|
|
|
mclusker
Starting Member
6 Posts |
Posted - 2009-07-12 : 03:56:10
|
| My definition of manipulate is 'Update' 'Delete' and 'Insert'. I need to manage edits constantly. I will use a base index (I think, still learning about indexes) that consists of 43 million + records. With that base record I need to join it with other tables that consist of less records. Then periodically I will have to update, delete, insert into the base record to keep it accurate. SQL is the right tool? Do I need to have dedicated server? If so, what are the minimum specs to have it run fast with cost savings in mind?Matt |
|
|
|
visakh16
Very Important crosS Applying yaK Herder
52326 Posts |
Posted - 2009-07-12 : 04:14:09
|
| yup sql is right tool. however whats your rule of manipulate? if more than one tables have reference to single record, which data you want to update? or is it like one table contain details of records to be deleted, one for insert etc.? perhaps you could explain your scenario with some sample data and table stuctures. |
|
|
|
mclusker
Starting Member
6 Posts |
Posted - 2009-07-12 : 04:30:34
|
| Hey visakh16,Thank you for the reply. I understand the joins. The nitty gritty is that I have a massive amount of data to deal with/manipulate...What is the best way to process this amount of data? Buy a powerful server for myself? Rent a dedicated server? (I don't know maintenance so I'm figuring I need to do this) or....any suggestion is welcome. |
|
|
|
visakh16
Very Important crosS Applying yaK Herder
52326 Posts |
Posted - 2009-07-12 : 04:51:16
|
| what would be approx amount of data? i think you need a moderate server as long as you use proper indexes as well as write optimised query |
|
|
|
mclusker
Starting Member
6 Posts |
Posted - 2009-07-12 : 05:34:23
|
| Thank you very much visakh16! I am gaining an understanding of indexes as I research so I know I can break this data stream into more manageable pieces. I will have to look more into optimizing my queries by testing them against the gigs of data I will be analyzing. I will conduct this research and be back definitely to this forum because of resources like yourself. I have a great start. Thanks again. |
|
|
|
visakh16
Very Important crosS Applying yaK Herder
52326 Posts |
Posted - 2009-07-12 : 08:26:24
|
| welcome |
|
|
|
|
Efficient Processing of Millions of Records??