SQL Server Forums
Profile | Register | Active Topics | Members | Search | Forum FAQ
 
Register Now and get your question answered!
Username:
Password:
Save Password
Forgot your Password?

 All Forums
 General SQL Server Forums
 New to SQL Server Programming
 FLAT File indexing Vs RDBMS Indexing
 New Topic  Reply to Topic
 Printer Friendly
Author Previous Topic Topic Next Topic  

abuhassan
Posting Yak Master

105 Posts

Posted - 09/22/2006 :  09:51:41  Show Profile  Reply with Quote
Hi

I want to know is a flat file faster than a RDBMS for indexing for example a search engine indexing would a flat file be better in terms of performance, scalability etc than a RDBMS?

X002548
Not Just a Number

15586 Posts

Posted - 09/22/2006 :  09:53:25  Show Profile  Reply with Quote
How would you "Index" a flat file.

Or are we talking Mainframe and VSAM vs. DB2?



Brett

8-)

Hint: Want your questions answered fast? Follow the direction in this link
http://weblogs.sqlteam.com/brettk/archive/2005/05/25/5276.aspx

Add yourself!
http://www.frappr.com/sqlteam
Go to Top of Page

abuhassan
Posting Yak Master

105 Posts

Posted - 09/22/2006 :  09:58:14  Show Profile  Reply with Quote
i had a discussion with a search engine architect he seem to think that the flat file was much better in terms of performance than having say an index in a relational database such as sql server or oracle.
Go to Top of Page

X002548
Not Just a Number

15586 Posts

Posted - 09/22/2006 :  10:04:09  Show Profile  Reply with Quote
Is this a troll?



Brett

8-)

Hint: Want your questions answered fast? Follow the direction in this link
http://weblogs.sqlteam.com/brettk/archive/2005/05/25/5276.aspx

Add yourself!
http://www.frappr.com/sqlteam
Go to Top of Page

Michael Valentine Jones
Yak DBA Kernel (pronounced Colonel)

USA
7020 Posts

Posted - 09/22/2006 :  10:18:07  Show Profile  Reply with Quote
quote:
Originally posted by abuhassan

i had a discussion with a search engine architect he seem to think that the flat file was much better in terms of performance than having say an index in a relational database such as sql server or oracle.



Why don't you ask the "search engine architect" to explain why it is better? Ask for a link to results using flat files for one of the standard TCP benchmark tests.




CODO ERGO SUM
Go to Top of Page

spirit1
Cybernetic Yak Master

Slovenia
11750 Posts

Posted - 09/22/2006 :  10:18:59  Show Profile  Visit spirit1's Homepage  Reply with Quote
it depends on what your application will be used for.
sometimes a db is an overkill and using flat files is a great option.

reading content from a flat file is a lot faster than reading from a DB or XML file.

you might want to tell us is this an academic discussion or a real world problem?

also in what way would you index your files?
would you index just the file names or their content? how would you go about that?
is the data in your files in chunks, evenly distributed etc...

The advantage of DB's is that you don't have to worry about all that

but in essence your friend has a point.
reading from a file is faster than reading from a db.




Go with the flow & have fun! Else fight the flow
blog thingie: http://weblogs.sqlteam.com/mladenp
Go to Top of Page

abuhassan
Posting Yak Master

105 Posts

Posted - 09/22/2006 :  11:12:08  Show Profile  Reply with Quote
i Asked the architect he said he doent want to give his design away but said look why do the major search indexer use flat file not RDBMS?
Go to Top of Page

spirit1
Cybernetic Yak Master

Slovenia
11750 Posts

Posted - 09/22/2006 :  11:14:11  Show Profile  Visit spirit1's Homepage  Reply with Quote
huh? major search indexer? like who?





Go with the flow & have fun! Else fight the flow
blog thingie: http://weblogs.sqlteam.com/mladenp
Go to Top of Page

X002548
Not Just a Number

15586 Posts

Posted - 09/22/2006 :  11:31:16  Show Profile  Reply with Quote
Using a fla file is akin to a table scan...actually worse...it's like using a cursor

So actually if you need to reference the flat file more than once, you'd have to cursor through the whole file every time.

Does that sound effecient?



Brett

8-)

Hint: Want your questions answered fast? Follow the direction in this link
http://weblogs.sqlteam.com/brettk/archive/2005/05/25/5276.aspx

Add yourself!
http://www.frappr.com/sqlteam
Go to Top of Page

abuhassan
Posting Yak Master

105 Posts

Posted - 09/22/2006 :  11:44:36  Show Profile  Reply with Quote
how does google index does it use flatfile or database?
Go to Top of Page

spirit1
Cybernetic Yak Master

Slovenia
11750 Posts

Posted - 09/22/2006 :  11:50:20  Show Profile  Visit spirit1's Homepage  Reply with Quote
this is how google stores the data:
http://labs.google.com/papers/bigtable.html

more here in the comments:
http://googlesystem.blogspot.com/2006/09/paper-about-bigtable.html

acctually reading a flat file is very fast because the file is read into the memory.
you can't acctually compare the two ways because flat files are read with procedural languages that
are optimized in the same way as RDBMS are optimized for working with sets.



Go with the flow & have fun! Else fight the flow
blog thingie: http://weblogs.sqlteam.com/mladenp

Edited by - spirit1 on 09/22/2006 11:54:55
Go to Top of Page

spirit1
Cybernetic Yak Master

Slovenia
11750 Posts

Posted - 09/22/2006 :  11:56:32  Show Profile  Visit spirit1's Homepage  Reply with Quote
and some more about Google file system:
http://en.wikipedia.org/wiki/Google_File_System



Go with the flow & have fun! Else fight the flow
blog thingie: http://weblogs.sqlteam.com/mladenp
Go to Top of Page

X002548
Not Just a Number

15586 Posts

Posted - 09/22/2006 :  12:09:08  Show Profile  Reply with Quote
quote:
Originally posted by spirit1

acctually reading a flat file is very fast because the file is read into the memory.



Actually most of my databases reside in memory



Brett

8-)

Hint: Want your questions answered fast? Follow the direction in this link
http://weblogs.sqlteam.com/brettk/archive/2005/05/25/5276.aspx

Add yourself!
http://www.frappr.com/sqlteam
Go to Top of Page

nr
SQLTeam MVY

United Kingdom
12543 Posts

Posted - 09/22/2006 :  12:33:08  Show Profile  Visit nr's Homepage  Reply with Quote
What is a database other than a wrapper for a flat file? The data is all held in files.
All a database is is an access method for those files.
If you have a application which needs to be efficient and the benefits outweigh the cost of effort in developing it then of course a bespoke access method and storage tailored to your system would be more efficient - you can concentrate on just retrieving the data rather than all the other stuff relational database do - think about updates and multi-user access though.
Of ciourse you probably wouldn't use windows or whatever and would devlope your own shell to access it and might go as far as building your own disks (or whatever) to match the structure of the data you are saving.

Anyone who comes from an asembler or engineering background will tell you that any generic solution will compromise on efficiency (but maybe not on cost).

I spent 7 years working on assembler based systems using a flat files for storage with hashing algorithms to access the data because the (ok it was primitive) isam methods that came with the system were too slow (and unreliable).

==========================================
Cursors are useful if you don't know sql.
DTS can be used in a similar way.
Beer is not cold and it isn't fizzy.

Edited by - nr on 09/22/2006 12:34:34
Go to Top of Page

robvolk
Most Valuable Yak

USA
15659 Posts

Posted - 09/22/2006 :  15:22:48  Show Profile  Visit robvolk's Homepage  Reply with Quote
quote:
Originally posted by abuhassan

i Asked the architect he said he doent want to give his design away but said look why do the major search indexer use flat file not RDBMS?
Translation: Search architect doesn't know what he's talking about and doesn't want to admit it to anyone. Be very suspicious of people who won't answer simple questions, they're hiding something.

He probably thinks SQL Server and Oracle are glorified spreadsheets, if he's even used them before, and doesn't understand relational theory. Probably doesn't understand advanced text searching and indexing algorithms either, because searching flat files that are not indexed is gonna be as slow as anything.
Go to Top of Page
  Previous Topic Topic Next Topic  
 New Topic  Reply to Topic
 Printer Friendly
Jump To:
SQL Server Forums © 2000-2009 SQLTeam Publishing, LLC Go To Top Of Page
This page was generated in 0.14 seconds. Powered By: Snitz Forums 2000