Any perl people in the house? - GoFuckYourself.com

gezzed · 02-24-2005, 04:43 PM

I'm trying to develop a script that searches through a simple .txt file and returns all the records matching a search query.

This is the general format of the .txt file

xxx----somecategory----123Mb
yyy----someothercategory----456Mb
zzz----onemorecategory---789Mb

Then, someone specifies a query through my site and I need to check whether any of the records in the .txt file match the query by scanning through the xxx, yyy, zzz parts.

Here's the snag, I have 50,000 entries in the .txt file so a simple foreach loop takes way too long.

Any ideas?

mortenb · 02-24-2005, 04:50 PM

The simplest solution would be to stop using a txt file and throw the data in a db..

gezzed · 02-24-2005, 04:52 PM

Quote:

Originally Posted by mortenb

The simplest solution would be to stop using a txt file and throw the data in a db..

I've always been a fan of minimalism. I suppose that if there is no other way I'll look into using SQL. But I'd like believe there is a way around this problem that doesn't complicate matters further.

NetRodent · 02-24-2005, 05:07 PM

[QUOTE=gezzed]Here's the snag, I have 50,000 entries in the .txt file so a simple foreach loop takes way too long./QUOTE]

Its not the foreach that's taking the time. Its the pattern matching on 50,000 entries.

The easiest way would be with a database, properly indexed.

If you're stuck on storing your data in text files, you could create a new set of textfiles to index what's in the main text file.

Tom_PM · 02-24-2005, 05:08 PM

Probably with grep. It's inherently line-based and fast with regex. I dont have a sample, but that should get you on the right path I think.

gezzed · 02-24-2005, 05:20 PM

Quote:

Originally Posted by PR_Tom

Probably with grep. It's inherently line-based and fast with regex. I dont have a sample, but that should get you on the right path I think.

Thanks Tom, I'll have a look into it.

gezzed · 02-24-2005, 05:51 PM

Looks like grep is the answer. Thanks again Tom.

sonofsam · 02-24-2005, 05:54 PM

sorry

swedguy · 02-24-2005, 06:01 PM

Is the xxx,yyy,zzz a set length (like always 3 chars long)?

MrJackMeHoff · 02-24-2005, 06:02 PM

Using a %hash you could search 50k text file easy

teksonline · 02-25-2005, 06:22 AM

use regular expressions

hit me up if you want on aim

freebsdteks

02-24-2005, 04:43 PM	#1
gezzed Confirmed User Join Date: Mar 2002 Posts: 317	Any perl people in the house? I'm trying to develop a script that searches through a simple .txt file and returns all the records matching a search query. This is the general format of the .txt file xxx----somecategory----123Mb yyy----someothercategory----456Mb zzz----onemorecategory---789Mb Then, someone specifies a query through my site and I need to check whether any of the records in the .txt file match the query by scanning through the xxx, yyy, zzz parts. Here's the snag, I have 50,000 entries in the .txt file so a simple foreach loop takes way too long. Any ideas?

02-24-2005, 05:07 PM	#4
NetRodent Confirmed User Join Date: Jan 2002 Location: In the walls of your house. Posts: 3,985	[QUOTE=gezzed]Here's the snag, I have 50,000 entries in the .txt file so a simple foreach loop takes way too long./QUOTE] Its not the foreach that's taking the time. Its the pattern matching on 50,000 entries. The easiest way would be with a database, properly indexed. If you're stuck on storing your data in text files, you could create a new set of textfiles to index what's in the main text file. __________________ "Every normal man must be tempted, at times, to spit on his hands, hoist the black flag, and begin slitting throats." --H.L. Mencken

02-24-2005, 05:08 PM	#5
Tom_PM Porn Meister Industry Role: Join Date: Feb 2005 Posts: 16,443	Probably with grep. It's inherently line-based and fast with regex. I dont have a sample, but that should get you on the right path I think. __________________ 43-922-863 Shut up and play your guitar.

02-24-2005, 05:54 PM	#8
sonofsam Too lazy to set a custom title Join Date: Dec 2004 Location: Vancouver Posts: 18,638	sorry __________________ I like turtles.

02-24-2005, 06:02 PM	#10
MrJackMeHoff Confirmed User Join Date: Mar 2004 Location: LOLLIPOP ISLAND =-=-=-=-=-=-=-=-=-=-= =-=-=-=-=-=-=-=-=-=-= =-=-=-=-=-=-=-=-=-=-= =-=-=-=-=-=-=-=-=-=-= =-=-=-=-=-=-=-=-=-=-= =-=-=-=-=-=-=-=-=-=-= =-=-=-=-=-=-=-=-=-=-= =-=-=-=-=-=-=-=-=-=-= =-=-=-=-=-=-=-=-=-=-= =-=-=-=-=-=-=-=-=-=-= Posts: 4,569	Using a %hash you could search 50k text file easy __________________

02-24-2005, 04:50 PM	#2
mortenb Confirmed User Join Date: Jul 2004 Location: Denmark ICQ: 7880009 Posts: 2,203	The simplest solution would be to stop using a txt file and throw the data in a db..

02-24-2005, 05:51 PM	#7
gezzed Confirmed User Join Date: Mar 2002 Posts: 317	Looks like grep is the answer. Thanks again Tom.

02-24-2005, 06:01 PM	#9
swedguy Confirmed User Industry Role: Join Date: Jan 2002 Posts: 7,981	Is the xxx,yyy,zzz a set length (like always 3 chars long)?

02-25-2005, 06:22 AM	#11
teksonline So Fucking Banned Join Date: Jan 2005 Location: At My Desk Posts: 2,904	use regular expressions hit me up if you want on aim freebsdteks