HQ

How do I make a perl script stay resident (compiled) in memory?

I have a few scripts that are executed multiple times per second. It makes no sense for my server to compile these scripts every single execution. Is there a way to leave the script in memory and, better yet, leave the script compiled in memory?

notjoe

FastCGI, it is an apache module, does what you want it to do.

Joe

HQ

Thanks Joe. Someone just told me modperl (mod_perl?) might do this. Know anything about it?

notjoe

I think your friend is right, i always get those two mixed up however, for optimized speed i would suggest using both. You might want to RTMF on the details of both the modules as i have not used/dealt with perl in years!

Here are a couple sites to get you started!

mod_perl - http://perl.apache.org/docs/1.0/guide/
FastCGI - http://www.fastcgi.com/
Joe

HQ

Great, thanks.

RTMF = Read The Manual First (Finally) ???

some_idiot

ModPerl just allows you to put Perl embedded in your
HTML. You want the FastCGI module for leaving scripts
up in memory.

HQ

some_idiot, thanks. Have you used either or both before? I would appreciate some comments from someone who has done this before. There always seems to be a list of common bugs/mistakes with these type of things that I would like to avoid.

I'm reading up on both right now; I will be posting my results on GFY of course.

notjoe

HA, i knew i was right, i guess i should have RTMF'd (i like to think of it as READ THE FUCKING MANUAL) but like i said i havent used perl for a long long time.

Why not redo your scripts in PHP? i think php is way better than perl anyways!

HQ

RTMF = Read the manual, fucker!

I need a quick fix for now... later on I will re-code my script in PHP or C. Even though perl has kick-ass text manipulation functions, PHP must be faster for the simple fact that it is not a interpretive language... right?

Phil21

Uhm, why don't you figure out what you're talking about before you spout off incorrect crap.

Mod_Perl does *EXACTLY* what he wants, but *IS NOT* CGI. It is not a tool to "embed" perl into html pages, it CAN be used to do that, but is one of those insanely powerful things.

In fact, mod_perl is used to build some extremely hard hit dynamic sites out there, where using CGI is impractical. Think of it something like a persistant script always running (i.e. if I say, increase a global variable $blah++, then the next visitor will be able to see that increase immediate, without any disk writing).

You need to code specificially for mod_perl though. I repeat, it's NOT CGI. CGI is just a spec as to how applications talk to a webserver, mod_perl is a perl interpreter directly compiled into the webserver (so to speak). It's also not easy to code correctly for mod_perl either, there are very few gurus in this area out in the world, I'd venture to say less than 100.

Figures on GFY someone would akin mod_perl to PHP. Go figure.


I'm not all too familiar with FastCGI, but from what I recall is it not just a means to cache things in memory? Shrug. It will probably help out, and be the easiest to use.

-Phil

TheFLY

Your acronyms are FUBAR -- it's ALWAYS been RTFM (Read The FUCKING Manual) -- not RTMF

hyper

Read The Manual Fuckhead .. works for me

HQ whats up? hit me up on icq

__________________

Phil21

PHP isn't interpreted? heh.

PHP is slow as hell when compared to other languages when you take away CGI. i.e. PHP vs. mod_perl there isn't even a contest.

PHP is very much so interpreted. In fact, it's based on Perl.

Of course, I am bigotted against PHP, I think it's a very poorly done language, with not too much regard towards security and availability. It's getting better, but is still an absolute headache to support in a hosting role, when upgrades tend to break random functions in peoples scripts and whatnot.

PHP is fine for many uses, but performance is not one of them. It may be better than */CGI in most cases, but not always.

-Phil

TheFLY

CGI = Common Gateway Interface

ROTFL = Rolling on the Floor Laughing

We must preserve the traditions of our forefathers!!!!!!!!!

TheFLY

Well to spam your hosting services -- I should say that my PHP scripts always run fine on your servers ;)

HQ

Phil21, you seem to know what you are talking about. I have never programmed PHP before, I thought it compiled? Why would it be faster than (basic) PERL if it does not compile?

TheFLY, both may be right:

1)

http://www.acronymfinder.com/af-quer...t&Acronym=RTMF

RTMF = Read The Manual First (Finally)

2)

http://www.acronymfinder.com/af-quer...t&Acronym=RTFM

RTFM = Read The Friendly Manual (polite form)

hahahaha

(excuse the popups from acronymfinder.com)

HQ

http://www.acronymfinder.com/af-quer...act&Acronym=HQ

HQ Headquarters
HQ Hardware Qualification
HQ Have Quick
HQ Hazard Quotient*
HQ High Cube
HQ High Quality*
HQ Highly Qualified*

* = the HQ you know and love.

TheFLY

Come on I used to hang out with guys that wrote code with paper punch cards LOL it's RTFM -- never heard of RTMF. Get it straight man. Preserve the history.

RTFM /R-T-F-M/ imp. [UNIX] Acronym for `Read The Fucking Manual'. 1. Used by gurus to brush off questions they consider trivial or annoying. Compare Don't do that, then!. 2. Used when reporting a problem to indicate that you aren't just asking out of randomness. "No, I can't figure out how to interface UNIX to my toaster, and yes, I have RTFM." Unlike sense 1, this use is considered polite. See also FM, RTFAQ, RTFB, RTFS, RTM, all of which mutated from RTFM, and compare UTSL

http://www.dpmms.cam.ac.uk/~gjm11/jargon/jargR.html

HQ

I agree. RTFM is more popular than RTMF. It probably started as RTM and then went to RTFM as RTM -> RTMF makes no sense.

http://www.dictionary.com/cgi-bin/di...db=*&term=rtfm

HQ

What is miniserv.pl? Is this perl's interpreter that exectutes everytime a perl script is executed?

cgiGeek

Phil21 is right, all others not sure what the heck they are talking about.
Php can do all text parsing perl can but in many more lines.
Php has an advantage over perl in performance for web output,
but is not worth the hassle in my professional opinion.
Mod_perl and fast cgi are not intented for kids, few of us perl hardcore fans can code for mod_perl or fast cgi.
Most of perl scripst out there wont run on mod_perl or fast cgi, they need to be ported, yes, even if they are both perl they diferent in requirements by a mile.
Get a good sys admin to hook you mod_perl,
and get a pro to check and probaly port your script.

Real men use Perl.
All php fans are perl reyects.

__________________
Need help dealing with a hacked website? Contact me via icq 163583431 :D
Premium Bandwidth Managed Servers, 12 Years in business xxxHOSTit.com
I work for xxxhostit.com

some_idiot

I've used both with the Liberty Health site coverme.com
in Canada.

We were taking 10,000 hits a hour on an insurance
system so it HAD to be solid.

The one thing we found, was check EVERYTHING twice
before putting it on the live server, and double your
server's memory from what you'd expect with just
Apache running. FastCGI gave us about 30% speed
increase over FreeBSD with soft updates turned on
on the file system.

some_idiot

Whoa, who pissed in your corn flakes this morning?

Mod_Perl DOES NOT LEAVE THE CODE RESIDENT, it's
an imbedded interupter. EmbPerl is the spec and
Mod_Perl is for embedding Perl into HTML. Geez buddy,
grab your coffee and relax until you can think straight.

Don't you hate it when you act like a jerk and you get
nailed to the wall in public for making stupid statements?

XXXManager

Well. Lucky for you this is GFY cause here everybody gets a chance to say stupid things about stuff they don?t have a clue about.

HQ. MOD_PERL will do what you want and more. Same thing about FAST-CGI. They are both aimed at improving the speed of your script execution.
While FAST_CGI is general CGI accelerator MOD_PERL is PERL ONLY.
As any specific target product MOD_PERL might do better for you if you use it correctly - that is if you are using PERL.
MOD_PERL is not for embedding perl into HTML. embed_perl is embedding the mod_perl module into Apache.
MOD_PERL compiles your PERL and stick it into an infinite loop where each call to the original script just releases another execution of the main loop.
Indeed, like Phil21 (who was, compared to some_idiot, at least half right) suggested, you need to know how to "modify" your scripts to MOD_PERL and there is something like global variables BUT unlike he said - dont do the $Blah++ mistake. Its a long technical explanation, but suffice to say, if you do that - you wont get what you expect. at least not the 1, 2, 3, 4, output you might expect.
Unlike some idiot has suggested, MOD_PERL DOES keep the code (compiled version) resident in memory (at least until the original code is modified or Apache is shut down) and that saves both compilation and loading time. Not only does it leave the code in memory, it even leaves multiple instances of it (unless you configure it differently).
In addition it enables you to use Database connection pooling that will accelerate greatly - db centered application.

FAST_CGI works in a different way and also requires special code modification. It works in a manner similar to MOD_PERL but outside the web-server. The disadvantage that FAST_CGI has - is a much smaller user-base than MOD_PERL - hence much smaller "support" group.

Using both like some idiot suggested here - is mostly idiotic (surprise, surprise).
Google "FAST CGI vs. MOD PERL" if you want to learn more.

That?s my

some_idiot

Sure, sure, sure... if you want to consider preloading
every script on the system that you may or may not
use and shuffling them to handler a good method in
the real world of (make a perl script stay resident).

Go head... Mod_Perl is not ment for the same things
that FastCGI is. One should not hammer with a screw
driver.

Just go to the Real World Scenarios Implementation
section on the Perl docs sites and you will see for yourself
which works better for the scripts of this industry and
which works better for middleware to a database.

(oops, board does not like quotes...)

Phil21

Er, mod_perl can indeed (in fact, I personally havn't seen otherwise, although my exerpience with it IS limited) keep code resident.

That's one reason why it's such a performance increase over CGI, you don't need to fork off a process that compiles the code, then runs it, then spits back output.

And no, the $blah++ was just to illustrate a point, wasn't trying to say it would act a certain way. The idea being that you can use "global" variables and have the accessable by all your apache children at once.

What I said was technically correct. And like I said I have nearly zero experience with fastCGI, but is it not really a form of a cacheing daemon? A proxy server to your scripts, so to speak.

No, I don't code for mod_perl. I tend to do the more basic Perl stuff myself. But, my business partner is hardcore into it, and I also contract out some of the more complex C/mod_perl/etc. stuff to another guy I know well.

Anyways, my point being that mod_perl is not just to "imbed perl into html pages" which would simply make it a glorified templating language. There are many ways to do that in mod_perl, EmbedPerl being one of them AFAIK.

To answer the question about why PHP is faster in some cases that Perl/CGI is that the interpreter of PHP is built-in to apache (in almost all cases at least). This saves having to start up the interpreter and load it into memory for each hit.

-Phil

HQ

Doing a bit of research... what about Velocigen, PerlEx, and Speedy::CGI?

PerlEX is only for windows systems, I *think*.

vending_machine

mod_perl is the biggest piece of shit in the whole universe. I can't believe how crappy that code is.

FastCGI is ok, but I'd stay clear of it. Write good code, or write an apache module. If you have good code, it shouldn't matter that it's only loaded a few times / sec.

HQ

So far, mod_perl appears to be the answer:

http://perl.apache.org/

HQ

"mod_perl gives you a persistent Perl interpreter embedded in your web server. This lets you avoid the overhead of starting an external interpreter and avoids the penalty of Perl start-up time, giving you super-fast dynamic content." - http://perl.apache.org/

Big E

FastCGI just encapsulations the input and output of CGI scripts, keeping it in memory..

So basically, your CGI script (Perl, C, whatever) just needs to accept the ENV parameters like usual and output text, and the FastCGI module "encapsulates" that script, feeding it input and displaying output.

I've used it quiet successfully when I needed a "running" counter in between iterations and for scalability purposes didn't want to load/increment a db table or file.

HQ

Who here has recently installed mod_perl or remembers installing it. If anyone has, I would appreciate them contacting me on ICQ. I have to go now, but I will be in touch with anyone who ICQs me.

Thanks.

ICQ: 94968434

vending_machine

I would strongly advise against mod_perl. It adds much more complexity to the system than needed. mod_perl is "good" for huge complex sites, not for your needs (good in quotes as mod_perl in itself is a beast of code that should never have been written).

Do you have some perl scripts killing your server or something? Hire someone to write them in C if possible.

HQ

I have some perl scripts killing my server, yes. *I* can write them in C, which I am going to do. However, I just need a QUICK fix until I get my C coding done.

TheFLY

Honestly I think most people wouldn't need to worry about increasing performance if their code was tight in the first place. All these gay libraries are just wasted CPU cycles. EX: Why waste so much CPU time on tracking every IP address for a trading script? Throw your whole IP database in the trash and trade on cookies. You fucking morons -- all you do to make money is this:

#!/usr/bin/perl

$click++;$click++;$click++;$click++; # MO TRAFFIC!!!
display_html($cash_money); # MO MONEY!!!

exit;

TheFLY

I used to love programming until windows came out. Dude I wish we all still used DOS. I used to love addressing the video memory directly in 320x200x256 -- Mode 3 or something, right?... And fuck all that OOP crap like garbage collection and portability -- like it made programming any better or more fun... Windows was the worst thing that ever happened to programming. I remember when programmers coded their own window libraries... If I wanted to draw a damn window I would draw the fucking window myself direct to the video memory with my own fucking code... If I wanted fonts -- I would draw the fucking letters myself! If I wanted the letters to have colors I'd code my own fucking algorithm to shade them! If I wanted to access the modem I could communicate directly with COM1 or COM2... If I wanted to make a sound I'd fucking talk directly to the fucking sound card! Now you got to jump through the hoops of anal retentive dumbasses...

It just ain't the same man. Too much bullshit.

HQ

TheFLY, not really...

Every time my perl script is executed, a compiler has to be loaded into memory and executed, then the script has to be compiled and placed into memory, then the compiled version of the script has to be execute...

There has to be a faster way. I did not even know of any way to keep a compiled version of the script in memory, but I just *knew* that someone had to have thought of this optimization before.

I am coding anything that is run per-click or per-hit in C from now on...

HQ

I hear you.

BTW, mode 13h dude... and you weren't cool until you could get into mode X and do triple buffering.

Nbritte

Yeah I like cookies I have a free site that uses 4 diffrent scripts, one to join a bbs, chat and IM script and the interface between the script is a cookie set on the users computer. If you are tracking surfers why store the information on your site when you can store it on their computer for free

Brian

HQ

"and" = "and/or"... I'm not sure if mode x had more than 2 buffers.

spanky

yup, PerlEx is just for windows. It's pretty solid, there are some pretty heavy sites using it without problems. We had a **huge** performance increase when we got it for one of our sites.

if you have the same web/cgi script ticking off several times a second you will probably see a huge improvement porting it to mod_perl. The porting would probably be pretty easy, you can look into the Apache::Registry module for running perl cgi scripts under mod_perl with little to no change and Apache::DBI for persistant database connections. There are issues with using non-thread safe perl libs under PerlEx and there probably will be with mod_perl and the new multi-threaded Apache.

I only noticed a small speed improvement using the apache c api over mod_perl except for a project that required custom crypto. The C crypto routines really took off compared to the perl (big surprise ;-) but I bet that if the crypto had been compiled as an xs lib the mod_perl version would have been almost as fast.

HQ

Using cookies, how do you keep track of stats of all the surfers in just one individual trade? For example, if you suspect a hitbotter and you want to list of all the details of everyone from that trade.

TheFLY

Well what are you trying to do that is making your script run so slow in the first place? Go through your code and take out all the lines that you don't REALLY need... LIke who the fuck cares about uniques? After about 6 months of trading on uniques -- I said to myself -- you know this is totally useless information -- FUCK IT -- so I removed all my code for trading on uniques -- I deleted my damn IP database... I've said it a million times -- the term UNIQUES was invented by cheating shavers... All you should care about is RAWS and PRODUCTIVITY. In.cgi $click++; $click++; $click++; out.cgi BAM you are making money... LOL
Ok where are the haters now? Where are the coders to tell me -- NO FLY You need that %.0001 accuracy for trading efficiently... Well fuck all that. We aren't sending a fucking man to the moon -- do some approximations! Who cares if SITE A sends 10512 and SITE B sends 9523!? All you need to know is that SITE A needs more traffic! Look at Choker -- shit he comes up with an ENTIRELY NEW SYSTEM of trading traffic... He should have a patent for that shit just like I should have a patent for my shit... I think his script worked something like this -- I send SITE A 100 hits -- I send SITE B 100 hits -- if A returns more than B, then fuck you B-otch. Trading is like the old programming Game of Life -- it's all about survival man -- let weak trades die -- let the strong trades thrive -- your site should be like a living breathing organism making decisions and generalizations -- not a damn science project doing pointless calculations just so we can look at pretty pie charts all day long.

TheFLY

YEAH DUDE EXACTLY!!!!!!!!!!!!!!!!!!!!!!

Why waste a bazillion CPU cycles with IP's!!!!! Cookies aren't exact -- but as long as you're making money who cares right?

TheFLY

That's my secret -- figure it out yourself ;)

To know if somebody is cheating -- I don't need the IP address of every surfer -- I just need to observe general behaviors -- and this doesn't require huge databases either. In fact -- your main script doesn't even have to be doing cheat calculations all day long! That's wasted CPU time! All you need to do is store the essential behaviors to disk... When you suspect cheating -- run an entirely different program to do cheat calculations and analysis. This way you don't bog down your site.

Wath the movie PI -- you know how he's watching the ticker -and he just feels the flow? Get into the flow man! Feel the traffic!

TheFLY

Anyway I'm not dissing the serious programmers out there -- I have mad respect for guys like Phil that take the time to use real databases instead of using text files like me. But all I do is count clicks so I don't really need a big database... I'm just trying to stir up some controversy -- some intellectual debate... Let's come up with some new ideas -- new ways of trading traffic... Creativity! I hate it when programming is too tedius -- I like short quick programs that make money -- then you can just tinker around with them like little toys... Fun as shit! Got to keep the programming fun or damn it's too much like a job.

HQ

hahahahaha

Ok now back to read the rest of your post...

HQ

My script is VERY similar (I think). I thought I invented some of the shit in my script until I found it in only 2 other scripts that I know of...

Mine uses unique IPs, however, so I do not count 1 hit in and 1,000 clickthrus the same as 800 hits in and 1,000 clickthrus. I do not let the same surfer account for more than ??? hits.

HQ

The more I read your posts the more respect I get for you dude...

TheFLY

Yeah man we got mad skills!!! This is the best job in the world! Writing programs that make money from thin air! It's a dream come true -- shit I just have to kick myself in the ass once and a while -- threads like these get me motivated.

But no programming for me tonight damn I need some beers and pussy in that order.