Retrieving external dynamic pages with variables through PHP

[Odin88]

I have encountered a bump on a project I am currently working on. The project requires a script to retrieve the content of an external page with variables for further processing, and I can't seem to get it to work even when using FSockOpen. This may, or should, be a failure on my part, but I looked the issue up in Google and found that others have had the same problem. Among other things, I found some interesting discussion forum threads, but the answers (if you can call it that) was rather vague and speculative and lead me no closer to a solution.

I know for a fact that what I am seeking to do is possible, some way or the other, but this is the first time I have had to retrieve content from external pages with variables. Any input would be highly appreciated.

Danke schön.

[igneus]

Not sure what you mean by "with variables" .. but for just saving the contents of a webpage to a variable, have you tried "file_get_contents"? Probably the easiest solution.

Could also use fopen and all that crap too

[Nathan]

odin:

simple, do the following:

the external page should look something like this:

&lt;? echo "<?"; ?&gt;
$var = "eek";
$foo = "bar";
$somemore = 1;
&lt;? echo "?>"; ?&gt;

Lets pretend that page would be http://www.fuckyoutoo.com/some.php

you would simply do:
include("http://www.fuckyoutoo.com/some.php"); in your main file that needs the vars.

You can of course also not use those strange echos and print the <? ?> directly in the file, but you have to make sure that the server that file is on will not parse it for php since it'll of course be empty for the include() then.

[IMP^or^SNiTL.e]

bah

[Odin88]

Originally posted by Nathan
odin:

simple, do the following:

the external page should look something like this:

&lt;? echo "<?"; ?&gt;
$var = "eek";
$foo = "bar";
$somemore = 1;
&lt;? echo "?>"; ?&gt;

Lets pretend that page would be http://www.fuckyoutoo.com/some.php

you would simply do:
include("http://www.fuckyoutoo.com/some.php"); in your main file that needs the vars.

You can of course also not use those strange echos and print the <? ?> directly in the file, but you have to make sure that the server that file is on will not parse it for php since it'll of course be empty for the include() then.

You do not quite understand what I am trying to do, but that is my fault I suppose. "Retrieving dynamic pages with variables" can be interpreted many ways. What I am trying to do is actually to fetch the output of dynamic pages such as http://www.google.com/search?hl=en&i...=Google+Search

I appreciate your time and effort though.

[Nathan]

Hey.. emm.

whats the problem about doing that? <G>

$lines = file("http://www.somepage.com?with=vars&foo=bar");

that doesn't work for you?

[Odin88]

Originally posted by Nathan
Hey.. emm.

whats the problem about doing that? <G>

$lines = file("http://www.somepage.com?with=vars&foo=bar");

that doesn't work for you?

There's no need to be condescending. That doesn't work either.

[artware]

Is PHP set up correctly to allow the fopen wrappers? That could be the reason why it doesn't work..?

Read about it here.

[rudeboi]

Originally posted by Odin88


You do not quite understand what I am trying to do, but that is my fault I suppose. "Retrieving dynamic pages with variables" can be interpreted many ways. What I am trying to do is actually to fetch the output of dynamic pages such as http://www.google.com/search?hl=en&i...=Google+Search

I appreciate your time and effort though.

Read up on the google api, and rss.. there are php scripts out
there to let you use the google api as rss/rdf feeds.

I've got one that workds with news.google.com if you need one as an example.

[Brian911]

just use wget

[fiveyes]

Get a Google Api account and then check out http://www.jappc.com/google

[KC]

Yep, Google API is the way to do it..

http://www.google.com/apis/api_faq.html

[KC]

If Google isn't the "dynamic pages" you want to pull variables from, then you'll probably need to do some sort of REG/EX to pull the correct values.

[IMP^or^SNiTL.e]

sexy

[KC]

Originally posted by IMP^or^SNiTL.e
sexy

How much swoit are you raking in a day with that thing?

[venturi]

The only way you can "trap" these variables is if you parse the URI in your php script. If they are hidden or session variables you are basically screwed unless you already know the variable names associated with the page you are attempting to 'scan'.

People that have accomplished what I think you are trying to do have been given explicit permission from the sites they are parsing and have been granted access to some sort of an API that said site(s) have available.

I may not be Completely accurate here but I think that using any kind of TCP socket connection to said web page will not garner you the info you wish to gather.

[rudeboi]

Also, look into CURL .. you can do a lot of
page scraping type work with CURL .. or just go to CPAN.org and search for page scrape and do your page scraping in perl