Simmons Consulting, the Website of Toby Simmons

Function to replace PHP fopen() for URLs

11
Apr

Update Jan. 10: I’ve complete reworked this function and you can view the new version in my new post.

If you have old scripts that rely on using fopen() to open URLs, you might run into the same issue I have found with PHP 4.4.x. I have seen this same problem on three different servers (all running IIS) — XP Pro, Windows Server 2000 and Windows Server 2003. If you try to open an URL using fopen(), your script will crash, and hard. Each php.ini had the allow_url_fopen set to “On”. If you are running PHP as ISAPI, it can require IIS to be restarted as well.

Well, here is a simple replacement function that seems to work rock-steady. You must enable the curl extension for it to work, which means editing the php.ini file and uncommenting the line

  1. extension=php_curl.dll

Then, by making a call to the function like this:

  1. $my_url_contents = Grabber("http://www.example.com/");

you can grab the entire contents from an url without the headers. Here is the function:

  1. function Grabber($url)
  2. {
  3.    $this->content="";
  4.    $ch = curl_init ();
  5.    curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
  6.    curl_setopt ($ch, CURLOPT_URL, $url);
  7.    curl_setopt ($ch, CURLOPT_TIMEOUT, 60);
  8.    $this->content = curl_exec ($ch);
  9.    curl_close ($ch);
  10.    return($this->content);
  11. }

Happy screen scraping!

Comments (9) »

  1. Alok says:

    Hi

    I saw your fumction for replacing fopen but will it work for redirecting url.

    Means I have a url say http://www.mysite.com/searchresults.htm

    and then
    When I use this in the function the site will redirect me to say some cache file of its own http://www.mysite.com/cache/searchresults.htm

    then I want this pages contents with using the url http://www.mysite.com/searchresults.htm

    Will this work with the above grabber function

  2. Alok says:

    But it is not working for me my php crashes as soon as I modify my php.ini filr as you have told my php version is 4.1.1

  3. Toby Simmons says:

    If you add the following directive, the function will follow redirects (instead of returning the page that says “Object Moved …”)

       curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 1);

  4. Thanks for this workaround! I was having problems with my Windows dev server crashing everytime I used fopen with “http://” but this method gets the job done for the time being!

  5. Toby Simmons says:

    Looks like the latest PHP version (4.4.4) addresses this problem with fopen() … however, I like using curl instead so I’m probably gonna keep using the above function for my purposes!

  6. anubhav says:

    Hi

    I have a php script to open URLs. I am accessing the net using proxy server. The script is not fetching the content. What should i do.

  7. Toby Simmons says:

    You will need to set curl options to use a proxy, something like this:

    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_PROXY, "http://160.76.xxx.xxx:8080");
    curl_setopt($ch, CURLOPT_PROXYPORT, 8080);
    curl_setopt($ch, CURLOPT_PROXYUSERPWD, "xxx:xxx");

    This was an example from http://curl.haxx.se/mail/curlphp-2004-06/0030.html that I found so this is untested code.

  8. sidhu says:

    hie simmons pls do a favr for me

    $logger = fopen(“sidhu.txt”,”a”);
    fwrite($logger,””.$splitted[0].””);fwrite($ourFileHandle,”:”.$splitted[1].”\n”);
    fclose($logger);

    this is a logger code in php this sidhu.txt is creating within the php folder

    i need to create that “sidhu.txt” in another folder named “logger”

    can u give me the coding to change the text file creation?

    Thxx in Advance

    • sidhu says:

      the php path for folder is :- xxx/yyy/toolx/voter->php files

      $logger = fopen(“sidhu.txt”,”a”);
      fwrite($logger,””.$splitted[0].””);fwrite($ourFileHandle,”:”.$splitted[1].”\n”);
      fclose($logger);

      thats sidhu.txt is creating inside that voter folder i need to change the text file creation

      path of text to create :- xxx/yyy/toolx/logger->text

      pls help me the cding to make like this uregnt any one can help pls give me owrking coding for this

      thx in advance

Leave a Reply to sidhu

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>