Speed up grab data from remote site in php

Ask Question Forum:

Model Library:2025-02-08 Updated:A.I. model is online for auto reply question page

Show

ASK RECENT

Speed up grab data from remote site in php

Asked by duncanb7

at 2024-04-26 23:09:32

Point:500 Replies:6 POST_ID:828487USER_ID:11059

Topic:

Hypertext Markup Language (HTML);PHP Scripting Language;Asynchronous Javascript and XML (AJAX)

I am using loadHTMLFile() to load remote site html page and do data extract by getelementsByName for
300 interation looping. The speed to grab the data for each time is really slow. So I try
to think whether I am using right function to do so.
There is alternative such as file_get_contents(url) and curl_setpos query. I want to know
Which is fastet method to grab 2 or 3 data from the remote site data for every iteration?
Why I have seen people sometimes using loadHTMlFile(), sometimes using curl_setpos, sometimes using
file_get_contents, what is different in term of program speed ?

If I switch back to file_get_contes($url), I need to do a loop to do a match to my data that will
cost execution time that is similar slow to getelementbtTagName by DOM, Is it Right ?

I also look curl_ method at http://www.phpfreaks.com/forums/index.php?topic=324412.0
but it doesn't tell me the advatnage of using curl_ . Could you provide this info ?

Please advise

Duncan

<?php$dom = new DOMDocument();/*$dom = file_get_contents($url2);*/@$dom->loadHTMLFile("http://www.remotesite.com/");$data3= $dom->getElementsByTagName('table')->item(4)->getElementsByTagName('div')->item(8)->getElementsByTagName('span')->item(0)->nodeValue;$data4= $dom->getElementsByTagName('table')->item(4)->getElementsByTagName('div')->item(12)->getElementsByTagName('span')->item(0)->nodeValue;?>

1:2:3:4:5:6:7:

Select All Code

Author: duncanb7 replied at 2024-04-27 22:23:27

Thanks for all of your reply
it is solved the issue by
this EE thread at
http://www.experts-exchange.com/Web_Development/Web_Languages-Standards/PHP/Q_26980898.html

Assisted Solution

Expert: Medo3337 replied at 2024-04-27 12:35:01

167 points GOOD

http://php.net/manual/en/book.curl.php
http://www.higherpass.com/php/Tutorials/Using-Curl-To-Query-Remote-Servers/

Author: duncanb7 replied at 2024-04-27 10:47:42

Ray_Paseur, we discuss curl here and I ask this in other thread.
So thsi thread might be going to test which method is the best,
it seems from your code in other thread, curl is more flexible
and down into detail,
Give me sometime to digest all the attach link

Expert: Medo3337 replied at 2024-04-27 07:57:42

If you want faster way, I recommend you to get faster internet for the server, this is the most effective way.

Assisted Solution

Expert: Ray Paseur replied at 2024-04-27 06:27:13

166 points GOOD

I cannot tell you which method of getting the remote content will be fastest, but I can give you a tool to measure the time of each different method. With a little instrumentation you can find the best method. However you probably want to test several times on different URLs - the remote server may cache the requests causing second and subsequent requests to look fast, when in practice they may not be very fast.

Try using this class and see what it tells you. HTH, ~Ray

<?php // RAY_oop_stopwatch.phperror_reporting(E_ALL);// DEMONSTRATE A SCRIPT TIMER FOR ALL OR PART OF A SCRIPT PHP 5+// MAN PAGE http://php.net/manual/en/function.microtime.phpclass StopWatch{    protected $a, $z;    public function __construct()    {        $this->a = array();        $this->z = array();    }    // A METHOD TO CAPTURE A START TIME    public function start($name='TIMER')    {        $this->a[$name] = microtime(TRUE);    }    // A METHOD TO CAPTURE AN END TIME    public function stop($name='ALL')    {        if ($name == 'ALL')        {            foreach ($this->a as $name => $start_time)            {                if (!isset($this->z[$name])) $this->z[$name] = microtime(TRUE);            }        }        else        {            $this->z[$name] = microtime(TRUE);        }    }    // A METHOD TO READ OUT THE TIMER(S)    public function readout($m=1000, $eol=PHP_EOL)    {        $str = NULL;        foreach ($this->a as $name => $start_time)        {            $str .= $name;            if (!isset($this->z[$name]))            {                $str .= " IS STILL RUNNING";            }            else            {                $lapse_time = $this->z[$name] - $start_time;                $lapse_msec = $lapse_time * $m;                $lapse_echo = number_format($lapse_msec, 1);                $str .= " $lapse_echo";            }            $str .= $eol;        }        return $str;    }}// DEMONSTRATE THE USE -- INSTANTIATE THE STOPWATCH OBJECT$sw  = new Stopwatch;// SET STOPWATCH NAMES$go = 'GOOGLE ONLY';$gy = 'GOOGLE AND YAHOO!';$yo = 'YAHOO! ONLY';// START SOME TIMERS$sw->start($go);$sw->start($gy);// PERFORM SOME ACTIVITY THAT YOU WANT TO TIME$page = 'http://google.com';$html = file_get_contents($page);// STOP ONE OF THE STOPWATCHES AND START THE OTHER$sw->stop($go);$sw->start($yo);// PERFORM SOME OTHER ACTIVITY THAT YOU WANT TO TIME$page = 'http://yahoo.com';$html = file_get_contents($page);// REPORT THE STOPWATCHES CONTENT (TWO WILL BE INCOMPLETE)echo nl2br($sw->readout());// STOP ALL OF THE REMAINING STOPWATCHES$sw->stop();// REPORT THE STOPWATCHES CONTENT AGAINecho nl2br($sw->readout());

1:2:3:4:5:6:7:8:9:10:11:12:13:14:15:16:17:18:19:20:21:22:23:24:25:26:27:28:29:30:31:32:33:34:35:36:37:38:39:40:41:42:43:44:45:46:47:48:49:50:51:52:53:54:55:56:57:58:59:60:61:62:63:64:65:66:67:68:69:70:71:72:73:74:75:76:77:78:79:80:81:82:83:84:85:86:87:88:89:90:91:92:93:94:95:96:

Select All Code

Accepted Solution

Expert: bportlock replied at 2024-04-27 02:01:17

167 points GOOD

Some general comments:

1. When grabbing data from remote websites you are limited by the speed at which the remote site responds. Increasing the speed at this end may make no difference if the remote site is a slow one.

2. cURL's main advantage over the other file methods is that cURL allows you to emulate a browser including cookies, sessions, https, ftp, gopher, etc. It is much, much more flexible than the other methods and whilst it looks more cumbersome, it is not that bad to use. Personally I never use anything except cURL to fetch HTML.

3. Use microtime to set up timing points in your code and see where the time is being used up. See Example 1 on this page http://uk2.php.net/microtime

"If I switch back to file_get_contes($url), I need to do a loop to do a match to my data that will
cost execution time that is similar slow to getelementbtTagName by DOM, Is it Right ?"

To an extent you are correct, but no matter how you fetch the page you will have to spend time processing and on most modern servers you can disregard the processing time because machines these days are fast and capable unless you use really, really cheap hosting.

DomDocument also supports loadHTML which loads a STRING rather than fetching a FILE so you could still load it into DOM http://uk2.php.net/manual/en/domdocument.loadhtml.php

In any case, insert timing points and see where the time is being spent. Then you'll have a better idea what the problem is.

新聞財經娛樂♬資訊 News Economy

About Us|Service|Pricing

Stock Index全球即時指數期貨:

&nbps;

News即時新聞:↓

News即時新聞:↑

On Balance Volume (OBV):

Youtube Video♬ Sharing by members:↓

Youtube Video♬ Sharing by members:↑

Ask Question Forum:

Speed up grab data from remote site in php

Your Information

Members/Profolio