Skip to main content

Curl HTTP Client

dinke Programming, Software

Frequently, in my daytime job I have to fetch data from various url’s, either by sending get or post request, binding to different IP address etc. Long time I used my own socket based HTTP class, although I wasn’t quite happy with perfomances and various other things with it. I already used curl cli tool (mostly for debugging purposes), but didn’t really liked it’s php api, so I’ve decided to take some spare time and make some kind of oop wrapper for it, which should be easier to use for easy stuff like sending get/post request etc.

I use this class several months since then, and it evolved over time whenever I needed some new feature. Since various of my colleagues found it very usefull (some of them even sent me new methods for it), I’ve decided to put it out for public. Recently I submitted a code to phpclasses.org, and today got confirmation that class is officially approved.

Update 03/01/2007
New version 1.1 released with new features: fetch into file, upload, proxy etc.

Update 15/02/2008
New version 1.2 released with few bug fixes. New features are ability to send post string as string argument in send_post_data method, ability to accept gzipped content, close curl session etc.

Update 25/03/2013
Version 2.0 released. Pretty much the same features as old versions, but with updated code in order to match latest PHP changes. Code will be regularly updated on github.

You can download class and example files directly from gitgub https://github.com/dinke/curl_http_client

Here are few usage examples.

<?php
/**
 * @version 2.0
 * @package dinke.net
 * @copyright © 2013 Lampix.net
 * @author Dragan Dinic 
 */

require_once("curl_http_client.php");

$curl = new Curl_HTTP_Client();

//pretend to be Firefox 19.0 on Mac
$useragent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:19.0) Gecko/20100101 Firefox/19.0";
$curl->set_user_agent($useragent);

//uncomment next two lines if you want to automatically manage cookies
//which will store them to file and send them on each http request
//$cookies_file = "/tmp/cookies.txt";
//$curl->store_cookies($cookies_file);

//Uncomment next line if you want to set credentials for basic http_auth
//$curl->set_credentials($username, $password);

//Uncomment next line if you want to set specific referrer
//$curl->set_referer('http://www.foo.com/referer_url/');

//if you want to send some post data
//form post data array like this one
$post_data = array('login' => 'pera', 'password' => 'joe', 'other_foo_field' => 'foo_value');
//or like a string: $post_data = 'login=pera&password=joe&other_foo_field=foo_value';

//and send request to http://www.foo.com/login.php. Result page is stored in $html_data string
$html_data = $curl->send_post_data("http://www.foo.com/login.php", $post_data);

//You can also fetch data from somewhere using get method!
//Fetch html from url
$html_data = $curl->fetch_url("http://www.foo.com/foobar.php?login=pera&password=joe&other_foo_field=foo_value");

//if you have more than one IP on your server,
//you can also bind to specific IP address like ...
//$bind_ip = "192.168.0.1";
//$curl->fetch_url("http://www.foo.com/login.php", $bind_ip);
//$html_data = $curl->send_post_data("http://www.foo.com/login.php", $post_data, $bind_ip);

//and there are many other things you can do like

//use proxy
//$curl->set_proxy('http://www.proxyurl.com');

//get http response code for last request
//$http_code = $curl->get_http_response_code();

//get last http request duration in sec
//$duration = $curl->get_request_duration();
?>

33 thoughts to “Curl HTTP Client”

  1. Hello,

    I have a requirement where i need to fetch an XMl file from an http request along with some cookies.I used this class and i could save that XML. But after that i require some requests along with the saved cookies and it should be redirected to some html page. How can this be possible ?

    How can i pass the cookies again in the coming requests. Also how can i redirect the data i get to a browser.

    Thanks

  2. I guess use something like this:

    require_once(“curl_http_client.php”);

    $curl = &new Curl_HTTP_Client();

    //pretend to be IE6 on windows
    $useragent = “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)”;
    $curl->set_user_agent($useragent);

    //cookies will be saved in $cookies_file
    //change to other location
    $cookies_file = “/tmp/cookies.txt”;
    $curl->store_cookies($cookies_file);

    //Uncomment next line if you want to set credentials
    //$curl->set_credentials($username, $password);

    //Uncomment next line if you want to set specific referrer
    //$curl->set_referrer(“http://my.referrer.url”);

    $xml_data = $curl->fetch_url(“http://www.foo.com/foo.xml”);

    //parse xml or whatever
    //don’t worry, cookies are saved in $cookies_file location
    //and will be sent on each requests automaticly

    // if page contains http redirection, you will be redirected automaticly
    // if page contains js redirection – you will have
    //to parse js and redirect manually
    $new_html_data = $curl->fetch_url(“http://www.foo.com/foo.html”);

  3. i want to send post-data with serial from my domain to external server to connect a database server, with
    “select serial, domain from server where user = test” for example. what can i do to, to send external server a return call back example : “serial ok”

  4. Thanks.
    I have tried the whole day with different approaches and couldn’t get it right. This was my last try – and it worked. I just trief to automatically log into my WP Blog and change the settings programmatically.
    Thanks for your effort – and providing something that works.
    Cora

  6. Your code works great in my localhost, but it doesnt work on my server…

    it’s like taking so long to connect and it didnt return any response…

    what could be the problem?
    is is the server settings?

  7. It could be that your server doesn’t have curl support? Did you get any error message (turn display_error On to make sure they are displayed). Also, perhaps it is timing out, so make sure timeout param in curl requests is sufficient.

  8. In the next release, you should include a function that closes the curl session, which allows for, among other things, the cookie file to be unlink-able. This is what I added to your curl_http_client class:

    function closeCurl() {
    curl_close($this->ch);
    }

  9. I found an error with cookie handling, when you call obj->store_cookies you only set the COOKIEJAR and not COOKIEFILE so it only stores cookies and never sends them. Took me a while to figure out as I thought I was doing something wrong. This is a great class and I was hoping to help people out in the future this should be

    function store_cookies($cookie_file)
    {
    // use cookies on each request (cookies stored in $cookie_file)
    curl_setopt ($this->ch, CURLOPT_COOKIEJAR, $cookie_file);
    curl_setopt ($this->ch, CURLOPT_COOKIEFILE, $cookie_file);

    }

  10. I found an error with cookie handling, when you call obj->store_cookies you only set the COOKIEJAR and not COOKIEFILE so it only stores cookies and never sends them. Took me a while to figure out as I thought I was doing something wrong. This is a great class and I was hoping to help people out in the future this should be

    function store_cookies($cookie_file)
    {
    // use cookies on each request (cookies stored in $cookie_file)
    curl_setopt ($this->ch, CURLOPT_COOKIEJAR, $cookie_file);
    curl_setopt ($this->ch, CURLOPT_COOKIEFILE, $cookie_file);

    }

    Before I added this line it would only send cookies on subsequent requests. But after requesting a page then submitting a post to the server it wrote out a new cookie file with a new session cookie.

  13. I have problem
    I want to copy 2 image file from one server to another server,but I dont know
    PLZ help me for solution this problem

  14. Found an issue when sending options in the send_post_data function. Wasn’t creating the necessary string with all the options.

    This fixed it for me.

    foreach($postdata as $key=>$value)
    {
    if(is_array($value)){
    // THIS IS IMPORTANT IF YOU ARE SENDING options
    foreach($value as $k=>$v){
    $post_array[] = urlencode($key) . “=” . urlencode($v);
    }

    }else{
    $post_array[] = urlencode($key) . “=” . urlencode($value);
    }
    }

  15. Hello!
    Great work!
    I have a question, why do I get an error
    “Occured in Curl Error number: 22 Error message: The requested URL returned error: 401” when I try to fetch an URL. But when I copy paste it in my browser, it works…

  16. @JCAVARD
    You ‘ve probably forgot to put ‘http://’ part. In browser it works without it, but not in curl 🙂

  17. Hi!! I’ve been using this class for some time without problems and suddenly it’s not working any more. I’ve found out my ISP has switched to Plesk control panel and PHP version plus some security settings have been changed.

    The error I get is this one:

    Warning: curl_setopt() [function.curl-setopt]: CURLOPT_FOLLOWLOCATION cannot be activated when in safe_mode or an open_basedir is set in /var/www/vhosts/xxxx.xxx/httpdocs/xxx/curl_http_client.php on line 74

    How can this be fixed??

    Thanks for your great work!!

    Regards,
    Alberto

  18. @Alberto
    Well, the best way to resolve problem is to ask your ISP to remove safe_mode/open_basedir restriction. You can see more info about those restrictions at http://www.php.net/safe_mode

    As an workaround you can simple comment line in curl_http_client.php which set curl to follow location header:

    curl_setopt($this->ch, CURLOPT_FOLLOWLOCATION, true);

    but in that case any location header sent by server when calling fetch_url/send_post_data methods would be ignored, so class my not work correctly in case such headers are sent in http response.

  20. Hi

    I thnik it is a great class!, thanks, but I’m having a little trouble…after requesting a page (and getting a cookie) then submitting a post to the server sometimes I get the correct information…. other times I get the error web page, any idea what is happening??

    thanks in advance

  21. Sorry but my crystall ball doesn’t work these days. Perhaps you should give me more infos, ie. your code etc. 🙂

  23. Right now you can’t do that by using class, but implementing additional method shouldn’t be a problem.

    Will add it for to-do list and implement it when I get some time.

  24. you are really great..its a superb classs for curl….I worked with another class and I lost 2 days….using your class I got output in 1 hour…thanks..thanks a lot…keep going..:)

  25. Pingback: Блог интернет-разработчика » Curl based HTTP Client vs. Snoopy.
  26. Pingback: Caught in a Web » Curl HTTP Client 1.2
  27. Pingback: Caught in a Web » Curl HTTP Client 2.0

  29. @Carlos Viana
    The best way to track this is by installing some browser extension, personally I use tamper data for firefox. When you click to “Pesquisar” it sends post request with plenty of data that you’d have to manually collect from page before you do actual submission (like __VIEWSTATE etc.).

    I’d write some tutorial about site scrapping once, it’s beyond subject of one blog post 😉

  30. I have tried to use your latest version of Curl HTTP Client v2.0.8 but I get a parse error:

    Parse error: syntax error, unexpected ‘[‘ in /home/mydomain/public_html/vt/vtwsclib/third-party/CurlHttpClient.php on line 45

    This section:

    /**
    * Default curl options
    * (more details about each option: http://www.php.net/manual/en/function.curl-setopt-array.php)
    * @var array
    * @access protected
    */
    protected $config = [
    CURLOPT_FAILONERROR => false, //whether to fail if http response code is >=400
    CURLOPT_FOLLOWLOCATION => true, //whether to follow any ‘Location:..’ header from response
    CURLOPT_AUTOREFERER => true, //whether to automatically set referer for http redirections
    CURLOPT_ENCODING => ‘gzip, deflate’, //The contents of the Accept-Encoding header in curl request
    CURLOPT_SSL_VERIFYPEER => false, //whether to verify ssl peer’s certificate
    CURLOPT_HEADER => false, //whether to add response headers to the output
    CURLOPT_USERAGENT => ‘CurlHttpClient/v2.0’, //default user agent if none is set
    CURLOPT_SSLVERSION => 1, //force cURL to use TLSv1 (prevent it from using SSLv3 ever)
    ];

    Can you help please?

  31. I have gone back to the older version 2, after not getting far using namespaces, but all versions have another problem with:

    public function init()
    {
    //create new curl handle
    $this->ch = curl_init();

    //set options
    curl_setopt_array($this->ch, $this->config);
    }

    I get warning:

    Warning: curl_setopt_array() [function.curl-setopt-array]: CURLOPT_FOLLOWLOCATION cannot be activated when safe_mode is enabled or an open_basedir is set in /home/mydomain/public_html/vt/vtwsclib/third-party/curl_http_client.php on line 73

    Can you help with this as well?

    Thanks

  33. @HRH
    I’m sorry but you should really consider changing your hosting. Having some old PHP version without namespaces and safe mode ON in year 2015 … that’ unacceptable!

Comments are closed.