Using curb with open-uri

Filed in: Ruby, web Add comments

Simple, but nice replacement for open-uri library (which uses rather slow Net::HTTP library) which uses libcurl (via curb).
It’s easy to swap the original one just by replacing ‘require open-uri’ with ‘require curb-openuri’. Should work as the original, even with better defaults (which, of course, can be changed).

http://github.com/romanbsd/curb-openuri/tree

12 Responses to “Using curb with open-uri”

  1. wynst Says:
    February 25th, 2010 at 20:53

    wow. I just realized this post was from a year ago. Keep on writing, you’ve got lots of interesting hacks :)

    Reply

  2. Stefan Huska Says:
    November 10th, 2010 at 13:10

    I’m trying integrate remote upload with paperclip (tutorial http://trevorturk.com/2008/12/11/easy-upload-via-url-with-paperclip/)

    It’s working with ‘open-uri’.

    With require curb-openuri, seems working, but it doesn’t know ‘base_uri’ method or what. Opening (downloading) file is ok – I see it in ifstat command. But paperclip cannot see it – and validation failed.

    Anyone can help?

    Reply

    Roman Reply:
    November 10th, 2010 at 13:14

    If you can provide me with a minimal setup, I can try to fix this.

    Reply

    Stefan Huska Reply:
    November 10th, 2010 at 13:32

    I will try to explain the following:

    In model Upload is code which download remote file.

    io = open(URI.parse(source_url).to_s)
    self.source = io

    (has_attached_file :source)

    Problem is “io” seems to be nil. Only with curb_openuri. Original open-uri returns file content.

    Looking in curl_agent.rb rows 80.-84. Is there any difference how original “open” method returns “io”?

    Reply

    Roman Reply:
    November 10th, 2010 at 13:49

    I don’t think that the io is nil. It’s StringIO that’s returned. But the original open-uri includes some mixin, which adds things like base_uri method.
    I can fix this, but I really want the testcase to test with.

    Roman Reply:
    November 10th, 2010 at 15:00

    I pushed a commit which should fix this issue: https://github.com/romanbsd/curb-openuri/commit/f0dfd7ef35c3ae013ad332fd6cf8e19e380d0193

    Stefan Huska Reply:
    November 11th, 2010 at 12:35

    Wow! It’s working! Only little problem yet, big files (1GB) causes timeout.

    Curl::Err::TimeoutError in UploadsController#create
    Timeout was reached

    I must find some config option to increase timeout :/ But with normal files it’s working perfect! Soo many faster than original open-uri. Thanks!

    Roman Reply:
    November 11th, 2010 at 12:37

    open(‘http://www.example.com/’, :timeout => 10)

  3. Harris Says:
    February 27th, 2011 at 21:19

    Thanks for the great gem.
    How can I get around this error:

    FINAL StandardError: Curl::Err::TooManyRedirectsError + Number of redirects hit maximum amount

    Can I increase the amount of allowed redirects?

    Thanks.

    Reply

    Roman Reply:
    February 27th, 2011 at 23:16

    Just pass curb’s :max_redirects => X, e.g.
    open(‘http://www.example.com/’, :max_redirects => 10)

    Reply

    Harris Reply:
    February 28th, 2011 at 03:00

    Thanks Roman, that worked but now I have a new issue.
    After running for a while, I get “Abort Trap”:

    curl agent.rb:33: [BUG] Segmentation fault

    Have you experienced this or know what it is?

    Reply

    Roman Reply:
    March 6th, 2011 at 13:31

    This is something with curb, try to ask the author. Perhaps it runs out of stack…

Leave a Reply

Pages

Adsense