Using curb with open-uri
Filed in: Ruby, web Add comments
Simple, but nice replacement for open-uri library (which uses rather slow Net::HTTP library) which uses libcurl (via curb).
It’s easy to swap the original one just by replacing ‘require open-uri’ with ‘require curb-openuri’. Should work as the original, even with better defaults (which, of course, can be changed).
February 25th, 2010 at 20:53
wow. I just realized this post was from a year ago. Keep on writing, you’ve got lots of interesting hacks 🙂
Reply
November 10th, 2010 at 13:10
I’m trying integrate remote upload with paperclip (tutorial http://trevorturk.com/2008/12/11/easy-upload-via-url-with-paperclip/)
It’s working with ‘open-uri’.
With require curb-openuri, seems working, but it doesn’t know ‘base_uri’ method or what. Opening (downloading) file is ok – I see it in ifstat command. But paperclip cannot see it – and validation failed.
Anyone can help?
Reply
Roman Reply:
November 10th, 2010 at 13:14
If you can provide me with a minimal setup, I can try to fix this.
Reply
Stefan Huska Reply:
November 10th, 2010 at 13:32
I will try to explain the following:
In model Upload is code which download remote file.
io = open(URI.parse(source_url).to_s)
self.source = io
(has_attached_file :source)
Problem is “io” seems to be nil. Only with curb_openuri. Original open-uri returns file content.
Looking in curl_agent.rb rows 80.-84. Is there any difference how original “open” method returns “io”?
Reply
Roman Reply:
November 10th, 2010 at 13:49
I don’t think that the io is nil. It’s StringIO that’s returned. But the original open-uri includes some mixin, which adds things like base_uri method.
I can fix this, but I really want the testcase to test with.
Roman Reply:
November 10th, 2010 at 15:00
I pushed a commit which should fix this issue: https://github.com/romanbsd/curb-openuri/commit/f0dfd7ef35c3ae013ad332fd6cf8e19e380d0193
Stefan Huska Reply:
November 11th, 2010 at 12:35
Wow! It’s working! Only little problem yet, big files (1GB) causes timeout.
Curl::Err::TimeoutError in UploadsController#create
Timeout was reached
I must find some config option to increase timeout :/ But with normal files it’s working perfect! Soo many faster than original open-uri. Thanks!
Roman Reply:
November 11th, 2010 at 12:37
open(‘http://www.example.com/’, :timeout => 10)
February 27th, 2011 at 21:19
Thanks for the great gem.
How can I get around this error:
FINAL StandardError: Curl::Err::TooManyRedirectsError + Number of redirects hit maximum amount
Can I increase the amount of allowed redirects?
Thanks.
Reply
Roman Reply:
February 27th, 2011 at 23:16
Just pass curb’s :max_redirects => X, e.g.
open(‘http://www.example.com/’, :max_redirects => 10)
Reply
Harris Reply:
February 28th, 2011 at 03:00
Thanks Roman, that worked but now I have a new issue.
After running for a while, I get “Abort Trap”:
curl agent.rb:33: [BUG] Segmentation fault
Have you experienced this or know what it is?
Reply
Roman Reply:
March 6th, 2011 at 13:31
This is something with curb, try to ask the author. Perhaps it runs out of stack…