Addons/web/gethttp
User Guide | Installation | Development | Categories | Git | Build Log
web/gethttp - Retrieve web pages/files from the Internet using Wget or cURL.
- Provides simple interface to external utilities for retrieving URL contents from the Internet
- Uses Wget on Windows and Linux, cURL on Mac
- Write URL contents to session or file
- Ability to provide full command line options to the utilities if desired
Browse history, source and examples using github.
Verbs available
urlencode v Encode string as valid url gethttp v Retrieve URI using Wget/cURL tools
Installation
Use JAL/Package Manager.
Load web/gethttp addon with the following line:
load 'web/gethttp'
Check it is working OK:
load 'web/gethttp/test/test_gethttp' test_gethttp passed
If you are behind a proxy server you will need to configure Wget/cURL accordingly, but you have probably already done this so as to get Package Manager to work for you.
Examples of use
Retrieve Jsoftware homepage to the session:
homepagehtml=: gethttp 'http://www.jsoftware.com' 106 {. 306}.homepagehtml <div id="Header"> <div id="Logo"> <a href="./zippy.htm"><img alt="JSoftware" src="./jsoftwarea.gif"></a>
Retrieve Zippy image and write it to current folder:
'file' gethttp 'http://www.jsoftware.com/zippy.gif' --11:28:50-- http://www.jsoftware.com/zippy.gif => `zippy.gif' Resolving server.domain.com... 123.123.123.123 Connecting to server.domain.com|123.123.123.123|:8080... connected. Proxy request sent, awaiting response... 200 OK Length: 43,143 (42K) [image/gif] 0K .......... .......... .......... .......... .. 100% 53.52 KB/s 11:28:51 (53.52 KB/s) - `zippy.gif' saved [43143/43143]
Retrieve Zippy image and write it to the ~temp folder as cartoon.gif:
('file';jpath '~temp/cartoon.gif') gethttp 'http://www.jsoftware.com/zippy.gif' --11:30:25-- http://www.jsoftware.com/zippy.gif => `C:/Users/thisuser/Documents/j602-user/temp/cartoon.gif' Resolving server.domain.com... 123.123.123.123 Connecting to server.domain.com|123.123.123.123|:8080... connected. Proxy request sent, awaiting response... 200 OK Length: 43,143 (42K) [image/gif] 0K .......... .......... .......... .......... .. 100% 14.93 MB/s 11:30:25 (14.93 MB/s) - `C:/Users/thisuser/Documents/j602-user/temp/cartoon.gif' saved [43143/43143]
Use custom options for external utility.
This is a Wget example that retrieves content from stdout and saves the log to file:
106 {. 306}. ('-O - -o ',jpath '~temp/gethttp.log') gethttp 'http://www.jsoftware.com' <div id="Header"> <div id="Logo"> <a href="./zippy.htm"><img alt="JSoftware" src="./jsoftwarea.gif"></a>
See test_gethttp.ijs for more usage examples.
Documentation
gethttp
gethttp (v) Retrieve URI using Wget/cURL tools form: [option] gethttp uri result: depends on options, Default is URI contents y is: URI to retrieve x is: Optional retrieval options. One of: 'stdout' (Default) 'help' 'file' or ('file';jpath '~temp/myfile.htm') Anything else is assumed to be a valid Wget/cURL option string eg: 'file' gethttp 'http://www.jsoftware.com' eg: ('-o - --stderr ',jpath '~temp/gethttp.log') gethttp 'http://www.jsoftware.com'
Known Issues
When retrieving a binary file to the session on Windows for example: myimg=: gethttp 'http://www.mydomain.com/myimage.jpg' , any LF characters in the file will be replaced by CRLF in the session output.
A workaround is to retrieve the binary file and write to disk for example: 'file' gethttp 'http://www.mydomain.com/myimage.jpg' and then read the file from disk into the session.
Author
See Also
- Scripts/HTTP Get is a J sockets solution for retrieving files from the Internet.