Home › Forums › General

If Igor could retrieve a web page, what would you do with it?

‹‹ Previous Topic Next Topic ››
JimProuty
 September 12, 2007 - 12:55
Posts: 15
 Joined: 2007-07-19
 Location: United States

I've been playing with an idea about adding to Igor the ability to retrieve the HTML of a web page into a string (probably) or a text wave (maybe). And probably a way to use HTTP to download a file, too, similar to FTPDownload.

So, two commands:

String text= GetHTTPText "http://www.whatever.com/file.html" // returns the HTML
 
HTTPDownload "http://www.whatever.com/file.dmg", "/Users/Me/Documents" // downloads the file to the directory

(Note: BrowseURL does something different than HTTPDownload: BrowseURL tells your default web browser to open a URL. GetHTTPText and HTTPDownload wouldn't invoke a web browser.)

It seems like a good starting point for adding more "web-savvy" to Igor.

If done, what would *you* do with a feature like this?

Software Engineer, WaveMetrics, Inc.
andyfaff September 12, 2007 - 16:19
Posts: 60
 Joined: 2007-09-12
 Location: Australia

Hmm,
some aspects might be interesting. On out neutron scattering instruments we have Data Acquisition servers that control all our detector banks. The public face of these is a webpage which we can interact with. One of the buttons on the webpage tells the computer to save a local copy of the data. This is saved as an HDF file that can then opened into IGOR.

To do the pressing of the buttons without using the webpage I use CURL (with a script), from the DOS command line. This is turn is controlled from IGOR, via ExecuteScriptText.

My wishlist during this period has been that some aspects of curl could be integrated into IGOR, so that I could press buttons, etc, and get the results.

However, this practice is somewhat of a stopgap... Some of our datafiles are 180 Mb in size, which takes a while to transfer, even over 100Mbit/s networks. I would also like IGOR to be able to perform TCP/IP socket communication (faster than file transfer). Some people out there have written some XOP code already, but this has to be adapted everytime.
aclight September 12, 2007 - 20:17
Posts: 246
 Joined: 2007-03-01
 Location: United States

JimProuty wrote:

If done, what would *you* do with a feature like this?

One possibility would be to do what I mention at http://www.igorexchange.com/node/175#comment-43

Basically, it's possible (though not currently enabled) to query the IgorExchange server to find out the most recent release of a project and various other information about it. The output is XML which could potentially be processed with Jeff Weimer's XML processing procedures. As is, I think the only potential way to do this automatically would be for the server to periodically write these XML files and then use FTPDownload to download the file. But being able to just grab the XML via GetHTTPText would be a lot easier and more flexible.

Of course other types of XML data feeds would be fetchable in a similar way, which I'm sure could be useful to many.

It would also be nice to be able to do extended HTTP connections and send data via GET and POST requests. SSL would be cool too but would probably involve a lot more work on your part. I'm thinking of the functionality of the Unix tool wget here. It might be possible for one to use ExecuteScriptText to call wget from within Igor and pipe the output of that to a file that is then opened within Igor.

Adam
andyfaff September 13, 2007 - 01:07
Posts: 60
 Joined: 2007-09-12
 Location: Australia

wget is pretty similar to curl, they do almost exactly the same thing.
hrodstein
 September 13, 2007 - 13:26
Posts: 93
 Joined: 2007-06-21
 Location: United States

JimProuty wrote:
I've been playing with an idea about adding to Igor the ability to retrieve the HTML of a web page into a string (probably) or a text wave (maybe). And probably a way to use HTTP to download a file, too, similar to FTPDownload.

I wonder what could be done with web services. I don't know enough to answer the question but I do think Igor should have more web-interaction capability.
tony September 14, 2007 - 11:23
Posts: 13
 Joined: 2007-08-28
 Location: United States

I would like to be able to do file transfer through an ssh tunnel (scp) from within Igor. Presumably on the mac scp is part of OSX, but on the pc one has to install a command line ssh client (eg putty) and invoke that through ExecuteScriptText. The built in FTPUpload and FTPDownload are of little use now that everyone requires secure server connections. When I use Igor for data acquisition I use ExecuteScriptText to run a batch file to sequentially zip and scp data to a web server using a private key, which satisfies the security requirements for the systems administrator. When I distribute the code I need to include open source code for command line compression, transfer and mail (7-zip, putty and blat for the pc), and making those things work cross platform is a pain.

I've been toying with the idea of controlling igor remotely by putting something on a server that Igor can check periodically. GetHTTPText could simplify that sort of thing. If I ever have the nerve to do this, it'll be done via the web server because I don't want to open up the computer that runs Igor.
Shaun_Roe January 19, 2008 - 13:09
Posts: 4
 Joined: 2008-01-04
 Location: Switzerland

JimProuty wrote:
I've been playing with an idea about adding to Igor the ability to retrieve the HTML of a web page into a string (probably) or a text wave (maybe). And probably a way to use HTTP to download a file, too, similar to FTPDownload.

So, two commands:

String text= GetHTTPText "http://www.whatever.com/file.html" // returns the HTML
 
HTTPDownload "http://www.whatever.com/file.dmg", "/Users/Me/Documents" // downloads the file to the directory

(Note: BrowseURL does something different than HTTPDownload: BrowseURL tells your default web browser to open a URL. GetHTTPText and HTTPDownload wouldn't invoke a web browser.)

It seems like a good starting point for adding more "web-savvy" to Igor.

If done, what would *you* do with a feature like this?

Could be quite a lot, if you use some standard tools...we are all supposed to be using XHTML these days so I would gear it towards using XML as the basic format, perhaps running any HTML through a validator and/or a utility like htmlTidy or xmllint to make it well formed. Then presumably you want to extract data. For that I would use an XSLT template and add an XSL engine in Igor to output the data to waves. Clearly the user has to know something about the format to be able to do this. (With formats which are not too complex it should be possible to parse directly, or generate the XSL template on-the-fly with a little user input).
And the use? Well (at CERN) we already have web pages and web services serving things like temperature and environmental conditions for our detectors, this is gradually being extended so our databases have web interfaces which return xml.
Some databases (Oracle) can generate XML directly from SQL and have a standard format, so it would be possible to write a standard XSLT which would be pretty flexible.
jjweimer January 21, 2008 - 21:22
Posts: 241
 Joined: 2007-08-14
 Location: United States

Shaun_Roe wrote:

... so it would be possible to write a standard XSLT which would be pretty flexible.

Out of curiousity, why build an XSLT engine into Igor Pro when both MacOS and WinXX have free, well-developed XSLT engines already available?

Or are you just saying to use a standard XSL transform????

--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAH


[ last edited January 21, 2008 - 21:23 ]
Shaun_Roe January 25, 2008 - 11:16
Posts: 4
 Joined: 2008-01-04
 Location: Switzerland

jjweimer wrote:
Shaun_Roe wrote:

... so it would be possible to write a standard XSLT which would be pretty flexible.

Out of curiousity, why build an XSLT engine into Igor Pro when both MacOS and WinXX have free, well-developed XSLT engines already available?

Or are you just saying to use a standard XSL transform????

--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAH

I mean write an XSL transform, and hopefully use the in-built xslt processor
RGerkin February 22, 2008 - 14:22
Posts: 19
 Joined: 2008-02-22
 Location: United States

At the moment, I can't think of anything scientific that I would do with it, but I would use it to scrape data from databases on the web. I do this now with PHP and curl, and then parse out the good stuff, and then put that info into Igor for visualization. A direct GetHTTP operation would save me a step.

In general, more network capability would be desirable. My friend is working on a networking XOP as we speak, although it's not his top priority right now.

Rick
andyfaff March 4, 2008 - 21:49
Posts: 60
 Joined: 2007-09-12
 Location: Australia

A networking XOP would be manna to me. At the moment we have an instrument control server that runs an experiment. But then I have to transfer data manually to IGOR to analyse, before returning to type in the results by hand. If I had an XOP to take care of the connection things would be so much simpler.

Andy
WhereIsMyData March 4, 2008 - 22:03
Posts: 1
 Joined: 2008-03-04
 Location: United States

All the synchrotrons post their status on a web page, e.g. here:
http://www.aps.anl.gov/aod/blops/status/srStatus.html
One could get information from here such as the ring curent when saving data.

One could also track stocks for viewing in Igor.
andyfaff March 9, 2008 - 18:17
Posts: 60
 Joined: 2007-09-12
 Location: Australia

An XOP is now available for dealing with URLS:
http://www.igorexchange.com/project/easyHttp

It is based on the LIBCURL libraries. You can get webpages, perform ftp downloads, get files from other filesystems, etc.

e.g.

 
//get a webpage and put it into the S_gethttp string
easyHttp "www.wavemetrics.com"
print S_gethttp
//save into a file
easyHttp/File="C:/file.dmg" "http://www.whatever.com/file.dmg"
//get a remote file over your network and put it into S_getHttp
easyhttp "file:///C:/Program%20Files/WaveMetrics/Igor%20Pro%20Folder/Igor%20Extensions/About%20Igor%20Extensions.txt"

Future plans include authentication + FORM/POST.


[ last edited March 9, 2008 - 18:19 ]
‹‹ Previous Topic Next Topic ››

Back to top