Fetching Data from a Web Server

ListManager can retrieve data from a remote machine using the httpget mail merge tag. This data must be accessible via http.

NOTE: IMPORTANT NOTE: Using httpget to fetch content for each recipient can significantly slow your ListManager sending speed. Each ListManager node can use httpget to fetch roughly 20,000 pages per hour. Fetching pages blocks other processes, so sending mail speeds will be significantly slower for concurrent mailings that are not using httpget.

If you are fetching a static page, it is strongly recommended that you use the before; httpget command (below) so ListManager fetches the page only once per mailing try.

If you need dynamic content where each possible page is static (such as ads), you should use the before command to fetch the data into Tcl variables and then programmatically choose between the different pages. Doing so will result in much higher sending speeds than fetching the page for each recipient.

%%httpget%%

The syntax of the httpget command is:

%%httpget url (error_mode (timeout (num_retries (sleep_interval (geturl_args))))) %%

The parenthesized expressions are optional, but if you wish to use sleep_interval, all the previous optional parameters (e.g., error_mode, timeout, num_retries) must be specified .

The URL argument can be virtually anything you can pull up with a web browser. The URL may point to a simple text file, some HTML code, or a CGI script.

To use httpget, you must be using ListManager Pro or Enterprise, and Scripting Level must be set to Full.

The arguments in parentheses are optional.

error_mode

If an error is returned by the httpget command, the error_mode setting specifies what is returned. The options are:

nothing (default) - An empty string is returned.

error - Returns the status and error string.

abort - Aborts the mailing.

skip - Causes the current recipient to be skipped or, in message-level (before) tags, aborts the mailing.

timeout

The HTTP request timeout, in milliseconds. The default is 5000 (5 seconds).

num_retries

The number of times ListManager will re-attempt to retrieve the URL. After the retries are exhausted, an error is handled based on the error_mode. The default is 3 retries.

sleep_interval

The number of milliseconds to wait in between retry attempts. The default is 1000 (1 second).

geturl_args

A Tcl list of arguments passed to the http::geturl function after the URL. This parameter should rarely be needed.

Example 1

You keep dozens of ads on your web server. You want them to be inserted into your newsletter using your script. The script's URL is http://yourserver/cgi-bin/ad.pl.

%%httpget "http://yourserver/cgi-bin/ad.pl"%%

Example 2

You have a script which creates a different page for each user, and would like these unique pages inserted for each email message sent.

%%httpget "http://localhost/cgi-bin/getdata.pl?id=[merge members_.emailaddr_]"%%

In the above example, ListManager first performs a substitution to merge in the member’s email address, then retrieves the data at the resulting URL. Note that you must use square brackets and the full mail merge tag, not the short tag. Let’s assume that bob@example.com is the member that ListManager is currently sending to. After substitution, the URL looks like:

http://localhost/cgi-bin/getdata.pl?id=bob@example.com

The nature of the script being called ("getdata.pl" in this case) is to generate some data unique to the member with that email address. This script can do whatever you design it to do, whether that means generating an ad or retrieving billing information.

Programmers have the ability to call a function in the middle of messages and insert whatever it returns right into the message at the point from which it was called.

While the httpget merge tag does not require any specific output from the URL, it is highly recommended that you include a standard 'HTTP/1.0 200 OK' status line, then a 'Content-Type: text/plain', or 'Content-Type: text/html' line followed by an empty line and the content of the text to be inserted. For example, a (Perl) script might have:

print "HTTP/1.0 200 OK\r\n";
print "Content-Type: text/plain\r\n";
print "\r\n";
print "Hello! Please buy our products!";

%%before; httpget%%

Httpget will repeatedly retrieve the data for each individual recipient. If, however, you only need to retrieve the data only once per mailing attempt, you can use the before; httpget tag instead. That is, before the mailing or retry begins, ListManager will pull the URL once, and fill in that same value for all recipients. Since the speed of httpget depends on the speed it can fetch data from your server, using before ; httpget can make your mailing speeds faster.

The syntax of the before httpget tag is:

%%before ; httpget "URL"%%

Note that the httpget will still be run again for each retry.

Example

You keep dozens of ads on your web server. You want them to be inserted into your newsletter, with every recipient getting the same one. The script's URL is http://www.lyris.com/getmailad.pl.

%%before ; httpget "http://www.lyris.com/getmailad.pl"%%

For more information, see Script Execution Points.

%%after; httpget%%

As of ListManager 8.0, the %%after; httpget%% command has been deprecated.

%%httpgetauth%%

The httpgetauth tag works the same as httpget, but there are two additional parameters that allow you to pass the user name and password as authentication headers to the server:

%%httpgetauth url user pass (error_mode (timeout(num_retries(sleep_interval(geturl_args)))))%%

The parenthesized expressions are optional, but if you wish to use sleep_interval, all the previous optional parameters must be specified.

geturl_args: A Tcl list of arguments passed to the http::geturl function after the URL. This parameter should rarely be needed.