EIW Fall 2000 Lecture Notes - Perl/CGI

CGI libraries

In general, all CGI programs share a set of common operations:

Get the query string that was sent by the browser. This includes determining what type of METHOD was used to submit the query (GET or POST) and actually getting the query string.
URL decoding of the query and extracting name,value pairs.
Generating HTML

There are a couple of widely used libraries (perl modules or packages) that support these operations:

The Perl CGI module is included with the perl interpreter. The CGI support is provided via an object-oriented interface (Perl has support for object-oriented programming!) and includes great support for generation of HTML. We won't look at this module (we aren't going to cover Perl object-oriented programming), but there is documentation available at http://www.activestate.com/ActivePerl/docs/lib/CGI.html. You can also find lots of perl resources at www.perl.org.
The cgi-lib perl library is a set of perl functions that are available as a library from http://cgi-lib.stanford.edu/cgi-lib/. This library contains good support for reading CGI input (the query), but support for HTML output is minimal. cgi-lib.pl does include some routines for generating the HTTP header necessary and for generating the HEAD and BODY tags, but there is no support for generating HTML (you have to use print statements).

You can use any perl library you want, the examples and lecture notes will use our own library to support CGI programming (based roughly on the cgi-lib library). Below we derive the functions included in "eiw-cgi.pl", the perl library we will use for the example CGI programs.

Building a CGI library for perl - `eiw-cgi.pl`

Our library will include the following functionality:

Get query string
URL Decoding
extraction of name/value pairs from the query
HTTP header generation
creation of HTML that provides error messages to the user (to tell the user something was wrong with the query)

Reading the query string

Recall that the query can be provided via two mechanisms:

METHOD=GET. The query is in the environment variable named "QUERY_STRING".
METHOD=POST. The query is coming from STDIN and the length of the query string is contained in the environment variable "CONTENT-LENGTH".

To read an environment variable in perl, you can simply use the associative array named %ENV. The index for each member of the associative array is the environment variable name, so to get the value of the environment variable "REQUEST_METHOD" you can simply do this:

$method = $ENV{"REQUEST_METHOD"};

. We now know everything we need to know to be able to write the code that gets the query string for a GET request:

# 

# First get the request method
$method = $ENV{'REQUEST_METHOD'};

if ($method eq "GET" ) {        # NOTICE I USE "eq" NOT "==" !!!!!!
    $request = $ENV{'QUERY_STRING'};
}

If the request method is "POST" we need to read the request from standard input, and we need to know how much to read. We can't read until EOF since we won't ever see EOF ! (stdin is actually connected back to the web server and the server won't ever tell us we've reached the end of the data). Perl has a read command that we can tell how many bytes (characters) we want to read:

# First get the request method
$method = $ENV{'REQUEST_METHOD'};

if ($method eq "POST" ) {        # NOTICE I USE "eq" NOT "==" !!!!!!
    $size = $ENV{'CONTENT_LENGTH'};
    read(STDIN,$query,$size);    # read $size bytes in to $query
}

Splitting up the query and URL-decoding

If the CGI program expects the query to be generated by an HTML form, the query string is a URL-encoded sequence of name/value pairs. Each name/value pair is represented with the syntax name=value and the sequence of these strings are glued together with the &character.

We must split the query string up in to name/value pairs before attempting to do any URL-decoding, otherwise we might end up introducing a new & or = that would confuse the splitting process later. To split the query string on the & character we can just do this:

#  $query holds the query string

@pairs = split("&",$query);

# @pairs is now an array that holds a string like "name=value" in each
# array location.

We can now work on each of the name/value pairs in @pairs by splitting on the "=" character:

# @pairs is an array that holds a string like "name=value" in each
# array location.

foreach $i (@pairs) {
    ($name,$value) = split("=",$i);
}

We obviously need to do a little more to save each name/value pair some place we can get at them later. Before we worry about this we need to do URL-decoding on each $name and on each $value.

URL decoding

Recall that each part of the query string is URL-encoded by the browser before being submitted to server as part of an HTTP request. This encoding ensures that there are no spaces in the query, and also makes sure that the name/value pairs can be recognized easily (so if the user enters something like "hi=dave" in a form we can't get something like "name=hi=dave"). The encoding is done as follows:

Replace all spaces with the character "+".

Replace all special characters (punctuation, including +, & and =) with the hex equivalent. The hex equivalent is a 3 character sequence that starts with the "%" character followed by 2 hexadecimal digits that represent the ASCII value of the character. For example:

Character	Hex equiv.
`=`	`%3D`
`+`	`%2B`
`&`	`%26`
`%`	`%25`

Given a perl variable named $string that we want to URL-decode, we can take care of the first part easily:

# $string is a URL-encoded string, we want to replace all "+" with " "

$string =~ s/\+/ /g;

Taking care of all the hex equivalent codes is a little harder. We could build a set of substitute commands that takes care of each possible hex-encoded character, but we can also use the perl pack command to build a single character from 2 hex digits. For more information on the pack command you need to refer to a Perl reference. Here is a single substitute command that will take care of all hex-encoded characters in a string (will replace each with the character encoded):

$string is a URL-encoded string that has already had the "+"s replace with spaces.

$string =~ s/%([0-9a-fA-F][0-9a-fA-F])/pack("c",hex($1))/eg;

The e modifier at the end of the substitute command means "executable", so perl treats the replacement text as a perl expression that should be evaluated to determine the replacement text. This means that the replacement text for each substitution is the value of the following expression:

pack("c",hex($1));

$1 is set by perl to be whatever part of the string is matched by the first parenthesized expression in the regular expression (the same thing as \1).

URL-decoding subroutine

It makes sense to create a subroutine that can do url-decoding on a string (returning the url-decoded string). Here goes:


# subroutine that does URL decoding
sub urldecode {
    my($string) = $_[0];

# convert all '+' to ' '
    $string =~ s/\+/ /g;    

# Convert %XX from hex numbers to ASCII 

    $string =~ s/%([0-9a-fA-F][0-9a-fA-F])/pack("c",hex($1))/eg; 

    return($string);  # don't really need this return with perl!
                      # by default the subroutine will return the
                      # value of the last expression.
}

We can now go back and add calls to this subroutine each time we extract a name/value pair:

#  $query holds the query string

@pairs = split("&",$query);

# @pairs is now an array that holds a string like "name=value" in each
# array location.

foreach $i (@pairs) {
    ($name,$value) = split("=",$i);

    #url-decode each string
    $name = urldecode($name);
    $value = urldecode($value);
}

We have now constructed each of the strings we need, however we still haven't save this information anywhere (as the code stands now - we would just end up being able to tell what the last $name and $value were). One way to save the name/value pairs is in an associative array - this way we can get at each $value as long as we know what the field $name was.

foreach $i (@pairs) {
    ($name,$value) = split("=",$i);

    #url-decode each string
    $name = urldecode($name);
    $value = urldecode($value);

    # add to the associative array named %fields
    $fields{$name} = $value;
}

The above code modifies the associative array %fields so that we can later get at any value using something like $val = $fields{"Age"}; (this would get the value of the field "Age").

Put the whole mess in one subroutine

Nearly every CGI program needs to get the query and convert it in to a set of url-decoded name/value pairs, so it makes sense to hide the details inside a subroutine. Here is a subroutine named GetQuery that will find out where the query string is, get it, split it into name/value pairs and take care of URL-decoding. The return value of this subroutine is an associative array that will contain all the name/value pairs found in the query:


# a subroutine that gets the query and return an 
# associative array holding the name/value pairs

sub GetQuery {
   local($query,$size,$method,$name,$value);
   local(%fields);

   # First get the request method
   $method = $ENV{'REQUEST_METHOD'};

   if ($method eq "GET" ) {        # NOTICE I USE "eq" NOT "==" !!!!!!
       $query = $ENV{'QUERY_STRING'};
   } else {
      # assume the method is a POST
      $size = $ENV{'CONTENT_LENGTH'};
      read(STDIN,$query,$size);    # read $size bytes in to $query
   }

   #  $query holds the query string - split on "&"
   @pairs = split("&",$query);

   # @pairs is now an array that holds a string like "name=value" in each
   # array location.

   foreach $i (@pairs) {
      ($name,$value) = split("=",$i);

      #url-decode each string
      $name = urldecode($name);
      $value = urldecode($value);

      # add to the associative array named %fields
      $fields{$name} = $value;
   }

   # return the associative array
   return %fields;
}


# ============================================
# subroutine that does URL decoding
sub urldecode {
    my($string) = $_[0];

    # convert all '+' to ' '
    $string =~ s/\+/ /g;    

    # Convert %XX from hex numbers to ASCII 

    $string =~ s/%([0-9a-fA-F][0-9a-fA-F])/pack("c",hex($1))/eg; 

    return($string);  # don't really need this return with perl!
                      # by default the subroutine will return the
                      # value of the last expression.
}

An example CGI that uses `GetQuery`

Here is a sample perl program that simply prints out the query as a series of name/value pairs formatted in a table. This CGI program uses the GetQuery subroutine we just wrote, so we don't show that code here.

#!/perl/bin/perl
#
# This CGI program creates a table from the name/value pairs found in 
# the query.

# get the name/value pairs in the assoc. array named %fields

%fields = GetQuery();

# print out HTTP header
print "Content-type: text/html\n\n";

# Print out the query as name/value pairs

print "<H3>Your request contained the following name/value pairs</H3>:\n";

print "<TABLE><TH>Name</TH><TH>Value</TH></TR>\n";

foreach $i (keys %fields) {
     print "<TR><TD>$i</TD><TD>$fields{$i}</TD></TR>\n";
}

print "</TABLE>\n";