:TITLE:  Invoking Subprocesses from Tcl - exec, open
;#
;# RCSID: $Header: /home/cvs/TclTutor3/lesson/Tcl/Tcl26.lsn,v 1.1.1.1 2007/09/15 13:30:24 clif Exp $
;# Copyright (c) 1995 Clif Flynt
;# 9300 Fleming Rd.
;# Dexter, MI  48130
;# clif@cflynt.com
;# See file "NOTICE" for licensing terms.
;#
::CMD:: if {([info exists tcl_platform])} {
  switch $tcl_platform(platform) {
    unix        {
                set Tutor(lsn.codeMod) {
		    {regsub -nocase wish [info nameofexecutable] tclsh largv0 }
                    {regsub TCL_INTERP $line $largv0 line}
                    {regsub TEMPDIR $line "/tmp" line}
		    {regsub "wish" $line "tclsh" line}
                }
        }
    windows     {
                set Tutor(lsn.codeMod) {
		    {regsub -nocase wish [info nameofexecutable] tclsh largv0 }
                    {regsub TCL_INTERP $line [file attributes $largv0 -shortname] line}
                    {regsub TEMPDIR $line "C:/temp" line}
		    {regsub "wish" $line "tclsh" line}
                }         
        }
    mac         {
                set Tutor(lsn.codeMod) {
		    {regsub -nocase wish [info nameofexecutable] tclsh largv0 }
                    {regsub TCL_INTERP $line $largv0 line}
		    {regsub "wish" $line "tclsh" line}
		}
                }
    default     {
                puts "I don't recognize the platform: $tcl_platform(platform)"
                puts "Can't set platform specific parameters"
                }
    } 
  }
:LESSON_TEXT_START_LEVEL 0:
So far the lessons have dealt with programming within the Tcl interpreter.
However, Tcl is also useful as a scripting language to tie other packages
together.
  To accomplish this function, Tcl has methods for invoking subprocesses.
<P>
There are two ways to invoke a subprocess from Tcl:
<UL COMPACT>
<B><LI>open</B> ...... run a new program with I/O connected to a file descriptor
<B><LI>exec</B> ...... run a new program as a subprocess
</UL>
<P>The <B>open</B> call is the same call that is used to open a file.
If the first character in the file name argument is a pipe symbol  <B>(|)</B>,
then open will treat the rest of the argument as a program name,
and will exec that program with the standard input or output
connected to a file descriptor.  A pipe can be opened to a sub-process for
reading,  writing  or both reading and writing.
<P>
If the file is opened for both reading and writing you must be aware that
the pipes are buffered.  The output from a <B>puts</B> command will be
saved in an I/O buffer until the buffer is full, or until you execute a
<B>flush</B> command to force it to be transmitted to the subprocess. 
The output of the subprocess will not be available to a <B>read</B> or
<B>gets</B> until the I/O buffer for the subprocess has filled its
output buffer. 
<P>
The <B>exec</B> call is similar to invoking a program ( or a set of
programs piped together) from the shell prompt or in a unix
shell script.  It supports several styles of output redirection, or it
can return the output of the sub-process as the return of the <B>exec</B> 
call.
<DL>
<B><DT>open |</B><I>progName ?access?</I>
<DD>Returns a file descriptor for the pipe. The <I>progName</I> 
argument must start with the pipe symbol.  If <I>progName</I> is 
enclosed in quotes or braces, it can include arguments to the
subprocess.  
<B><DT>exec</B> <I>?switches? arg1 ?arg2? ... ?argN?</I>
<DD><B>Exec</B> treats its arguments as the names and arguments for
a set of subprocesses to execute.  If the first <I>args</I> start 
with a <B>"-"</B>, then they are treated as <I>switches</I> to the
exec command, instead of being invoked as subprocesses or 
subprocess options.
<P>
<I>Switches</I> are:
<DL>
<B><DT>-keepnewline</B>
<DD> Retains a trailing newline in the pipeline's output.  Normally a
trailing newline will be deleted. 
<B><DT>--</B>
<DD>Marks the end of the switches.  The next string will be treated
as <I>arg1</I>, even if it starts with a <B>"-"</B>
<P>
</DL>
<I>Arg1 - argN</I> can be one of:
<UL COMPACT>
<LI> the name of a program to execute
<LI> an command line argument for the subprocess
<LI> an I/O  redirection instruction.
</UL>
There are many permutations to the I/O redirection commands.
The main subset of these commands is:
<DL>
<B><DT> | </B>
<DD> Pipes the standard output of the command preceeding the pipe
symbol into the standard input of the command following the pipe symbol.
<B><DT>&lt fileName</B>
<DD> The first program in the pipe will read input from <B>fileName</B>.
<B><DT> &lt @ fileID</B>
<DD> The first program in the pipe will read input from the Tcl
descriptor <B>fileID</B>.  
<B>FileID</B> is the value returned from an <B>open ... "r"</B> command.
<B><DT> &lt &lt value</B>
<DD>The first program in the pipe will read <B>value</B> as its input.
<B><DT>&gt fileName</B>
<DD> The output of the last program in the pipe will be sent to <B>fileName</B>.
Any previous contents of <B>fileName</B> will be lost.
<B><DT>&gt &gt fileName</B>
<DD> The output of the last program in the pipe will be appended to <B>fileName</B>.
<B><DT>2&gt fileName</B>
<DD> The standard error from all the programs in the pipe will be sent to <B>fileName</B>.
Any previous contents of <B>fileName</B> will be lost.
<B><DT>2&gt &gt fileName</B>
<DD> The standard error from all the programs in the pipe will be appended to <B>fileName</B>.
<B><DT>&gt @ fileID</B>
<DD> The output from the last program in the pipe will be written to <B>fileID</B>.
<B>FileID</B> is the value returned from an <B>open ... "w"</B> command.
</DL>
</DL>
:TEXT_END:

:LESSON_TEXT_START_LEVEL 1:
So far the lessons have dealt with programming within the Tcl interpreter.
However, Tcl is also useful as a scripting language to tie other packages
together.
  To accomplish this function, Tcl has methods for invoking subprocesses.
<P>
There are two ways to invoke a subprocess from Tcl:
<UL COMPACT>
<B><LI>open</B> ...... run a new program with I/O connected to a file descriptor
<B><LI>exec</B> ...... run a new program as a subprocess
</UL>
<P>
The <B>open</B> call is the same call that is used to open a file.
If the first character in the file name argument is a pipe symbol  <B>(|)</B>,
then open will treat the rest of the argument as a program name,
and will exec that program with the standard input or output
connected to a file descriptor.  A pipe can be opened to a sub-process for
reading,  writing  or both reading and writing.
<P>
If the file is opened for both reading and writing you must be aware that
the pipes are buffered.  The output from a <B>puts</B> command will be
saved in an I/O buffer until the buffer is full, or until you execute a
<B>flush</B> command to force it to be transmitted to the subprocess. 
The output of the subprocess will not be available to a <B>read</B> or
<B>gets</B> until the I/O buffer for the subprocess has filled its
output buffer. 
<P>
The <B>exec</B> call is similar to invoking a program ( or a set of
programs piped together) from the shell prompt or in a unix
shell script.   It supports several styles of output redirection, or it
can return the output of the sub-process as the return of the <B>exec</B> 
call.
<DL>
<B><DT>open |</B><I>progName ?access?</I>
<DD>Returns a file descriptor for the pipe. The <I>progName</I> 
argument must start with the pipe symbol.  If <I>progName</I> is 
enclosed in quotes or braces, it can include arguments to the
subprocess.  
<B><DT>exec</B> <I>?switches? arg1 ?arg2? ... ?argN?</I>
<DD><B>Exec</B> treats its arguments as the names and arguments for
a set of subprocesses to execute.  If the first <I>args</I> start 
with a <B>"-"</B>, then they are treated as <I>switches</I> to the
exec command, instead of being invoked as subprocesses or 
subprocess options.
<P>
<I>Switches</I> are:
<DL>
<B><DT>-keepnewline</B>
<DD> Retains a trailing newline in the pipeline's output.  Normally a
trailing newline will be deleted. 
<B><DT>--</B>
<DD>Marks the end of the switches.  The next string will be treated
as <I>arg1</I>, even if it starts with a <B>"-"</B>
<P>
</DL>
<I>Arg1 - argN</I> can be one of:
<UL COMPACT>
<LI> the name of a program to execute
<LI> an command line argument for the subprocess
<LI> an I/O  redirection instruction.
</UL>
There are many permutations to the I/O redirection commands.
The main subset of these commands is:
<DL>
<B><DT> | </B>
<DD> Pipes the standard output of the command preceeding the pipe
symbol into the standard input of the command following the pipe symbol.
<B><DT>&lt fileName</B>
<DD> The first program in the pipe will read input from <B>fileName</B>.
<B><DT> &lt @ fileID</B>
<DD> The first program in the pipe will read input from the Tcl
descriptor <B>fileID</B>.  
<B>FileID</B> is the value returned from an <B>open ... "r"</B> command.
<B><DT> &lt &lt value</B>
<DD>The first program in the pipe will read <B>value</B> as its input.
<B><DT>&gt fileName</B>
<DD> The output of the last program in the pipe will be sent to <B>fileName</B>.
Any previous contents of <B>fileName</B> will be lost.
<B><DT>&gt &gt fileName</B>
<DD> The output of the last program in the pipe will be appended to <B>fileName</B>.
<B><DT>2&gt fileName</B>
<DD> The standard error from all the programs in the pipe will be sent to <B>fileName</B>.
Any previous contents of <B>fileName</B> will be lost.
<B><DT>2&gt &gt fileName</B>
<DD> The standard error from all the programs in the pipe will be appended to <B>fileName</B>.
<B><DT>&gt @ fileID</B>
<DD> The output from the last program in the pipe will be written to <B>fileID</B>.
<B>FileID</B> is the value returned from an <B>open ... "w"</B> command.
</DL>
</DL>
<P>
If you are familiar with shell programming, there are a few differences to
be aware of when you are writing Tcl scripts that use the <B>exec</B>
and <B>open</B> calls.
<UL COMPACT>
<LI> You don't need the quotes that you would put around arguments to
escape them from the shell expanding them.  In the example, the
argument to sed is not put in quotes.  If it were put in quotes, the
quotes would be passed to sed, instead of being stripped off (as the
shell does), and sed would report an error.
<LI> If you use the <B>open |cmd "r+"</B> construct, you must follow
each puts with a flush to force Tcl to send the command from its
buffer to the program.  The output from the subprocess may be buffered
in its output buffer.
<P>
You can sometimes force the output from the sub-process to flush by
sending an exit command to the process.
<P> You can also use the <CODE>fconfigure</CODE> command to make a 
channel unbuffered.
<P>
The <B>expect</B> extension to Tcl provides a much better interface
to other programs, which handles the buffering problem.

<LI>If one of the commands in an <B>open |cmd</B> fails the <B>open</B>
does not return an error.  However, attempting to read input from the
file descriptor with <B>gets $file</B> will return an empty string. 
Using the <B>gets $file input</B> construct will return a character
count of -1. 
<P>
Put quotes around the <B>s/.Q//g</B> in the example to see this behavior.
<LI>If one of the commands in an <B>exec</B> call fails to execute,
the <B>exec</B> will return an error, and the error output will include
the last line describing the error.
</UL>
:TEXT_END:

:LESSON_TEXT_START_LEVEL 2:
So far the lessons have dealt with programming within the Tcl interpreter.
However, Tcl is also useful as a scripting language to tie other packages
together.  This is similar to
the behavior of the DOS .bat files.  Like the .bat files, you can invoke
other programs from Tcl.  The new program is called a subprocess.
<P>
There are two ways to invoke a subprocess from Tcl:
<UL COMPACT>
<B><LI>open</B> ...... run a new program with I/O connected to a file descriptor
<B><LI>exec</B> ...... run a new program as a subprocess
</UL>
<P>
The <B>open</B> call is the same call that is used to open a file.
If the first character in the file name argument is a pipe symbol  <B>(|)</B>,
then open will treat the rest of the argument as a program name,
and will exec that program with the standard input or output
connected to a file descriptor.  A pipe can be opened to a sub-process for
reading,  writing  or both reading and writing.
<P>
Many computer operating systems buffer their input and output.  This means
that the string you send to a <B>puts</B> may not be transmitted immediately.
Instead it is written into a buffer, and when the buffer is full (or the
program executes a <B>flush</B> command) the output is actually sent.
When these programs are run interactively, they usually flush the buffer
immediately after each output, but when they are run in a pipeline between
other processes, they buffer and batch the output.  This is more efficient
for the computer.
<P>
If the file is opened for both reading and writing you must be aware that
the pipes are buffered.  The output from a <B>puts</B> command will be
saved in an I/O buffer until the buffer is full, or until you execute a
<B>flush</B> command to force it to be transmitted to the subprocess. 
The output of the subprocess will not be available to a <B>read</B> or
<B>gets</B> until the I/O buffer for the subprocess has filled its
output buffer. 
<P>
The <B>exec</B> call is similar to invoking a program ( or a set of
programs piped together) from the shell prompt or in a unix
shell script.     It supports several styles of output redirection, or it
can return the output of the sub-process as the return of the <B>exec</B> 
call.
<DL>
<B><DT>open |</B><I>progName ?access?</I>
<DD>Returns a file descriptor for the pipe. The <I>progName</I> 
argument must start with the pipe symbol.  If <I>progName</I> is 
enclosed in quotes or braces, it can include arguments to the
subprocess.  
<B><DT>exec</B> <I>?switches? arg1 ?arg2? ... ?argN?</I>
<DD><B>Exec</B> treats its arguments as the names and arguments for
a set of subprocesses to execute.  If the first <I>args</I> start 
with a <B>"-"</B>, then they are treated as <I>switches</I> to the
exec command, instead of being invoked as subprocesses or 
subprocess options.
<P>
<I>Switches</I> are:
<DL>
<B><DT>-keepnewline</B>
<DD> Retains a trailing newline in the pipeline's output.  Normally a
trailing newline will be deleted. 
<B><DT>--</B>
<DD>Marks the end of the switches.  The next string will be treated
as <I>arg1</I>, even if it starts with a <B>"-"</B>
<P>
</DL>
<I>Arg1 - argN</I> can be one of:
<UL COMPACT>
<LI> the name of a program to execute
<LI> an command line argument for the subprocess
<LI> an I/O  redirection instruction.
</UL>
There are many permutations to the I/O redirection commands.
The main subset of these commands is:
<DL>
<B><DT> | </B>
<DD> Pipes the standard output of the command preceeding the pipe
symbol into the standard input of the command following the pipe symbol.
<B><DT>&lt fileName</B>
<DD> The first program in the pipe will read input from <B>fileName</B>.
<B><DT> &lt @ fileID</B>
<DD> The first program in the pipe will read input from the Tcl
descriptor <B>fileID</B>.  
<B>FileID</B> is the value returned from an <B>open ... "r"</B> command.
<B><DT> &lt &lt value</B>
<DD>The first program in the pipe will read <B>value</B> as its input.
<B><DT>&gt fileName</B>
<DD> The output of the last program in the pipe will be sent to <B>fileName</B>.
Any previous contents of <B>fileName</B> will be lost.
<B><DT>&gt &gt fileName</B>
<DD> The output of the last program in the pipe will be appended to <B>fileName</B>.
<B><DT>2&gt fileName</B>
<DD> The standard error from all the programs in the pipe will be sent to <B>fileName</B>.
Any previous contents of <B>fileName</B> will be lost.
<B><DT>2&gt &gt fileName</B>
<DD> The standard error from all the programs in the pipe will be appended to <B>fileName</B>.
<B><DT>&gt @ fileID</B>
<DD> The output from the last program in the pipe will be written to <B>fileID</B>.
<B>FileID</B> is the value returned from an <B>open ... "w"</B> command.
</DL>
</DL>
<P>
There are a few points to
be aware of when you are writing Tcl scripts that use the <B>exec</B>
and <B>open</B> calls.
<UL COMPACT>
<LI> If you are used to shell programming, or typing a command from the
shell prompt, you don't need the quotes that you would put around arguments to
escape them from the shell expanding them.  In the example, the
argument to sed is not put in quotes.  If it were put in quotes, the
quotes would be passed to sed, instead of being stripped off (as the
shell does), and sed would report an error.
<LI> If you use the <B>open |cmd "r+"</B> construct, you must follow
each puts with a flush to force Tcl to send the command from its
buffer to the program.  The output from the subprocess may be buffered
in its output buffer.
<P>
You can sometimes force the output from the sub-process to flush by
sending an exit command to the process. 
<P> You can also use the <CODE>fconfigure</CODE> command to make a 
channel unbuffered.
<P>
The <B>expect</B> extension to Tcl provides a much better interface
to other programs, which handles the buffering problem.

<LI>If one of the commands in an <B>open |cmd</B> fails the <B>open</B>
does not return an error.  However, attempting to read input from the
file descriptor with <B>gets $file</B> will return an empty string. 
Using the <B>gets $file input</B> construct will return a character
count of -1. 
<P>
Make the <CODE>This will come back backwards</CODE> string just
<CODE>th</CODE> to see this behavior in the example.

<LI>If one of the commands in an <B>exec</B> call fails to execute,
the <B>exec</B> will return an error, and the error output will include
the last line describing the error.
</UL>
<P>
The example code shows how to use the <B>open</B> and <B>exec</B> calls
in a couple ways.  These aren't necessarily the best way to achieve a result,
but they work.  In particular, if you are processing a large amount of text,
(and are on a system that has text oriented tools)
it is frequently faster to <B>exec</B> a copy of sed and pipe the text through
there instead of writing a Tcl script to do the processing.
<P>
The example starts out by writing a simple Tcl program that will
reverse the order of characters to a temporary file.  We'll be running
that program in a separate Tcl interpreter to test reading and writing
to another program.
<P>
Once the program is written, and the file is closed, we can execute
that Tcl script.  The command:
<P>
<CODE>
set io [open "|/usr/local/bin/tclsh $tempFileName" r+]
</CODE>
<P>
opens a pipe for reading and writing to the new program.  When a string
is written to the program, it must be followed by a <CODE>flush</CODE> 
command to force the string to be sent.
<P>
When the Tcl script reads a string from <CODE>stdin</CODE> it
reverses the letters, writes it to stdout, and exits.  When this program
exits, the buffer is flushed, and the example can read the string 
from the channel.
<P>
The <CODE>exec</CODE> example shows a different method for invoking
our sample program.  A single line of input can be sent to another
program with a single Tcl command.
:TEXT_END:

:CODE_START:
# Create a unique (mostly) file name for a Tcl program
set tempFileName "TEMPDIR/inv_[pid].tcl"

# Open the output file, and 
#   write a simple program to it

set outfl [open $tempFileName w]

puts $outfl {
    set len [gets stdin line]
    if {$len < 5} {exit -1}

    for {set i $len} {$i >= 0} {incr i -1} {
        append l2 [string range $line $i $i]
    }
    puts $l2
    exit 0;
}

# Flush and close the file
flush $outfl
close $outfl

# Run the new Tcl file interactively

# Open a pipe to the program
set io [open "|TCL_INTERP $tempFileName" r+]

# send a string to the new program
#     *MUST FLUSH*
puts $io "This will come back backwards."
flush $io

# Get the reply, and display it.
set len [gets $io line]

puts  "To reverse: 'This will come back backwards.'"
puts "Reversed is: $line"
puts "The line is $len characters long"

# Run the program with input defined in an exec call

set invert [exec TCL_INTERP $tempFileName << \
	"ABLE WAS I ERE I SAW ELBA"]

# display the results
puts "The inversion of 'ABLE WAS I ERE I SAW ELBA' is \n $invert"

# Clean up
file delete $tempFileName
:TEXT_END:
