Using BAT files to run GEMPACK Programs automatically

Prerequisite: To create and use BAT (batch) files you need to be familiar with running GEMPACK from the command-prompt (also called 'DOS box'). You may need to review this page.

Introduction

When working on a project you might find yourself repeatedly running the following commands from the command-prompt:

tablo -sti model3.sti  generate GSS, GST files for GEMSIM
gemsim -cmf sim1.cmf   run simulation 1
gemsim -cmf sim2.cmf   run simulation 2
gemsim -cmf sim3.cmf   run simulation 3

You would type the black text above (not the explanatory red text). Each time the TAB file changed, you would need to re-run all 4 commands. This might become tedious !

A BAT file to run a sequence of commands

To speed your work, you might create a text file, DoSims.BAT, containing the lines:

tablo -sti model3.sti  
gemsim -cmf sim1.cmf   
gemsim -cmf sim2.cmf   
gemsim -cmf sim3.cmf

Then if you typed at the command-prompt:

dosims.bat

all 4 commands would be executed automatically -- perhaps while you went for coffee!

Although handy, DoSims.BAT has some weaknesses: for example, it will not alert you if one of the commands failed for some reason. The sections below show how to make BAT files more robust.

Advantages of using BAT files

As well as saving effort, BAT (or batch) files have several other advantages:

Mistakes (eg, forgetting one step in a long job) are reduced. Indeed BAT files are really the only way to ensure that complex processes are repeatable.
BAT files make it easier to replace one big TAB file by several smaller files, each doing one part of the job. The smaller files will be easier to understand, maintain and reuse for other projects.
Some tasks are inconvenient or impossible to do with a TAB program; for example, extracting results from an SL4 file. Your BAT file could use the GEMPACK program SLTOHT to perform this task. Or it could use non-GEMPACK tools (such as Perl, MatLab or GAMS) for some parts of the job.
BAT files help you share work with others: you send your colleague all necessary ingredient files together with a BAT file to run your jobs. You can be sure she is running the same jobs as you.
Returning to a project after a gap of some months, you find several versions of the TAB, HAR, and CMF files -- and you are not sure which version was used to generate the results in your report! If a single BAT file had been used to generate all the results, you could examine that BAT file to accurately trace back which input files were used. The BAT file is a form of documentation that is brief and accurate -- in contrast to other methods of documentation (eg, notes) which may become out-of-date.

Building robust BAT files

Consider the following BAT file for a recursive-dynamic (one-year-at-a-time) CGE model:

tablo -sti dynagem.sti 
gemsim -cmf year2001.cmf  
gemsim -cmf year2002.cmf  
gemsim -cmf year2003.cmf  
gemsim -cmf year2004.cmf

Suppose that year2001.cmf reads base2001.har and produces an updated database y2002.har; year2001.cmf reads y2002.har to produce y2003.har, and so on. Suppose you have run this BAT several times before, and now you run it again over lunch-time with altered shocks. If the first (year2001) simulation crashed, it is possible that the second (year2002) simulation would run OK, using as input a file y2002.har produced yesterday. You could return from your lunch and never realize that an error had occurred!

To avoid such a danger, your BAT file should enforce the following rule:
First, delete all files that you expect to produce.
The revised BAT file might read (red text not included):

del *.log         delete all LOG files
del dynagem.gs?   delete GSS and GST files needed by GEMSIM, and made by TABLO
del year20??.sl4  delete solution files which should be made
del year20??.har  delete updated data which should be made
tablo -sti dynagem.sti 
gemsim -cmf year2001.cmf  
gemsim -cmf year2002.cmf  
gemsim -cmf year2003.cmf
gemsim -cmf year2004.cmf
dir/od year20??.sl4

After this change, it is impossible for any job to run, unless every preceding job produced the expected output files. You can be sure that all outputs (including log files) were indeed produced during the most recent run. You could tell that the job crashed during the first year since the last command would list no solution files.

Adding comments and trapping output

Use comments to document your BAT. Any line beginning REM (short for 'remark') is treated by Windows as a comment (ie, ignored).

By default, TABLO and GEMSIM will send thousands of lines of output to the screen. Instead, you can use the ">" redirection operator to save output to a file, or to discard it. Our BAT file becomes:

REM dynamic simulation, using input files: 
REM   dynagem.tab dynagem.sti base2001.har and year20yy.cmf
REM first, delete files we expect to produce 
del *.log         
del dynagem.gs?   
del year20??.sl4  
del year20??.har  
tablo -sti dynagem.sti >tbdynagem.log
gemsim -cmf year2001.cmf  >NUL
gemsim -cmf year2002.cmf  >NUL  
gemsim -cmf year2003.cmf  >NUL
gemsim -cmf year2004.cmf  >NUL 
dir/od year20??.sl4

Now, output from TABLO will be stored in file tbdynagem.log. We're assuming that each CMF file contains the line:

log file = yes;

meaning that log output will already be stored in files named (after the cmf file) year200?.log, so that GEMSIM screen output may safely be thrown away. The expression ">NUL" causes Windows to simply suppress the output (ie, NUL is Windows' name for the bit bucket.)

Running only part of the BAT

Suppose you altered year2003.cmf, so that the last 2 simulations must be run again -- and you are too impatient to run the whole BAT. You could edit the BAT to jump to a label:

REM dynamic simulation, using input files: 
REM   dynagem.tab dynagem.sti base2001.har and year20yy.cmf
goto skip     jump to label "skip" ; no colon here
REM first, delete files we expect to produce 
del *.log         
del dynagem.gs?   
del year20??.sl4  
del year20??.har  
tablo -sti dynagem.sti >tbdynagem.log
gemsim -cmf year2001.cmf  >NUL
gemsim -cmf year2002.cmf  >NUL  
:skip         arrive here; note that label line must begin with a colon 
gemsim -cmf year2003.cmf  >NUL
gemsim -cmf year2004.cmf  >NUL 
dir/od year20??.sl4

This of course would be a temporary hack; it evades the output file deletion which protects us from error. Usually the goto jumps to a label in a subsequent line; however backward jumps are also possible -- allowing endless loops!

Better error-checking

A well-written program, when it completes, informs Windows of its exit status: either errorlevel 0 (success), or errorlevel 1 (failure). The BAT can test the errorlevel, and jump according to the result. With these checks our BAT file might become:

REM dynamic simulation, using input files: 
REM   dynagem.tab dynagem.sti base2001.har and year20yy.cmf
REM first, delete files we expect to produce 
del *.log         
del dynagem.gs?   
del year20??.sl4  
del year20??.har  
tablo -sti dynagem.sti >tbdynagem.log
if errorlevel 1 goto error  jump to "error" if any problem
gemsim -cmf year2001.cmf  >NUL
if errorlevel 1 goto error  jump to "error" if any problem
gemsim -cmf year2002.cmf  >NUL  
if errorlevel 1 goto error  jump to "error" if any problem
gemsim -cmf year2003.cmf  >NUL
if errorlevel 1 goto error  jump to "error" if any problem
gemsim -cmf year2004.cmf  >NUL 
if errorlevel 1 goto error  jump to "error" if any problem

echo Job done OK     if arrive here, all must be OK
dir/od year20??.sl4  list results
goto endOK          finish now -- do NOT go on to next lines

:error               ---- GET HERE IF ANY ERROR
dir/od *.log         list Logs, most recent last
echo PROBLEM !!!! examine most recent Log
exit /b 1            exit BAT, setting errorlevel to 1

:endOK              exit BAT file

Note the following:

After each command that might go wrong, there is an errorlevel test. If there is a problem, execution jumps to "error"; otherwise execution continues on the next line.
The expression "errorlevel N" is true if the errorlevel is equal to or greater than N. So it would be wrong to write " if errorlevel 0 goto ...".
The Echo command merely displays some text on the screen.
If the BAT succeeds, 'Job done OK' is echoed, and the main outputs (the solution files) are listed in order of creation.
If the BAT fails, a warning is echoed, and log files are listed in order of creation. You can open the most recent log with TABmate, and find the cause of the problem, probably described near the end of that log file.
If the BAT fails, the command "exit /b 1" quits, setting errorlevel to 1 (ie, indicating error). That errorlevel could be queried by another BAT (if you had one BAT calling another BAT).

BAT with source-code GEMPACK

For source-code GEMPACK, which allows you run a model-specific EXE instead of GEMSIM, our example BAT file needs minor alterations (see red comments):

REM dynamic simulation, using input files: 
REM   dynagem.tab dynagem.sti base2001.har and year20yy.cmf
REM first, delete files we expect to produce 
del *.log         
del dynagem.ax?    delete obsolete AXS and AXT files
del dynagem.exe    delete previous EXE file
del year20??.sl4  
del year20??.har  
tablo -sti dynagem.sti >tbdynagem.log
if errorlevel 1 goto error  
Call LTG dynagem            need "call" since LTG is really LTG.BAT
if errorlevel 1 goto error  
dynagem -cmf year2001.cmf  >NUL  Dynagem instead of GEMSIM
if errorlevel 1 goto error  
dynagem -cmf year2002.cmf  >NUL  
if errorlevel 1 goto error  
dynagem -cmf year2003.cmf  >NUL
if errorlevel 1 goto error  
dynagem -cmf year2004.cmf  >NUL 
if errorlevel 1 goto error  

echo Job done OK     
dir/od year20??.sl4  
goto endOK          

:error               
dir/od *.log         
echo PROBLEM !!!! examine most recent Log
exit /b 1            exit BAT, setting errorlevel to 1

:endOK

Note the use of CALL with LTG: CALL is needed because LTG is actually a BAT; without the CALL, execution of the BAT would not resume after LTG had finished. Use CALL to make execution resume at the line following LTG.

Waiting for Windows programs to finish

In the examples so far, we have assumed that BAT lines are executed in order, and that each command finishes before the next one begins. However, if a BAT line launches a Windows (GUI) program, such as Excel or ViewHAR, BAT execution continues straight onto the next line, without waiting for Excel to close. You can see the same phenomenon if you type "ViewHAR" into the command-prompt: ViewHAR starts but does not prevent you typing in more commands.

If you want to make sure that Excel finishes before the next BAT line is run, you should include a BAT line like:

start/wait "C:\Program Files\Microsoft Office\Office\EXCEL.EXE" results.xls

Above note that (a) Excel is not usually on the PATH, so the actual location of Excel.exe needs to be given; and (b) the full filename contains spaces, so must be quoted. Actually, because Windows knows that XLS files are "associated" with Excel, the following simpler command would also work.

start/wait results.xls

Similarly you could display an HTM file, without knowing which browser (IE or Firefox) will be chosen by Windows:

start info.htm

Passing parameters to BAT files

Very often we need to run "TABLO -sti xxxx" followed by "LTG xxxx", where "xxxx" is the name of a TAB file. We could create a file COMPILE.BAT which combined both operations and worked for any TAB file, as follows:

REM run TABLO and LTG for %1 model 
REM assumes that both %1.TAB and %1.STI exist
del *.log         
del %1.ax?   delete obsolete AXS and AXT files
del %1.exe   delete previous EXE file
tablo -sti %1.sti >tb%1.log
if errorlevel 1 goto error  
Call LTG %1    need "call" since LTG is really LTG.BAT        
if errorlevel 1 goto error  
echo Job done OK     
dir/od %1.?x?   list produced AXS,AXT and EXE files
goto endOK          

:error               
dir/od *.log         
echo PROBLEM !!!! examine most recent Log
exit /b 1            exit BAT, setting errorlevel to 1
:endOK

To run this BAT you could type, for example:

compile.bat oranig

Above, "oranig" is the first (indeed the only) parameter of the BAT. As each line of the BAT is run, each occurrence of "%1" is replaced by "oranig".If the BAT required more parameters, they would be represented by %2, %3 and so on (up to %9).

You could also type just:

compile oranig

ie, omit the ".bat". This however carries a risk -- if there is a file called compile.exe, that file will run instead of compile.bat [the PathExt environment variable controls this feature].

A BAT file to create a snapshot of your work

All sorts of utility BAT files can be created to speed your work. The next example, SNAPSHOT.BAT, (a) deletes various junk files, and (b) creates a snapshot zip of important files in the current folder. You would run it every few hours to make a "restore point" or backup. Then if, for example, you accidentally deleted an important file, a recent backup copy would be available.

REM clean up junk and back up important files
del *.bak
del *.log
del gpxx*.*
del *.slc
del *.upd
REM now zip up all files needed to repeat the work
del snap.zip     ensure no existing snap.zip
zip snap *.TAB *.CMF *.BAT *.STI basedata.har  put important files in snap.zip
md snapshots       create subfolder "snapshots" if it does not already exist
move snap.zip snapshots\snap%RANDOM%.zip   put snap.zip in snapshots folder with new (randomly-generated) name
dir/od snapshots   list snapshot zips in order of creation

Learning more about BAT files

We have barely scratched the surface of this huge topic -- we have not even mentioned the fascinating and powerful SET and FOR commands. However, if you followed the examples above, you have learned enough to do useful work. Below are some resources to find out more.

To find Microsoft's (scant) online help about batch files, click on the Desktop, and press F1 -- "Help and Support Centre" should appear. Then search for batch. You can also get help from the command prompt by typing a command with the "/?" option, for example type "set/?" or "echo/?".

Some more advanced GEMPACK examples may be found in Archive Item TPMH0129.

Other resources are www.ss64.com, www.dostips.com and www.robvanderwoude.com. There is a book, Windows NT Shell Scripting, by Timothy Hill. Or just Google DOS batch

Batch helpers

are small programs to help do something that would otherwise be difficult in a BAT file. Three are included in the GEMPACK "bundle" that you can download from here. Each is intended to assist branching using the "if errorlevel" construct.

LATER sets the errorlevel according to whether all of one group of files is created after a second group. You could use LATER to only run jobs again if any of the input files had changed.
TSTHAR sets the errorlevel by reading a value from a HAR file. This provides way for a TAB program to control which BAT commands are executed.
SETERR just sets the errorlevel to a specified value.

Type Later/? or TstHar/? or SetErr/? to find out more.

Keep your BAT files simple

BAT files can be addictive ! Some people (you will find them on the internet) develop an interest that goes beyond the utilitarian. You will find sites that discuss:

undocumented quirks of the BAT language; and how it differs slightly between various versions of DOS and Windows.
how you can ingeniously accomplish almost any task with BAT files -- if you try hard enough!
how you can use ECHO command to hide the details of the BAT file from the user.
the weaknesses of BAT files compared to more powerful (but less common) shells or scripting languages.

You should avoid the more esoteric techniques, especially if you hope that other people can use and understand your BAT files, and that they will run on PCs with other versions of Windows.