The previous section showed how to use quoting in the run program of the job scheduler systems BICsuite and schedulix. But sometimes we need to access environment variables, in order to adapt the behaviour of our program to the special requirements of the execution environment. As an example we assume that some input file lies in the $HOME/input directory.
We’d like to tell our printargs program about it. To do so, we assume that printargs expects two command line arguments, a -i indicates that the input has to be found in the directory that is specified by the argument that follows it.
In other words, we’d like to have a run program that looks like:
run program = printargs -i "$HOME/input" ...
but we already know that it won’t work in this form. The scheduling server will complain that it cannot resolve the parameter HOME. And if we escape the dollar sign, printargs will have to deal with the string $HOME/input, but doesn’t know that this string still contains unresolved variables. Hence both approaches don’t work.
If we think a bit deeper, we realize that in fact a variable reference like $HOME is a syntax element of a shell. In order to resolve the $HOME variable, we need to invoke a shell.
This leads us to the next attempt:
run program = sh -c "printargs -i \$HOME/input "’$TESTPARM = ’"$TESTPARM"
The result almost looks good:
Number of arguments: 6 arg[0] = -> printargs <- arg[1] = -> -i <- arg[2] = -> /opt/schedulix/input <- arg[3] = -> = <- arg[4] = -> Hello <- arg[5] = -> World <-
Note that we don’t see the arguments of the sh program, but only the arguments of the prinziemlichtargs program, which has been invoked by the shell. Still there are a few details that need further investigation. First of all, we look into the GUI and look at the run commandline field on the run tab of the Job. It shows:
sh "-c" "printargs -i $HOME/input $TESTPARM = Hello World"
It is important to understand that the printed command line isn’t necessarily the exact command line that has been executed. In fact, the run program is parsed and subdivided into a list of single arguments. This list is again converted back into a string by enclosing each of the arguments in double quotes. The simple principle of rendering the run commandline often works quite well and gives a good picture of what’s going on. In the above case, it is easy to see that the original array of arguments, looks like:
Number of arguments: 3 arg[0] = -> sh <- arg[1] = -> -c <- arg[2] = -> printargs -i $HOME/input $TESTPARM = Hello World <-
And now we’re able to understand the previous output of our attempt. The shell invokes the printargs command and provides the arguments as it finds it within the command string. It is able to resolve $HOME, but can’t find the variable $TESTPARM. The latter is therefore replaced with an empty string, while the former is replaced with /home/schedulix.
Even if we now understand why the output looks like it does, it doesn’t actually match the thing we were hoping for. In the end we just wanted to add the -i $HOME/input to the arguments of the printargs utility, and we wanted to keep the original behaviour of printing $TESTPARM = Hello World as a single argument.
It is the shell that tears apart our carefully quoted run program. Thus we’ll have to use some sophisticated quoting to get it right:
run program = sh -c "printargs -i \$HOME/input ’\$TESTPARM = $TESTPARM’"
The entire command line that has to be executed by the shell is enclosed in double quotes. These double quotes will be removed by the scheduling server, before passing the array of arguments on to the Jobserver. Since the latter $TESTPARM is only enclosed in double quotes (from the perspective of the scheduling server), it’ll be resolved. The first \$TESTPARM doesn’t refer to a parameter since the dollar sign has been escaped and remains as it is, but without the backslash, which is eaten by the scheduling server. Running the job yields the desired result:
Number of arguments: 4 arg[0] = -> printargs <- arg[1] = -> -i <- arg[2] = -> /opt/schedulix/input <- arg[3] = -> $TESTPARM = Hello World <-
That looks pretty good. Because of this great result, we get a bit carefree and require that the contents of the TESTPARM parameter has to be enclosed in double quotes (for the same reason why printargs uses the ”arrows”). This turns out to be tricky! The solution to the problem looks like this:
run program = sh -c "printargs -i \$HOME/input ’\$TESTPARM = "’"’"$TESTPARM"’"’"’"
It would be somewhat unfair not to explain these impressive lists of alternating single and double quotes. So let’s have a closer look:
If we carefully now remove the start and end quotes, we’ll find the string that is digested by the shell. For clarity we first observe that everything that follows the $HOME/input argument will be considered to be a single argument by the scheduling server. But some parts will be replaced by values, and some backslashes will be removed as well as some quotes, before it is handed over as an argument to the shell. Hence the (shell) command line starts with printargs -i \$HOME/input and is then followed by a single quote (line 2). The next character will be a double quote (line 6). The TESTPARM parameter is resolved and replaced by its value (line 9). Another double and then a single quote is added, resulting in
printargs -i $HOME/input ’$TESTPARM = "Hello World"’
which is then passed on as a single argument to the shell. Testing this with our test job shows that it works:
Number of arguments: 4 arg[0] = -> printargs <- arg[1] = -> -i <- arg[2] = -> /opt/schedulix/input <- arg[3] = -> $TESTPARM = "Hello World" <-
Part 1: Introduction and simple usage
Part 2: Advanced usage
Part 3: Backticks
Part 4: Other interpreters
Part 5: Circumventing limitations