The BICsuite Run Program Part 5: Circumventing limitations

The maximum length of the run program is limited to 512 characters. Usually this doesn’t feel like any kind
of limitation. Sometimes though, especially when creating long inline scripts, the limit is easily reached.
But even if the run program field is limited in length, the command line, that is executed after parameter
substitution, isn’t. This is because there is also no practical limitation to the maximum length of a parameter
value, parameters can be used to circumvent the 512 bytes limitation.
If we look at the previous example, wouldn’t it be nice to write something like

run program = /bin/bash -c "$REPORTIPS"

where REPORTIPS is a parameter (of type constant) that contains the script to be executed?
Again, the main challenge is to get the quoting right, so that all variables are evaluated by the correct
interpreter. In our case, NUM should be interpreted by the scheduling server, $foo and $bar should be
interpreted by the shell and last but not least $3 has to be interpreted by awk.

We’ve seen that the parameter substitution is a recursive process; while replacing the parameter reference
with its value, the value is parsed and if any parameters are found, they will be replaced as well.
It’ll be a mind twisting task to get this right, which is why BICsuite offers some help. A pragma which
instructs the server not to look for dollar signs, but some other character instead, can be added to the
”script”. It would then look like this:

# BICSUITE_PRAGMA_VPFX:&;
lastb -i -f /var/log/btmp |
awk ’$3 ~ /[0-9]+.[0-9]+.[0-9]+.[0-9]+/ { cnt[$3]++ }
    END {
    for (ip in cnt)
        if (cnt[ip] > "&NUM")
        print ip "\t" cnt[ip]
    }’ | (
while read foo bar; do
    if ! grep -q " $foo/" /etc/sysconfig/iptables; then
        if [ "&VERBOSE" == "Y" ]; then
        echo "# $foo $bar";
        fi;
        echo "-A INPUT -s $foo/32 -j DROP";
        echo "-A OUTPUT -d $foo/32 -j DROP";
    fi;
done) | sort

What we see here is the use of a ”pragma”, BICSUITE_PRAGMA_VPFX (”VPFX” stands for Variable PreFiX), followed by a colon and a new character, in this case an ampersand. Since that pragma is not a bash command (and even if it were, we are addressing another system), we put a hash sign in front of it, in order to make it look like a comment to the bash. The effect of the pragma is that the scheduling server doesn’t look for dollars, but for ampersands instead. This way all variable references, regardless if in the awk code or in the bash code, are left untouched. Only the
&NUM and &VERBOSE are replaced with their values.

Due to the double quotes in the run program, the script is passed on as a single parameter to the bash. Hence
the quoting within the parameter’s value now follows the normal bash rules, which makes it substantially easier to get it right.

System dependent issues

It is well known that different operating systems have a different view of lines of text, especially if it is about
how to end a line. Unix like systems think that merely a newline is sufficient to end a line. Windows systems
not only need a newline but also a carriage return in front of it. And Apple systems only want to see a
carriage return.

Depending on how and where a job definition is created (though the GUI, through sdmsh, or with some
other utility), the parameter value will contain different line ends. Unfortunately the behaviour of the bash
depends on those line ends and the operating system that runs the bash. If for example the bash running
on Linux encounters a carriage return, it gets confused (and it doesn’t see it as plain white space) and the
script won’t run.

In order to repair the scripts on the fly, there exists a not yet documented configuration parameter for
Jobservers called CONVERT_NEWLINE (This requires the latest patch for release 2.9 and higher; the configuration parameter isn’t available for release 2.8 and earlier). If this parameter is set, the line ends will be converted according to the
parameter’s value:

Value   Effect
    0   Nothing happens (the default)
    1   CR-LF is converted to LF
    2   LF without preceding CR is converted to CR-LF

As with every Scope configuration parameter, it can be set at any level withing the hierarchy and the set
value will be valid for the entire subtree (unless overridden). Since the configuration parameter can’t be set
through the GUI (yet), it has to be set using the shell in the GUI or by using sdmsh. Assuming the parameter
has to be set for scope GLOBAL.SVR1, the statement looks like this:

alter scope GLOBAL.’SVR1’
with
        config = (
            ’CONVERT_NEWLINE’ = ’1’
        );

Part 1: Introduction and simple usage

Part 2: Advanced usage

Part 3: Backticks

Part 4: Other interpreters

Part 5: Circumventing limitations

Part 6: Security considerations and conclusion