IO Templating is a feature on Mimir Classroom that allows instructors to scan for specific things on their student's outputs. The feature utilizes regex string matches to allow for more customization when checking student output.

Use cases would include:

Checking for a certain output type but it could be different for other submissions
Checking for structure of output without matching the contents exactly
Allowing for fuzzy matching of output

The way the language works is that anything outside of the template tags will be matched exactly and anything inside the tags will be used as a regex in line to match output.

How to use it

To activate the template language for an I/O test case, you just need to include some of the template tags in your expected output. With the test case rework, the IO template language toggle switch has been removed, and by default Mimir Classroom will intelligently determine whether or not to use the templating language.

The template tags are <~ and ~> . anything in between these two tags is taken as a regex and used to compare the template to the output from the student. It is compared line by line (newlines are strict), and the score returned is the number of matching lines over the total number of lines, expressed as a percentage from 0 to 100. Anything not in tags will be compared exactly.

Examples

This is a sample output from a program:

The answer to the question: 100

a matching template:

The answer to the question: <~[0-9]+~>

Another example:

1 2 3 4 5
hello world
aaaaa
ExAmPlE
abc123

a possible matching template:

<~[0-9]~> <~[0-9]~> <~[0-9]~> <~[0-9]~> <~[0-9]~>
<~[a-z]+[\s][a-z]+~>
<~.....~>
<~[A-Za-z]+~>
<~[a-z]+[0-9]+~>

Better output diffs

The latest update to the IO template language brings more informative diffs, allowing you to see exactly where your regex may be failing to match, as well as line numbers. You can also let your students see this on their submissions if you'd like, but be aware that it will show them the template lines side by side with their output lines.

Example of the new diff format return:

Missed Lines: {number of mismatched lines}/{total lines in the file}

Line {line number of mismatch}
| {template line}
| {output line}

Note:

This output diff view will show white space characters as well to alleviate visual confusion for lines that appear to match when perhaps the student has spaces or tabs on the end of the line.

space = ␣
tab   = ⇄

Multiline enabled checking

Multiline checking requires that your regex does a full match of the student output. The regex will be compiled with the MULTILINE and DOTALL flags.

To enable the feature, add --multiline in the Run Command Line Arguments field on your test case. This flag will be used by us on the grading side, but will not be passed on to the student's program.

For an example of this functionality, consider the following simple python file:

print('this is line 0')
print('this is line 1')
print('this is the last line')

There are many different regexes that would work here, but a simple one could look like this:

<~(this).+(last line).*~>

Of course this isn't a very strict regex, but it would result in a full match against the program's output. As with the DOTALL flag, a . character can match new lines, so the first capture group is the first occurrence of the word this, and the .+ captures the rest of the output until the final capture group of (last line). The .* at the end of the regex is to ensure that we still match the students output whether they have added a trailing newline or not.