Mimir offers a convenient suite of tests types to help you evaluate your student's HTML/JS/CSS code ranging from automating a browser, automated quality checking, as well as a full custom test case environment with bash scripting.

Browser Test Cases

Browser Test Cases let us automate test the DOM in a human readable way. If you want to follow along with the examples, you can download the source code for this project here. Create a web project with this zip as the perfect code, and you'll get the same results as these examples.
To create a browser test case, navigate to the "Edit" page in your Web Project, click "Grading", and click "Add Test Case". Click "Start From Scratch", then select "Browser Test Case" in the "Test Case Type" drop down.

We use a library called Codecept to test the contents of a page. Codecept enables you to write UI tests in a format that's easy for humans to read and understand. It has simple and concise helpers for just about everything you can do in a browser. To learn more about the helpers you can use for the project you're testing, you can dive into the codecept help documentation.

For now, we'll write a simple test using three of Codecept's most common test helpers, see, click , and amOnPage. First, set the "Main File Name" to be the name of name of the page you want to test. For example, if you want to the test to start on index.html, set index.html in the dropdown. For this test, we'll set index.html as our main file. Then, add the following to the "Codecept Script" box on the test case form.

I.see('Hi there.', 'h1');
I.screenshot('1');
I.see('This is a link to another page');
I.click('This is a link to another page');
I.amOnPage('/otherpage.html');
I.screenshot('2');
I.see('This is a different page', 'h1');

Though the test is self-explanatory, let's break it down step by step:

The test checks to see if there is an h1 tag with the text "Hi there."
The test takes a screenshot
The test checks to see if there is a link (a tag) with the text "This is a link to another page".
The test clicks on the link to the other page.
The test checks to see if the URL of the app is now "otherpage.html".
The test takes a screenshot
The test checks to see if there is an h1 tag with the text "This is a different page", which is present on otherpage.html but not on index.html .

While experimenting, you may notice that when you type I, you'll get auto-completions for the different functions on I within the editor on the test case form. This is an easy way to dive into Codecept and get familiar with the functions at your disposal. While hovering over a suggestion, you can press ctrl+space to see details on the function, including parameter types and a description of the function.

I.screenshot is a custom function that we created that stores screenshots to a special directory which is then saved and shown in the test case results. This lets you automate the process of taking screenshots of student code, and lets you review them manually. The first argument of I.screenshot is the name of the file that will be saved, less the .png file extension. We may add features to compare screenshots in the future. It should be noted that Codecept has a built in I.saveScreenshot method. The I.screenshot method builds on top of this and saves screenshots to a folder that we upload to the server, so you should use I.screenshot instead.

There's more to learn in Codecept, but you'll find any information you need in the Codecept docs.

Metadata

On browser test cases, you also have access to certain metadata about the user who's submitting and the submission, so you can do things like check to see if the user put their name in the HTML. You can access this with the mimir variable in your testing script. Here's the variables that are available:

mimir.SUBMISSION     # submission namespace object
    .id              # id of the submission
    .created_at      # int timestamp of the created submission

mimir.TEST_CASE      # test case namespace object
    .id              # id of the test case
    .name            # name of the test case

mimir.USER           # user namespace object
    .id              # id of the user
    .email           # email of the user
    .first_name      # first name of the user
    .last_name       # last name of the user

Advanced Usage:

In addition to the normal features of CodeceptJS, there are some advanced features that are useful for writing certain types of tests. The first thing to learn is executeScript, which is a powerful utility that lets you run code in the browser instance, and return it so it can be tested. For example:

let value = await I.executeScript(() => {
    let element = document.querySelector('article'); 
    let styles = window.getComputedStyle(element); 
    return compStyles.getPropertyValue('width');
});

I.assertEqual(value, '50px', 'Your article element should be 50px wide');

You can use executeScript to test for lots of things that couldn't ordinarily be tested in Codecept. Try it out, and let us know if you can't figure out how to test something my messaging us with the bubble in the bottom right corner, we'll be glad to help you design a test! You can learn more about the various assert functions by typing I.assert in your test case, which will reveal the autocomplete options.

Custom Web Test Cases

Custom test cases on web projects are similar to our normal custom test cases. You are given a bash script that you can use to test the student's submission however you see fit, including installing libraries or using third-party tools. Custom web test cases have one additional nifty feature on top of ordinary custom test cases, the student's code is hosted at http://localhost on port 80 . Lets create an example to show how this can be useful. Create a new test case, but this time select "Custom Web Test Case" in the test type drop down. Then add the following into the script input.

curl -XGET http://localhost -o request_index.html

if grep "Hi there." request_index.html -q; then
  echo "Correct" >> DEBUG
  echo 100 > OUTPUT
else 
  echo "Incorrect" >> DEBUG
  echo 0 > OUTPUT
fi

This test pulls down the index.html file from the server, saves it as request_index.html , and checks to see if the file contains the string "Hi there.".

As in normal custom test cases, all the files of the students submission are present in the directory, so you can also run any linters or other checks on the files within the bash script.

HTML/CSS Code Quality Test Cases

Our code quality test cases utilize W3 standards on HTML and CSS files. The file to be checked is specified by Mail File Name field. The linter we use is called "Nu Checker", and you can find more information about it here. When student's have errors in their HTML/CSS code, Nu Checker will give them concise instructions on what they are, and how to fix them, allowing them to self-learn how to properly structure their code.

Lighthouse Test Cases

Lighthouse is a tool by Google that lets you audit web pages for performance, accessibility, search engine optimization, coding best practices, and adherence to progressive web application standards.

Create a test case with the lighthouse test case type, and paste this in the "Lighthouse Config" section:

performance > 99
accessibility >= 50
seo >= 25
best-practices >= 90
pwa > 35

In this test case, we're checking to make sure the performance of the page scores more than 99, accessibility is greater than or equal to 50, SEO is greater than or equal to 25, best practices are greater than or equal to 90, and progressive web app grade is greater than 35.

Currently, Mimir's lighthouse tests are still in beta, and may not provide much in terms of debugging information.

To learn more about lighthouse and how it audits web pages, you can visit their documentation here.

Web Test Cases