Python snippets

Very often, it is convenient to specify test parameters that are python objects. There are two ways to do this: either with expressions that evaluate to the desired object, or with multi-line snippets that assign the desired object to a predetermined variable name. Both cases are described below:

Expressions

For a concrete example where it would make sense to parametrize a test with python expressions, let’s revisit the dot product example from the Getting started tutorial:

vector.py

class Vector:

    def __init__(self, x, y):
        self.x = x
        self.y = y

def dot(a, b):
    return a.x * b.x + a.y * b.y

The a and b arguments to the dot() function are meant to be vector objects, so it makes sense to use python syntax when specifying these arguments in the parameter file. This approach provides a lot of flexibility compared to not using python syntax (e.g. just specifying coordinates and constructing the actual Vector object in the test function, as we did in the Getting started tutorial):

• We can test subclasses, e.g. Vector3D(0, 0, 1).

• We can instantiate the vectors in different ways, e.g. Vector.null() or Vector(1, 0) + Vector(0, 1).

• We can test invalid inputs, e.g. None, and make sure the proper exception is raised. See the Exceptions tutorial for more information on how to do this elegantly.

Below is the parameter file from the Getting started tutorial rewritten using python syntax. Note that the file is also rewritten in the NestedText format. NestedText never requires any quoting or escaping, so I strongly recommend it for any tests with code-like syntax[1]. Alternative formats like YAML and TOML tend to require a lot of quoting and/or escaping for values that look like code[2].

test_vector.nt

test_dot:
  -
    a: Vector(0, 0)
    b: Vector(1, 2)
    expected: 0
  -
    a: Vector(1, 0)
    b: Vector(1, 2)
    expected: 1
  -
    a: Vector(0, 1)
    b: Vector(1, 2)
    expected: 2
  -
    a: Vector(1, 1)
    b: Vector(1, 2)
    expected: 3

The simplest way to load a python object from a string is to use the built-in eval function. But there is an additional wrinkle, which is that we want the name Vector to be available when evaluating each expression. More generally, we will want a lot names from the package being tested to be available to the expressions we write. This is tedious to do with eval, so Parametrize From File includes an Namespace helper class to make this easier:

test_vector.py

import vector
import parametrize_from_file as pff
from pytest import approx

# Define this object globally, because it is immutable and will be useful for 
# many tests.
with_vec = pff.Namespace('from vector import *')

@pff.parametrize
def test_dot(a, b, expected):
    a, b, expected = with_vec.eval(a, b, expected)
    assert vector.dot(a, b) == approx(expected)

Multi-line snippets

For an example of when it would be useful to specify multiple lines of python code as a test parameter, consider a function that is meant to instantiate a vector from several different kinds of input:

vector.py

def to_vector(obj):
    if isinstance(obj, Vector):
        return obj

    # If the input object is a container with two elements, use those elements 
    # to construct a vector.

    try: return Vector(*obj)
    except: pass

    # If the input object has x and y attributes, use those attributes to 
    # construct a vector.
    
    try: return Vector(obj.x, obj.y)
    except: pass

Note

This function should raise an exception in the case where the given object can’t be converted to a vector. Testing exceptions is covered in the Exceptions tutorial, though, so for now we’ll ignore that detail.

In order to thoroughly test this function, we’ll need to instantiate an object with x and y attributes. This could be done with an expression, but it’s more clear to define and instantiate a custom class:

test_vector.nt

test_to_vector:
  -
    given: obj = Vector(1, 2)
    expected: Vector(1, 2)
  -
    given: obj = 2, 3
    expected: Vector(2, 3)
  -
    given:
      > class MyObj:
      >     pass
      >
      > obj = MyObj()
      > obj.x = 3
      > obj.y = 1
    expected:
      > Vector(3, 1)

Beyond this simple example, multi-line snippets are also very useful when testing classes that are meant to be subclassed and objects that need several steps of initialization.

As in the previous section, we can use the Namespace helper class to easily execute these snippets in a context where the name Vector is defined. Note that Namespace.exec returns a dictionary of all the variables defined while executing the snippet, and we are specifically interested in the value of the obj variable:

test_vector.py

import vector
import parametrize_from_file as pff

with_vec = pff.Namespace("from vector import *")

@pff.parametrize
def test_to_vector(given, expected):
    given = with_vec.exec(given)['obj']
    converted = vector.to_vector(given)
    expected = with_vec.eval(expected)

    assert converted.x == expected.x
    assert converted.y == expected.y

Schema argument: Be careful!

Be careful when using Namespace.eval and especially Namespace.exec with the schema argument to parametrize (e.g. via cast). This can be a convenient way to clearly separate boring type-munging code from interesting test code, but it’s important to be cognizant of the fact that schema are evaluated during test collection (i.e. outside of the tests themselves). This has a couple consequences:

• Any errors that occur when evaluating parameters will not be handled very gracefully. In particular, no tests will run until all errors are fixed (and it can be hard to fix errors without being able to run any tests).

• Even if you only ask to run a single test, the parameters for every test will have to be evaluated. This can take a substantial amount of time if you’ve written snippets that do a lot of work (more likely with Namespace.exec than Namespace.eval, but possible with both).

My general recommendation is to only use Namespace.eval and Namespace.exec with the schema argument if the snippets in question don’t involve any code from the package being tested (e.g. built-ins or third-party packages only). For example, here is a version of test_dot() that uses a schema:

test_vector.py

import vector
import parametrize_from_file as pff
from pytest import approx

with_math = pff.Namespace('from math import *')
with_vec = pff.Namespace(with_math, 'from vector import *')

@pff.parametrize(
        # *expected* is just a number, so let the schema handle it.
        schema=pff.cast(expected=with_math.eval),
)
def test_dot(a, b, expected):
    # *a* and *b* are vectors, so instantiate them inside the test function.
    a, b = with_vec.eval(a, b)
    assert vector.dot(a, b) == approx(expected)

A few things to note about this example:

• In this case, using a schema isn’t really an improvement over processing the expected value within the test function itself, like we do for the a and b parameters. When it is advantageous to use schema is when you have many test functions with similar parameters. Sharing schema between such functions often eliminates a lot of boiler-plate code.

• This also is a good example of how the Namespace class can be used to control which names are available when evaluating expressions. Here we make two namespaces: one for just built-in names (including all the names from the math module), and another for our vector package. This distinction allows us to avoid the possibility of evaluating vector code from the schema.

• This example uses cast, which is one of the schema functions provided by Parametrize From File. This function is commonly used in conjunction with defaults, error, and error_or.

• The Vector expressions used in these examples are actually a bit of a grey area, because they’re simple enough that (i) they’re unlikely to break in confusing ways and (ii) they won’t significantly impact runtime. If I were writing these tests for a real project, I would probably be ok with having the schema evaluate such expressions. Regardless, the above example shows the “right” way to do things. Use your best judgment.

[1]

In fact, I recommend NestedText for every kind of test, but the advantage is most clear with parameters that involve a lot of syntax-like characters.

[2]

For example, consider a test parameter that can either be a string or a callable. pandas.DataFrame.aggregate has func argument that works like this: a string can be used to identify one of a handful of standard algorithms, while a callable can be used to provide a custom algorithm:

In NestedText:

test_agg:
  -
    func: 'string'
  -
    func: lambda x: x

In YAML:

test_agg:
  - func: "'string'"
  - func: "lambda x: x"

In TOML:

[[test_agg]]
func = "'string'"

[[test_agg]]
func = "lambda x: x"

Note that YAML and TOML both have to double-quote the string, because the first set of quotes are interpreted as part of the file format. YAML additionally needs to quote the lambda function, because it looks too much like a dictionary item.