Very often, it is convenient to specify test parameters that are python objects. There are two ways to do this: either with expressions that evaluate to the desired object, or with multi-line snippets that assign the desired object to a predetermined variable name. Both cases are described below:
For a concrete example where it would make sense to parametrize a test with python expressions, let’s revisit the dot product example from the Getting started tutorial:
class Vector: def __init__(self, x, y): self.x = x self.y = y def dot(a, b): return a.x * b.x + a.y * b.y
The a and b arguments to the
dot() function are meant to be vector
objects, so it makes sense to use python syntax when specifying these arguments
in the parameter file. This approach provides a lot of flexibility compared to
not using python syntax (e.g. just specifying coordinates and constructing
Vector object in the test function, as we did in the
Getting started tutorial):
We can test subclasses, e.g.
Vector3D(0, 0, 1).
We can instantiate the vectors in different ways, e.g.
Vector(1, 0) + Vector(0, 1).
We can test invalid inputs, e.g.
None, and make sure the proper exception is raised. See the Exceptions tutorial for more information on how to do this elegantly.
Below is the parameter file from the Getting started tutorial rewritten using python syntax. Note that the file is also rewritten in the NestedText format. NestedText never requires any quoting or escaping, so I strongly recommend it for any tests with code-like syntax. Alternative formats like YAML and TOML tend to require a lot of quoting and/or escaping for values that look like code.
test_dot: - a: Vector(0, 0) b: Vector(1, 2) expected: 0 - a: Vector(1, 0) b: Vector(1, 2) expected: 1 - a: Vector(0, 1) b: Vector(1, 2) expected: 2 - a: Vector(1, 1) b: Vector(1, 2) expected: 3
The simplest way to load a python object from a string is to use the built-in
eval function. But there is an additional wrinkle, which is that we
want the name
Vector to be available when evaluating each expression. More
generally, we will want a lot names from the package being tested to be
available to the expressions we write. This is tedious to do with
eval, so Parametrize From File includes an
Namespace helper class to make this easier:
import vector import parametrize_from_file as pff from pytest import approx # Define this object globally, because it is immutable and will be useful for # many tests. with_vec = pff.Namespace('from vector import *') @pff.parametrize def test_dot(a, b, expected): a, b, expected = with_vec.eval(a, b, expected) assert vector.dot(a, b) == approx(expected)
For an example of when it would be useful to specify multiple lines of python code as a test parameter, consider a function that is meant to instantiate a vector from several different kinds of input:
def to_vector(obj): if isinstance(obj, Vector): return obj # If the input object is a container with two elements, use those elements # to construct a vector. try: return Vector(*obj) except: pass # If the input object has x and y attributes, use those attributes to # construct a vector. try: return Vector(obj.x, obj.y) except: pass
This function should raise an exception in the case where the given object can’t be converted to a vector. Testing exceptions is covered in the Exceptions tutorial, though, so for now we’ll ignore that detail.
In order to thoroughly test this function, we’ll need to instantiate an object
y attributes. This could be done with an expression, but
it’s more clear to define and instantiate a custom class:
test_to_vector: - given: obj = Vector(1, 2) expected: Vector(1, 2) - given: obj = 2, 3 expected: Vector(2, 3) - given: > class MyObj: > pass > > obj = MyObj() > obj.x = 3 > obj.y = 1 expected: > Vector(3, 1)
Beyond this simple example, multi-line snippets are also very useful when testing classes that are meant to be subclassed and objects that need several steps of initialization.
As in the previous section, we can use the
Namespace helper class to easily execute
these snippets in a context where the name
Vector is defined. Note that
Namespace.exec returns a dictionary of all the variables defined while executing the
snippet, and we are specifically interested in the value of the
import vector import parametrize_from_file as pff with_vec = pff.Namespace("from vector import *") @pff.parametrize def test_to_vector(given, expected): given = with_vec.exec(given)['obj'] converted = vector.to_vector(given) expected = with_vec.eval(expected) assert converted.x == expected.x assert converted.y == expected.y
Schema argument: Be careful!
Be careful when using
Namespace.eval and especially
Namespace.exec with the schema
parametrize (e.g. via
cast). This can be a convenient way to
clearly separate boring type-munging code from interesting test code, but it’s
important to be cognizant of the fact that schema are evaluated during test
collection (i.e. outside of the tests themselves). This has a couple
Any errors that occur when evaluating parameters will not be handled very gracefully. In particular, no tests will run until all errors are fixed (and it can be hard to fix errors without being able to run any tests).
Even if you only ask to run a single test, the parameters for every test will have to be evaluated. This can take a substantial amount of time if you’ve written snippets that do a lot of work (more likely with
Namespace.eval, but possible with both).
My general recommendation is to only use
Namespace.exec with the
schema argument if the snippets in question don’t involve any code from the
package being tested (e.g. built-ins or third-party packages only). For
example, here is a version of
test_dot() that uses a schema:
import vector import parametrize_from_file as pff from pytest import approx with_math = pff.Namespace('from math import *') with_vec = pff.Namespace(with_math, 'from vector import *') @pff.parametrize( # *expected* is just a number, so let the schema handle it. schema=pff.cast(expected=with_math.eval), ) def test_dot(a, b, expected): # *a* and *b* are vectors, so instantiate them inside the test function. a, b = with_vec.eval(a, b) assert vector.dot(a, b) == approx(expected)
A few things to note about this example:
In this case, using a schema isn’t really an improvement over processing the expected value within the test function itself, like we do for the a and b parameters. When it is advantageous to use schema is when you have many test functions with similar parameters. Sharing schema between such functions often eliminates a lot of boiler-plate code.
This also is a good example of how the
Namespaceclass can be used to control which names are available when evaluating expressions. Here we make two namespaces: one for just built-in names (including all the names from the math module), and another for our vector package. This distinction allows us to avoid the possibility of evaluating vector code from the schema.
Vectorexpressions used in these examples are actually a bit of a grey area, because they’re simple enough that (i) they’re unlikely to break in confusing ways and (ii) they won’t significantly impact runtime. If I were writing these tests for a real project, I would probably be ok with having the schema evaluate such expressions. Regardless, the above example shows the “right” way to do things. Use your best judgment.