Parametrized testing, v2

Multiply your Testing E ectiveness with
Parameterized Testing
Brian Okken
@brianokken
Code and markdown for slides
github.com/okken/presentation_parametrization
Slides on slideshare slideshare.net/testandcode
1 / 25

Work
Podcasts
Book
Brian Okken
2 / 25

Outline
Value of Tests
Parametrize vs Parameterize
Examples:
without Parametrization
Function Parametrization
Fixture Parametrization
pytest_generate_tests()
Choosing a Technique
Combining Techniques
Resources
3 / 25

Value of Tests
Automated tests give us confidence. What does confidence look like?
4 / 25

Value of Tests
Automated tests give us confidence. What does confidence look like?
A passing test suite means:
I didn't break anything that used to work.
Future changes won’t break current features.
The code is ready for users.
I can refactor until I'm proud of the code.
Code reviews can focus on team understanding and ownership.
Only works if:
New features are tested with new tests.
Tests are easy and fast to write.
5 / 25

Parametrize vs Parameterize
parameter + ize
parameterize (US)
parametrize (UK)
pytest uses parametrize, the UK spelling.
I've tried to get them to change it.
They don't want to.
I've gotten over it.
6 / 25

Something to Test
triangles.py:
def triangle_type(a, b, c):
"""
Given three angles,
return 'obtuse', 'acute', 'right', or 'invalid'.
"""
angles = (a, b, c)
if (sum(angles) == 180 and
all([0 < a < 180 for a in angles])):
if any([a > 90 for a in angles]):
return "obtuse"
if all([a < 90 for a in angles]):
return "acute"
if 90 in angles:
return "right"
return "invalid"
Obtuse , Acute , Right
7 / 25

without Parametrization
test_1_without_param.py:
from triangle import triangle_type
def test_obtuse():
assert triangle_type(100, 40, 40) == "obtuse"
def test_acute():
assert triangle_type(60, 60, 60) == "acute"
def test_right():
assert triangle_type(90, 60, 30) == "right"
def test_invalid():
assert triangle_type(0, 0, 0) == "invalid"
Obtuse , Acute , Right
8 / 25

Function Parametrization
test_2_func_param.py:
import pytest
many_triangles = [
(100, 40, 40, "obtuse"),
( 60, 60, 60, "acute"),
( 90, 60, 30, "right"),
( 0, 0, 0, "invalid"),
]
@pytest.mark.parametrize('a, b, c, expected', many_triangles)
def test_type(a, b, c, expected):
assert triangle_type(a, b, c) == expected
9 / 25

output without
(venv) $ pytest -v test_1_without_param.py
===================== test session starts ======================
collected 4 items
test_1_without_param.py::test_obtuse PASSED [ 25%]
test_1_without_param.py::test_acute PASSED [ 50%]
test_1_without_param.py::test_right PASSED [ 75%]
test_1_without_param.py::test_invalid PASSED [100%]
====================== 4 passed in 0.02s =======================
10 / 25

output with
(venv) $ pytest -v test_2_func_param.py
collected 4 items
test_2_func_param.py::test_type[100-40-40-obtuse] PASSED [ 25%]
test_2_func_param.py::test_type[60-60-60-acute] PASSED [ 50%]
test_2_func_param.py::test_type[90-60-30-right] PASSED [ 75%]
test_2_func_param.py::test_type[0-0-0-invalid] PASSED [100%]
====================== 4 passed in 0.02s =======================
11 / 25

You can still run just one
(venv) $ pytest -v 'test_2_func_param.py::test_type[0-0-0-invalid]'
collected 1 item
test_2_func_param.py::test_type[0-0-0-invalid] PASSED [100%]
====================== 1 passed in 0.02s =======================
This uses the node id.
12 / 25

or more
(venv) $ pytest -v -k 60 test_2_func_param.py
collected 4 items / 2 deselected / 2 selected
test_2_func_param.py::test_type[60-60-60-acute] PASSED [ 50%]
test_2_func_param.py::test_type[90-60-30-right] PASSED [100%]
=============== 2 passed, 2 deselected in 0.02s ================
An example with -k to pick all tests with 60 degree angles.
13 / 25

Once again, this is Function Parametrization
test_2_func_param.py:
import pytest
many_triangles = [
(100, 40, 40, "obtuse"),
( 60, 60, 60, "acute"),
( 90, 60, 30, "right"),
( 0, 0, 0, "invalid"),
]
@pytest.mark.parametrize('a, b, c, expected', many_triangles)
14 / 25

Fixture Parametrization
test_3_fixture_param.py:
import pytest
many_triangles = [
(100, 40, 40, "obtuse"),
( 60, 60, 60, "acute"),
( 90, 60, 30, "right"),
( 0, 0, 0, "invalid"),
]
@pytest.fixture(params=many_triangles)
def a_triangle(request):
return request.param
def test_type(a_triangle):
a, b, c, expected = a_triangle
15 / 25

output
(venv) $ pytest -v test_3_fixture_param.py
======================= test session starts =======================
collected 4 items
test_3_fixture_param.py::test_type[a_triangle0] PASSED [ 25%]
test_3_fixture_param.py::test_type[a_triangle3] PASSED [100%]
======================== 4 passed in 0.02s ========================
16 / 25

adding an id function
test_4_fixture_param.py:
import pytest
many_triangles = [
(100, 40, 40, "obtuse"),
( 60, 60, 60, "acute"),
( 90, 60, 30, "right"),
( 0, 0, 0, "invalid"),
]
def idfn(a_triangle):
return f'{a}_{b}_{c}_{expected}'
@pytest.fixture(params=many_triangles, ids=idfn)
def a_triangle(request):
return request.param
17 / 25

output
(venv) $ pytest -v test_4_fixture_param.py
======================= test session starts =======================
collected 4 items
test_4_fixture_param.py::test_type[100_40_40_obtuse] PASSED [ 25%]
test_4_fixture_param.py::test_type[60_60_60_acute] PASSED [ 50%]
test_4_fixture_param.py::test_type[90_60_30_right] PASSED [ 75%]
test_4_fixture_param.py::test_type[0_0_0_invalid] PASSED [100%]
======================== 4 passed in 0.02s ========================
18 / 25

pytest_generate_tests()
test_5_gen.py:
many_triangles = [
(100, 40, 40, "obtuse"),
( 60, 60, 60, "acute"),
( 90, 60, 30, "right"),
( 0, 0, 0, "invalid"),
]
def idfn(a_triangle):
return f'{a}_{b}_{c}_{expected}'
def pytest_generate_tests(metafunc):
if "a_triangle" in metafunc.fixturenames:
metafunc.parametrize("a_triangle",
many_triangles,
ids=idfn)
19 / 25

output
(venv) $ pytest -v test_5_gen.py
=================== test session starts ====================
platform darwin -- Python 3.7.3, pytest-5.2.1, py-1.8.0, pluggy-0.13.0 -- /Users/o
cachedir: .pytest_cache
rootdir: /Users/okken/projects/presentation_parametrization/code
collected 4 items
test_5_gen.py::test_type[100_40_40_obtuse] PASSED [ 25%]
test_5_gen.py::test_type[60_60_60_acute] PASSED [ 50%]
test_5_gen.py::test_type[90_60_30_right] PASSED [ 75%]
test_5_gen.py::test_type[0_0_0_invalid] PASSED [100%]
==================== 4 passed in 0.03s =====================
20 / 25

Choosing a Technique
Guidelines
1. function parametrization
use this if you can
2. fixture parametrization
if doing work to set up each fixture value
if cycling through pre-conditions
if running multiple test against the same set of "setup states"
3. pytest_generate_tests()
if you need to build up the list at runtime
if list is based on passed in parameters or external resources
for sparse matrix sets
21 / 25

more test cases
test_6_more.py:
...
triangles = [
( 1, 1, 178, "obtuse"), # big angles
( 91, 44, 45, "obtuse"), # just over 90
(0.01, 0.01, 179.98, "obtuse"), # decimals
(90, 60, 30, "right"), # check 90 for each angle
(10, 90, 80, "right"),
(85, 5, 90, "right"),
(89, 89, 2, "acute"), # just under 90
(60, 60, 60, "acute"),
(0, 0, 0, "invalid"), # zeros
(61, 60, 60, "invalid"), # sum > 180
(90, 91, -1, "invalid"), # negative numbers
]
@pytest.mark.parametrize('a, b, c, expected', triangles)
22 / 25

Combining Techniques
You can have multiple parametrizations for a test function.
can have multiple @pytest.mark.parametrize() decorators on a test
function.
can parameterize multipe fixtures per test
can use pytest_generate_tests() to parametrize multiple parameters
can use a combination of techniques
can blow up into lots and lots of test cases very fast
23 / 25

Not covered, intentionally
Use with caution.
indirect
Have a list of parameters defined at the test function that gets passed
to a fixture.
Kind of a hybrid between function and fixture parametrization.
subtests
Sort of related, but not really.
You can check multiple things within a test.
If you care about test case counts, pass, fail, etc, then don't use
subtests.
24 / 25

Python Testing with pytest
The fastest way to get super productive
with pytest
pytest docs on
parametrization, in general
function parametrization
fixture parametrization
pytest_generate_tests
podcasts
Test & Code
Python Bytes
Talk Python
slack community: Test & Code Slack
Twitter: @brianokken, @testandcode
This code, and markdown for slides, on github
Presentation is at slideshare.net/testandcode
Resources
25 / 25

Parametrized testing, v2

More Related Content

What's hot (12)

Similar to Parametrized testing, v2 (20)

Recently uploaded (20)

Parametrized testing, v2