How to Test Jupyter Notebooks with Pytest and Nbmake

Originally published at:

Learn how you can use the pytest plugin nbmake to automate end-to-end testing of notebooks.

Hi! Thanks for great article!

One question: what is tested here? Are only errors are checked? Or can I somehow remember reference outputs and test if they are unchanged???

Hi @karelin. Mainly errors are tested here, not the outputs. Although you add assertions into the notebooks and these will be executed, this is one way you can test the outputs.

Thank yon @TomFern, but where can I learn about errors being tested?
Unfortunately standard examples it is not quite clear in this regard.

Hi Karelin, thanks for trying this out.

nbmake only checks that notebooks run without error.

If you would like to check that cells have the same output as before, please raise a feature request

Hello @alex-treebeard, thanks

this looks promising, much better than pytest-notebook, which is very low-level.
But I tried running the code and immediately got a problem: when notebooks read files, workdir is the same where the notebook is. But pytest runs, working directory is project root, notebooks crash with OS Error (file not found).

I quickly checked nbmake code and don’t see any options.

I’d try fixing it and making a pull request, where do I go to in the code?