Writing unit tests using orca datasets

Hana 2017-01-04 22:21:24 UTC #1

I'm having an issue with unit tests for variables defined on orca datasets and was wondering if anybody came across the same issue. I have several files with unit tests in

Each of them passes when ran individually, but the test fails if run all at once, throwing a KeyError for the tested variable.

I suspect it has to do something with the setup/teardown function orca.clear_all(). For example, the following file is the only one that contains more than one test and it would not work with the orca.clear_all() calls but it does without, again only when called individually:

github.com

psrc/urbansim2/blob/master/psrc_urbansim/tests/test_parcel5_variables.py

import pandas as pd
import pytest
import orca

#def setup_function(func):
#    orca.clear_all()

#def teardown_function(func):
#    orca.clear_all()
    
@pytest.fixture
def input_pcl():
    df = pd.DataFrame(
        {
            "parcel_id":  [1,   2,    3,   4,   5],
            "x_coord_sp": [1000,   3000,    3000, 3000, 4000],
            "y_coord_sp": [2000,   1000,    2000, 2000, 2000],
            "grid_id":    [5,   3,    7,   7,   8]
        })
    return df.set_index("parcel_id")

This file has been truncated. show original

However, removing those calls also from the other files causes other table columns not being found.

I was also experimenting with setting a different scope for the pytest fixtures, but it did not work. Did anyone write successful unit tests for orca variables? Thanks!

scott_b 2017-01-30 16:38:34 UTC #2

It seems like the setup/tear down functions are being called for each test, but the inputs method is not. So variables are being removed from the environment and not added back in again?

I would try setting the scope on the inputs fixture, to function, so that it gets called with each test. I think the other fixtures are probably fine as they are.

@pytest.fixture(scope='function')
def inputs(input_pcl, input_bld, input_jobs, input_gcl, input_settings):
...

scott_b 2017-01-30 16:48:12 UTC #3

It also seems like you could just ditch the setup/tear down functions and put the orca.clearall()_ call at the top of the inputs fixture. You would still need to set the scope to function.

Hana 2017-01-30 19:21:40 UTC #4

Thanks so much for looking into it Scott! I changed the scope and made the orca.clear_all() change. It absolutely makes sense. However, it does not quite help. Two observations:
1. If the test file tests more than one variable, it does not work when orca.clear_all() is called from inputs() (or setup/tear as before). When I comment it out, it does pass when pytest runs on that single file.
2. I identified two files that do not like to be tested together (test_parcel5_variables.py and test_parcel_variables.py). They both pass independently, but only one (either of them) can be part of testing the whole batch. If both, it fails. Not sure if it's because they test the same dataset?
Thanks again!