Using injectables in orca columns

[Hana]

I'm trying to figure out how to use injectables, such as "year", in orca columns. Specifically, I'd like to define a variable determining the age of buildings, i.e. the variable needs to have an access to the simulation year. Year is defined in urbansim_defaults/datasources.py as:

@orca.injectable('year')
def year(iter_var):
    return iter_var

Our age variable uses it as follows:

@orca.column('buildings', 'age', cache=True, cache_scope='iteration')
def age(buildings, year):
    year_built = buildings.year_built
    year_built[buildings.has_valid_age_built==0] = np.nan
    return np.maximum(0, year - year_built)

This works in an estimation setting. However, in a simulation setting orca throws an error complaining about iter_var, see below. The error appears even if the age variable is not used. If I comment out this column definition, the simulation works. What am I doing wrong?

Error:

Traceback (most recent call last):
  File "simulate.py", line 22, in <module>
    ], iter_vars=[2015], data_out="simresult.h5")
  File "/Users/hana/udst/orca/orca/orca.py", line 1912, in run
    write_tables(data_out, out_base_tables, 'base')
  File "/Users/hana/udst/orca/orca/orca.py", line 1855, in write_tables
    store[key_template.format(t.name)] = t.to_frame()
  File "/Users/hana/udst/orca/orca/orca.py", line 229, in to_frame
    df[name] = col()
  File "/Users/hana/udst/orca/orca/orca.py", line 632, in __call__
    expressions=self._argspec.defaults)
  File "/Users/hana/udst/orca/orca/orca.py", line 952, in _collect_variables
    variables[label] = thing()
  File "/Users/hana/udst/orca/orca/orca.py", line 747, in __call__
    expressions=self._argspec.defaults)
  File "/Users/hana/udst/orca/orca/orca.py", line 949, in _collect_variables
    thing = all_variables[expression]
KeyError: 'iter_var'

[scott_b]

The iter_var injectable is only available when injectables are evaluated within the context of an orca.run(). So we define an additional injectable called baseyear that serves as a default year:

orca.add_injectable('base_year', 2014)

@orca.injectable()
def year(base_year):
    if 'iter_var' in orca.list_injectables():
        year = orca.get_injectable('iter_var')
        if year is not None:
            return year
    
    # outside of a run, return the base/default
    return base_year

@orca.column('buildings', 'building_age')
def bldgs__building_age(buildings, year):
    return year - buildings.year_built

If there's a different, specific year, you need to evaluate, this should work as well:

orca.add_injectable('year', 2020)
bldg_ages_20 = orca.get_table('buildings').building_age

[Hana]

Thanks so much Scott! I thought the simulation is indeed run within the orca.run() context to which the iter_vars argument is passed, as can be seen from the first three lines of the error traceback above. Your solution works, which means that somehow the variables are also computed outside of the "run" context when called from orca.run()? Anyway, thanks again, I'm still scrambling to figure out how all this works.

[scott_b]

OK, I think I see what is happening, and I think it's a bug I introduced when I tried to fix the issue of the 1st year simulation results not getting exported (https://github.com/UDST/orca/pull/11). Sorry.

The issue is in writing out the base year tables (~ line 1911):

# write out the base (inputs)
    if data_out:
        write_tables(data_out, out_base_tables, 'base')

    # run the steps
    for i, var in enumerate(iter_vars, start=1):
        add_injectable('iter_var', var)

The call to write_tables is calling buildings.to_frame() which in turn is evaluating all the columns on the data frame, regardless of whether or not you are using them (all columns on a table are always written to the store, it would probably be nice if this could be configured). So the issue is that the iter_var injectable is not getting adding to the environment until later when the simulation years start.

So you don't get this issue when doing estimations, because your are probably not exporting data tables in that run call.

I think this can get fixed with the following, I'll try to get this submitted in the next few days.

# write out the base (inputs)
    if data_out:
        add_injectable('iter_var', iter_vars[0])
        write_tables(data_out, out_base_tables, 'base')

[Hana]

Thanks Scott - that makes sense. I agree that writing all variables regardless if they are used or not is not very convenient - it takes ages and the file is huge. For example, I can't configure a small test simulation with just one model that runs within seconds, because the (unnecessary) variable computation delays the processing significantly. But that might be a subject for another thread. Thanks for fixing this!

[janowicz]

Nice troubleshooting @scott_b!

Agreed that writing all variables when using orca.run's data_out argument is often not helpful. I'd support the adding of configuration options around this, and even making the default be that only local columns get written out.