Hi everyone, I am working on building another tutorial for my website, but this time for hash tables. My hope is that I could show simulations for deciding optimal time to double the hash array/optimal collision handling. My issue is that the space needed would explode if cadCAD keeps track of the hash table in each simulation. Is there a recommended way to handle this that anyone knows of?
Hey Sean!
Right now, there are two ways to minimize memory footprint, although both of them are a bit dirty. One is to erase the state history on each timestep, and another is to use a tweaked version of cadCAD which doesnât make deepcopies across state transitions.
For the first one, you would create a State Update Function which iterates over the state history, and deletes all the previous instances of the object. This would be as simple as something like:
ERASE_VARIABLES = [âvariable_2â, âvariable_2â]
def s_used_memory(params, substep, state_history, prev_state, policy_input):
for a in state_history:
for b in a:
for key in ERASE_VARIABLES:
b[key] = None
return (âused_memoryâ, None)
For the second one, you could install the âno_deepcopyâ branch from a forked repo (GitHub - danlessa/cadCAD at no_deepcopy). To install it, pass the following command on pip:
pip install git+https://github.com/danlessa/cadCAD@no_deepcopy
Thank you! This is exactly what I was looking for.
I believe it also works if you clear only the last set of substates after timestep zero, rather than looping through the entire state history each time:
def p_free_memory(params, substep, state_history, state):
if state['timestep'] > 0:
for key in ['state_variable_to_clear']:
substates = state_history[-1]
for substate in substates:
substate[key] = None
return {}
It works perfectly - but the cadCAD version is unfortunately stuck at 0.4.18, which has the params[0]/params[âactual param that I wantâ] problem - could you please update this to 0.4.23?
Just put the no_deepcopy patch on 0.4.23 in my fork, seems to work fine
pip install git+https://github.com/randomshinichi/cadCAD
Iâve just updated the branch to 0.4.23 btw
beautiful, thank you!
FYI: it is easier to consume that branch hack now. Thereâs a tweaked version of cadCAD with shallow copies + multi_proc + progress bars on https://pypi.org/project/cadCAD-tweaked/
To install, just pass pip install cadCAD-tweaked
Iâm going to maintain informally that package so it is always sort of up to date, but be watchful for unexpected results