Reposed from @mzargham’s Quests for Quests, aka more Demo Models post. The goal of this thread is to offer the outline of a good Quest so that community members can create their own quest proposals.
Quest Outline
Part 1: The Word Problem
This section needs to sound like a user story, with some extra information about assumptions we’re making. "As a data scientist, … " or “As a token engineer,…” I am faced with a question, and I need to make some decision, etc. At This is generally 2-3 sentences.
Part 2: Formal Specification
If we’re asking someone to do the coding we need to provide them with some specs. In this case this should include listing out the policies, and mechanisms that need to be modeled. I make a variety of different diagrams when mapping systems and which types fit vary by problem statement. Marketing funnels and bonding curves are both pretty well characterized by system dynamics style stock and flow drawings because the conservation laws dictate most of the systems structure.
Details:
- any helpful diagrams
- a list of mechanisms and policies
- equations for each mechanism and policy introduced
- a list of state variables aka “statespace” in the model (this is a data structure)
- a list of parameters in the model (also a data structure); generally its good to clarify which variables are under the system designers control versus when are uncontrolled (eg slashing fee as a percentage of stake) and thus are “environmental” processes (eg, non-strategic failure rate of honest nodes in a peer to peer network)
- define the KPIs, these are metrics which can be computed from the other data in the simulation
- it may also make sense to recommend parameter values or initial conditions values for the state variables.
Part 3: The write up
These quests are not just to run simulations for simulations sake, they exist to answer real world questions. That is why we started with the world problem. Part through is about how the quest doer should organize their results and present them in a notebook.
Details
- restate the question. The reader should not need to go back to the original quest or issue to read the notebook
- describe the model. Again, the model may be heavily defined in the quest BUT interesting things could have happened, maybe you learned something interesting or needed to make some tweaks so the write up needs to state the model spec as it is implemented (whether or not it has changed from the quest).
- Describe in words the experiments being run. Are we parameter sweeping, A/B testing or just running Monte Carlo experiments to demonstrate a property holds. Make explicit any assumptions made, and explain why you made that assumption as well as what implications the assumption might have on any conclusions taken from the analysis.
- Describe the plots provided. For the quest writer, be clear about the types of plots expected, as the quest doer make sure to label the data and axis, as well as explain what the plots tell us relating it back to the word problem.
- finally, its always good to talk about “next steps” how this analysis fits into a broader set of questions or design challenges faced in a practical setting. What new questions arise as a result of this analysis. What might one want to do next.
If you have any questions about the process or would like to suggest improvements to this guide please comment in this thread!