class: center, middle, inverse, title-slide # SBC Intro ### Martin Modrák ### 2021/08/24 (updated: 2022-01-26) --- <img src="elixir_logo.png" width="780px" height="auto" /> This work was supported by ELIXIR CZ research infrastructure project (MEYS Grant No: LM2018131) including access to computing and storage facilities. --- # Model workflow - Prior predictive checks -- - Validate computation -- - Evaluate model fit to real data - Posterior predictive checks -- - Use the model --- # Model workflow - Prior predictive checks - Validate computation **⬅ SBC lives here** - Evaluate model fit to real data - Posterior predictive checks - Use the model --- # SBC Context -- - SBC is a tool to check you implemented your model correctly. -- - It just one of tools to validate your model in a Bayesian workflow -- - SBC can be run even before you collect data -- - The goal of this tutorial is to show you that with the `SBC` package, there is little cost to including (some form of) SBC in your everyday modelling workflow. -- - In this tutorial, we'll let you use SBC to find and diagnose a bug. --- # Two types of problems with Stan model 1. Bug in model -- 2. Data - model mismatch -- We will focus on 1. --- # Simulation to the rescue! -- How to seperate bugs from model-data mismatch? -- - If we simulate data _exactly_ as the model assumes, any problem has to be a bug. --- class: split-two .column[.content[ ## Stan ```stan data { int<lower=0> N; vector<lower=0>[N] y; vector[N] x; } parameters { real a; real b; real<lower=0> sigma; } model { vector[N] mu = a + b * x; a ~ normal(5, 3); b ~ normal(0, 1); sigma ~ normal(0, 2); y ~ lognormal(mu, sigma); } ``` ]] .column.bg-gray[.content[ ## R ```r # N and x is not modelled, # choose to match your actual data N <- 10 x <- rnorm(n = N, mean = 0, sd = 2) a <- rnorm(n = 1, mean = 5, sd = 3) b <- rnorm(n = 1, mean = 0, sd = 1) sigma <- abs( rnorm(n = 1, mean = 0, sd = 2)) mu <- a + b * x y <- rlnorm(n = N, meanlog = mu, sdlog = sigma) ``` ]] --- # How to spot failure? ``` variable mean sd q5 q95 alpha 0.372 0.224 0.00794 0.737 ``` Is this OK, if we simulated - `alpha = 0.3` -- - `alpha = 0.007` -- - `alpha = -30` -- Diagnostics (divergences, Rhat, ESS) # How to spot success?! --- # Consistency requirement "In 95% of simulations, the true variable lies within the central 95% posterior credible interval." - Variable = parameter or a function of parameter(s) -- "In x% of simulations, the true varible lies within the x% posterior credible interval (of any kind)" -- ## SBC for single parameter: -- 1. Thin posterior to get `\(S\)` independent samples. -- 2. For each simulation take the rank of the true value within the samples - Rank: no. of samples < true value -- 3. Across simulations, this rank should be uniformly distributed between `\(0\)` and `\(S\)` --- class: split-three .row[.content[ # SBC visualisations - Ranks ]] .row[.content[ <!-- --><!-- --><!-- --> ]] .row[.content[ <!-- --><!-- --> ]] --- class: split-three .row[.content[ # SBC visualisations - ECDF a.k.a. "black line outside blue region is a problem" ]] .row[.content[ <!-- --><!-- --><!-- --> ]] .row[.content[ <!-- --><!-- --> ]] --- class: split-three .row[.content[ # SBC visualisations - ECDF diff a.k.a. "black line outside blue region is a problem, rotated for readability" ]] .row[.content[ <!-- --><!-- --><!-- --> ]] .row[.content[ <!-- --><!-- --> ]]