# Scalable Bayesian models

**URL:** <https://community.intuitivebayes.com/t/scalable-bayesian-models/331>\
**Category:** Causal Inference Book Club\
**Created:** [July 11, 2022, 11:08am UTC](https://community.intuitivebayes.com/t/scalable-bayesian-models/331 "2022-07-11T11:08:36Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![rahulB](https://yyz1.discourse-cdn.com/flex007/user_avatar/community.intuitivebayes.com/rahulb/32/140_2.png) [@rahulB](https://community.intuitivebayes.com/u/rahulB)\
**Post date:** [July 11, 2022, 11:08am UTC](https://community.intuitivebayes.com/t/scalable-bayesian-models/331/1 "2022-07-11T11:08:37Z")

</div>

Hi all,

At our organization we use R’s Lme4 for hierarchical mixed effects models. This requires the whole data to be loaded in memory. The data is quite huge and generally the lowest granularity (which is a random effect) has to be averaged. There’s also a push to adopt Bayesian approaches to generate credible intervals as they make most sense to business.  
Given these requirements I was wondering if PyMC/Bambi or tf-probability does batch processing and can scale well with big data?

---

<div class="post-metadata">

**Author:** ![Nforde](https://yyz1.discourse-cdn.com/flex007/user_avatar/community.intuitivebayes.com/nforde/32/44_2.png) [@Nforde](https://community.intuitivebayes.com/u/Nforde)\
**Post date:** [July 11, 2022, 2:50pm UTC](https://community.intuitivebayes.com/t/scalable-bayesian-models/331/2 "2022-07-11T14:50:06Z")

</div>

I’m not sure if this is exactly what you were thinking but i saw this announced in the pymc experimental repo the other week: [pymc-experimental/histogram\_utils.py at main · pymc-devs/pymc-experimental · GitHub](https://github.com/pymc-devs/pymc-experimental/blob/main/pymc_experimental/distributions/histogram_utils.py)

I’ve been meaning to try it, but if i understand the idea correctly i think you can sample from a histogram of your large data. Kind of like a bag of little bootstraps design…

---

<div class="post-metadata">

**Author:** ![Nforde](https://yyz1.discourse-cdn.com/flex007/user_avatar/community.intuitivebayes.com/nforde/32/44_2.png) [@Nforde](https://community.intuitivebayes.com/u/Nforde)\
**Post date:** [August 18, 2022, 7:59am UTC](https://community.intuitivebayes.com/t/scalable-bayesian-models/331/3 "2022-08-18T07:59:07Z")

</div>

Just following up here. I wrote up an example of using this technique here:

> **[Bayesian Estimation and Histogram Approximation](https://nathanielf.github.io/post/bayesian_ab_tests/)**
>
> Imagine you’re pretty confident of your recent bet. All the other gamblers at the table have taken the other side of the bet. Just for fun, assume all other gamblers in the casino have bet against you. The game is in the first quarter and things are...

---

<div class="post-metadata">

**Author:** ![LarsKar](https://yyz1.discourse-cdn.com/flex007/user_avatar/community.intuitivebayes.com/larskar/32/211_2.png) [@LarsKar](https://community.intuitivebayes.com/u/LarsKar)\
**Post date:** [August 19, 2022, 7:30am UTC](https://community.intuitivebayes.com/t/scalable-bayesian-models/331/4 "2022-08-19T07:30:14Z")

</div>

Hi, maybe this is what you are looking for? [pymc.Minibatch — PyMC 4.1.4 documentation](https://www.pymc.io/projects/docs/en/stable/api/generated/pymc.Minibatch.html)