In a large sauce pan, heat peanut butter, whole milk and brown sugar over medium heat stirring until brown sugar is fully dissolved and incorporated--about 10 minutes. Transfer to a large bowl/tupperware, cover and chill for at least for hours.
Stir in heavy cream then transfer to ice cream maker and freeze according to ice cream manufacturer instructions. Add vanilla at the last few minutes of churning. Once ice cream is finished churning (it should resemble at least soft serve consistency, if not firmer), stir in chopped Reese's. Cover in a freezer-safe container and transfer to the freezer to finish freezing completely.