negative binomial likelihood function