likelihood function of negative binomial distribution