First, it is important to mention that Calhoun was a legitimate scientist and his research, albeit controversial, was scientific in nature. Nevertheless some of the criticism presented on his work includes:
Lack of control group.
Lack of genetic diversity. The rodent populations may have been too small to have a healthy genetic pool, leading to inbreeding. Some of the behaviors, in fact, may be explained by inbreeding, for example high infant mortality rates and psychotic behavior. This however could only play a factor in half the experiments as the minimum population was too high using the 50/500 rule since 56 mice were introduced in the beginning. There were also too few generations of mice to allow for genetic drift to play a significant factor.
Space. The areas were small even for rodents.
Soon after publication, studies were conducted in dense urban environments and found that humans would react differently and even opposite how the rats reacted leaving most of these studies as inconclusive. Overall no evidence could be found of a behavioral sink in the areas researched.
https://rationalwiki.org/wiki/Behavioral_sink