Cartilage repair by bone marrow derived mesenchymal stem cells (MSCs) can be influenced by inflammation in the knee. Next to synovium, the infrapatellar fat pad (IPFP) has been described as a source for inflammatory factors. Here, we investigated whether factors secreted by the IPFP affect chondrogenesis of MSCs and whether this is influenced by different joint pathologies or obesity. Furthermore, we examined the role of IPFP resident macrophages. First, we made conditioned medium from IPFP obtained from osteoarthritic joints, IPFP from traumatically injured joints during anterior cruciate ligament reconstruction, and subcutaneous adipose tissue. Additionally, we made conditioned medium of macrophages isolated from osteoarthritic IPFP and of polarised monocytes from peripheral blood. We evaluated the effect of different types of conditioned medium on MSC chondrogenesis. Conditioned medium from IPFP decreased collagen 2 and aggrecan gene expression as well as thionin and collagen type 2 staining. This anti-chondrogenic effect was the same for conditioned medium from IPFP of osteoarthritic and traumatically injured joints. Furthermore, IPFP from obese (Body Mass Index >30) donors did not inhibit chondrogenesis more than that of lean (Body Mass Index <25) donors. Finally, conditioned medium from macrophages isolated from IPFP decreased the expression of hyaline cartilage genes, as did peripheral blood monocytes stimulated with pro-inflammatory cytokines. The IPFP and the resident pro-inflammatory macrophages could therefore be targets for therapies to improve MSC-based cartilage repair.