Accumulating evidence suggests that the immune system shows subtle but relevant differences between men and women. These differences may have an impact on cancer development and TME composition as well as responses to and adverse events elicited by immunotherapies. Several, albeit not all, clinical trials indicate a greater benefit from mono-immunotherapies over chemotherapies for male patients than for female patients, especially in non-small cell lung cancer and melanoma. Vice versa, female patients might benefit more from chemo-immunotherapies. In human as well as animal models, sex differences in cancer microenvironment composition were described, with partially divergent results. Sex-specific factors such as the levels of hormones, in particular testosterone and estrogen, or X- or Y-chromosome associated genes are likely to drive the observed differences, but are often confounded by external influences such as smoking behavior, diet, or UV exposure. Therefore, large clinical and mechanistic knowledge gaps remain regarding the influence of sex on cancer immunotherapies and strategies to optimize response in either sex. More clinical as well as experimental research in this field is required to close these knowledge gaps, and clinical trials should include large enough groups of male and/or female patients to allow robust sex-specific analyses.