They also won’t ever want sex
This is not true. Historically, "Many castrati lived rather promiscuous lives. Because their unions could produce no embarrassing offspring to explain, women saw them not only as beautiful, ethereal celebrities of the opera stage, but as prime candidates for affairs."
their penises will never grow to adult size
I bottomed for a trans women with a highly atrophied penis and had a great time. Tbh I don't know why anyone cares about penis size.
their voices will always sound like that of a little boy.
Their larynx won't drop, but they won't sound like little boys. Only some features of adult male voices are caused by biological puberty—there are a lot of other vocal characteristics that distinguish adults from children and men from women. I've known trans men who hadn't gone on testosterone who still had masc-leaning voices. And adult women sound different from little girls even though their voices don't drop during puberty.
Now, maybe you’re lesbian or asexual or something, and you’re fine with all that, but I imagine some women would not be pleased.
I'm bisexual, and I'd be totally down to date a guy who looked like this. As pointed out above, castrati were highly sought-after by women in the past. Obviously contraceptives negate the appeal of their infertility, but still, they must have been otherwise sexually appealing enough for women to want to sleep with them in the first place. Sure, it would be a loss for those who only like super-manly men, but that seems like an acceptable trade-off to me.
That's true. I don't see what it has to do with my argument, though. I'm pretty sure that testosterone increases risk-tolerance, and that's part of why it correlates with aggression. Are you suggesting that men have elevated risk-tolerance for reasons other than testosterone, and that risk-tolerance is responsible for aggression instead of testosterone? Or are you saying that risk-taking is important so it's worth keeping men the way they are even if it causes most serious crime?
Most people see violent crime as a problem, but few see it as a problem with men. When people discuss crime, I never hear them frame the problem as "there's something causing men to commit 10 times as much rape and murder as women: what is it and how do we stop it?" Even feminists who talk about male violence generally don't frame it that way.
No empirical data can lead us to accept something as "necessarily true," but it stretches credulity to think that castration would reduce aggression in pretty much every kind of male mammal we try it on except humans and further that the most aggressive humans coincidentally have elevated testosterone levels. I don't think that you actually believe that, since you said:
I specifically listed the other effects I could think of. If you think something else bad might happen, just say what it is. If your objection is that we should be cautious because there might be unexpected effects... well sure, that's true, but it's also a general-purpose objection to any suggestion to change anything ever. You can't really have any interesting opinions if you accept that reasoning.
I'm in favor of that. But I think there's a limit to how much you can improve society via culture alone. You could probably design a culture where people would be a lot less selfish than they are today, for example. But I don't think you could get people to never be selfish at all, because some amount of selfishness is part of human nature. I think the same is true for aggression, and that the minimum amount of aggression you could get from people is in large part of function of testosterone levels.
Furthermore, "make society less toxic" is a goal, not a policy. A policy to reduce violence by making society less toxic could be something like teaching children to play cooperative games instead of competitive ones. That would probably have a small effect in a few decades. But I think chemically castrating men would have a bigger effect in a shorter amount of time than just about any other policy you could think of, and those effects would be in addition to anything else you did.