Vitex actually is pretty well known to affect prolactin levels. The relationship to libido then is somewhat indirect and probably depends on your original hormonal situation. For example, from wikipedia:
Prolactin provides the body with sexual gratification after sexual acts: The hormone counteracts the effect of dopamine, which is responsible for sexual arousal. This is thought to cause the sexual refractory period.[4] The amount of prolactin can be an indicator for the amount of sexual satisfaction and relaxation. Unusually high amounts are suspected to be responsible for impotence and loss of libido (see hyperprolactinemia symptoms).
I personally think my sex addiction is related to no having enough prolactin production. For example, I can orgasm multiple time, with no break in between. I basically have no refractory period and can orgasm (full ejaculation) up to 11 times before I start to lose desire.
You can also see how drugs used to combat high prolactin levels (like bromocriptine) are also known to increase libido and male sexual function generally:
http://www.biopsychiatry.com/cabergolinesex.htm
You can also read about how prolactin causes loss of libido:
http://www.onlinedatingmagazine.com/col ... actin.html
The interesting thing is that prolactin basically competes for dopamine, so is very intertwined with our pleasure-seeking, and seems to be the hormonal control -- prolactin is a major factor in why humans don't just have sex constantly, it creates the satisfied feeling.
Anyway, the point then is that Vitex can affect prolactin levels. You'll see varying contradictory evidence saying that it either raises or lowers levels, but most interesting is that many herbalists say that it "normalizes" levels. In other words, if you have low prolactin it may raise it but if you have high prolactin it may lower it.
Anyway, this is the theory of Vitex. It seems to affect prolactin, and prolactin is related directly to sexual satiation.
I personally find that Vitex helps a lot. It is very cheap.
I really don't know about long term effects and recovery because I haven't stopped taking it since I started. A couple times I started to stop but then after a couple days realized I was getting too horny again and so I started again.
I can still be sexual with Vitex, but it is just not an overwhelming urge.
I am very sure that Vitex is doing something because for the first several days you take it your testicles will ache, and you'll get hot flashes. I've never had hot flashes before, and they really surprised me where I would be sitting at work or wherever and my face would feel hot and suddenly I would be sweating lightly all over my body. Lastly, the feeling in my genitals and my nipples changed a lot -- they are not really erotic anymore. They are not numb, but they are now just like any part of my body. This is a big deal for me as my libido was so high that I would masturbate multiple times per day, and also would get unwanted erections throughout the day. I also normally have very quick ejaculation and also have the multiple orgasms I mentioned before. Now I am much slower, with more endurance.
I do worry a bit about the long term effects of taking anything. But for me the long term effects of high libido were even more dangerous.
I know I didn't answer your question about trying it, but it is definitely something that is cheap, does have some effect (in my opinion, for some people like me), and is regularly taken by pregnant women so can't be that dangerous.