Latency is not in itself an impediment to a successful VoIP conversation; latency is just uncustomary for humans, and causes them to talk over each other not unlike satellite or microwave long-distance telephone conversations between hemispheres.
The problem that is often present in connection with the underlying causes of high latency but is a different phenomenon than latency is jitter. Jitter is the presence of nonlinear temporal deltas between the arrival of RTP packets. RTP arrival at non-constant intervals causes audio distortion, "drop-outs," static, repetition, and stream synchronisation problems.
The bottom line: if there is high but very constant latency, you can still have a manageable conversation. However, the same things that cause high latency -- for example, link congestion -- often cause jitter. The degree of packet queue saturation and/or bandwidth exhaustion often varies across time and creates stochastic, indeterminate effects that jitter buffers cannot be designed around, and certainly not perfectly at any rate.
I would bet that jitter is the cause of your problem, simply because it often goes hand-in-hand with latency.