I had not seen such a bad signal yet in my digimode life. 30m, BPSK63, 2009-06-18. I did everything on my FT-817, IPO, ATT, volume control but to no avail: I could still see the signal twice and with many distortion products.
I worked him with 1W, so his signal was indeed very strong. I mentioned the bad IMD (-15dB) but nothing changed. I gave RSQ 555 but should have been 591.
A couple of hints:
Make sure the ALC is not active when transmitting. With ALC=0, just below its ALC>0 point you are already transmitting at full power. After that ALC reduces your audio drive, creates non-linearities in the TX chain and sends out splatters. Moreover with ALC>0 your measured power is distributed amongst all unwanted signals, thus you actually reduce your main signal power (see below)!
Transmit above 1500 Hz in the waterfall. This helps not to send the signal twice. If you sit at around 1000 Hz, your TX SSB bandpass filter will let through 2000 and 3000 Hz too, so you'd transmit any harmonic generated within your computer of first TX audio stages. Better stay between 1500 and 2000 waterfall Hz, and retune your RTX main dial. This is a clever hint that I have not seen mentioned anywhere.
"below". A proof that his distorted 50W did no more than my 2.5W comes from pskmap: we both reached the same stations, including one in the US East Coast.