Looks like I have finally gotten around the proper deviation for a 50 MHz FM TX, but many problems still have to be tackled down.
I tried to "select" the desired harmonic with the RLC circuit found on a paper online and I had to play a lot with L and C values. The L was wound on a T50-6 core, while C was a 3-16pF trimmer. On the frequency meter I could read 2x XTAL fundamental and a weird 2.5x, depending on trimmer position. After removing almost half of the originally planned turns and making extensive use of the oscilloscope, the output frequency is now steady on 50 MHz.
I even noticed that the frequency changed when loading the "PA" (note that a buffer gate is between harmonic selector and PA), so I did the fine tuning with a 7dB 50ohm attenuator pad and a 50 ohm dummy load.
On the oscilloscope I can read almost 3V peak on the dummy load, which is about 80mW or +19dBm. Given the 7dB pad ... +19+7 = 26 dBm that is 400mW of RF. Not bad for a 74HC240 chip pulled at its operating limit!
In order to minimize stray capacitances I moved components around and now both AF and RF sit on the same copper ground.
It's time to add an LPF and ask for an online report! After a check at the spectrum analyzer, of course!