This paper analyzes nonlinear signal interference to optimize coherent optical transmission systems (COTS) using PM-16QAM modulation for various baud rates and silica fiber. It employs Gaussian noise models to improve accuracy and performance in determining the optimum optical signal-to-noise ratio (OSNR) and bit error rate (BER) through simulations. Results indicate that OSNR penalties increase with higher line rates, emphasizing the importance of capacity enhancements and spectral efficiency in optical networks.