Interaction of polarization mode dispersion and nonlinearity in optical fiber transmission systems