Michael's RSA cryptosystem uses a large modulus n, and Nikita encrypts messages by converting characters to integers and applying RSA encryption. However, this method is not secure as it can be equated to a Caesar cipher, which is vulnerable to frequency analysis. Thus, the encryption does not provide sufficient security against determined attackers.