I'm pretty sure you're wrong. These asymmetric crypto keys, with which it takes a long time to encode/decode, are used only at handshake to securely negotiate a per-session key that's used in symmetric crypto, for which encoding and decoding is extremely fast.