It's extremely unlikely that Nintendo developed their own codec. It's fairly standard Miracast stuff. Nintendo has no expertise (or call) to do such a thing.
I'm pretty sure i said Nintendo
must have developed their own proprietary codec to reduce the bitrate and latency further (they have the money and can hire engineers to do so). I have come to this conclusion because I have noticed that WiiU latency is lower than standard Miracast when compared to Miracast on Phones I have used.
I don't understand where you are getting 640x480 from. That's SD rez and besides not even the right aspect ratio for Vita. Surely with dedicated hardware and 802.11n at their disposal, Sony can stream qHD to the Vita with very little problem.
I'm sure you understand that I'm talking about Remoteplay on PS3 and Vita and not PS4. Current remoteplay on Vita and PS3 runs at PSP native resolution then, upscaled to PS Vita native resolution. Yoshida clarifies in a statement in 2012 that titles designed around the constraints of PS3 to support remoteplay will be able to stream a 480p video (could be 854×480 which is the same aspect ratio as PS Vita) then upscaled to PS Vita rather than the PSP's 480 x 272.
I envision remoteplay on PS4 to be greatly improved by;
1. Just the fact that PS4 and Vita uses Wifi N improves remoteplay as it is.
2. The fact that PS4 has a dedicated video encode and decode hardware that does video encoding on the fly without any lag or resources needed from the APU also improves latency a lot.
3. Remoteplay on PS4 now also uses Sony's Gaikai underlying technology which means Remoteplay over internet will have an improved latency.