I thought the second analog stick was a key feature that many people felt the PSP lacked, that the Vita provided. But aside from that, I think a lot of people are rightfully turned off by the expensive proprietary "memory sticks" that Sony requires.
I'm not sure about a black-and-white handheld from Sega. The only way I could see it having some kind of additional appeal beyond the Gameboy is if the screen would be a lot more clear, but that gets back into consuming more batteries. I think the only way to beat the Gameboy with a black-and-white handheld would have been to be first to market. But at that time, Sega had no third party support. In the end, the Game Gear did pretty well, all things considered - it sold decently, and smartly expanded upon the existing 8-bit library and infrastructure.