And I'd argue that was where they went wrong. I remember saying Nintendo were in for a rocky time with their next console due to the wii (before we knew about the Wii u). People thought I was crazy, but it seems as though I was right.
Even if it wasn't a run away hit, the Gamecube at least tried to appeal to core gamers. Whereas, the Wii, actively shunned them for a new direction on non gamers. There's nothing wrong with that of course, but that market is very difficult to keep interested, and as soon as something more shiny arrived (tablets/smartphones), Nintendo lost their audience. Their next console needed to get those they lost back on side, but as soon as it was seen, people saw straight through their idea as another crack at the casual market.
It was a huge mistake by Nintendo.