It's not madness at all. That's how you gain an early market advantage. At the end of the day they maintained a strong position in the game industry while introducing a format that, come a few years, will dominate the high-definition market for all optical media. And they've got the patent.
They were never going to repeat the dominance of Playstation 2 anyway. The preconditions for the launch of that console were unique. It defined an entire generation on its own. They lost huge sums of money on Playstation 2 for the first 50 or so million units that they pushed as well for what it's worth, though. It's just the way they do things in order to gain advantages. It's a risky business when it does fail, but it hasn't failed so far for them. It's gone more or less the way they hoped it would. Most of the negative effects are in the developer end, but no company that spends $50 million on a video game is stupid enough to make it an exclusive for a particular format unless they're in-house.
To return to the case of Rockstar, they heavily begrudged the fact that it was so hard to cross-develop the game but in the end they gladly did it anyway. To NOT do it would have meant that they'd miss out on millions of players and that simply wasn't an alternative. Currently Sony have shipped around 50 million consoles and they're just getting started. They've outlined sales of 20-30 million more units as the price keeps dropping and the game library expands. As a comparison Microsoft have sold a few more million consoles but they also lost a lot of money due to their support of HD DVD. And it's not like they're making any profit otherwise either as far as Xbox is concerned. They've continuously lost money ever since they decided to fight their way into the game industry.