Umm, the problems Sega and Atari hard were somewhat of similar cause:
Atari had problems due to dual management from Atari and Warner and bureaucratic red tape, but especially distribution problems. (especially inflated demand/sales figures)
The 5200, ET, Pac Man, etc were really symptoms of the conflicting management and distribution issues and overlying problems throughout the industry. (in fact Ray Kassar of Atari explicitly rejected ET when Universal asked too much money for the license but Warner then went over his head and did an inside deal with Spielberg for several million dollars and Spielberg's condition that the game HAD to be released by Christmas -albeit the contemporary Atari 8-bit computer game "ET Phone Home" was just as rushed but considerably better -I'm not sure if that made it to the 5200 or not)
By late 1983 James Morgan became president and was working hard to reform Atari and things were looking extremely promising up to the point where Warner haphazardly broke up and sold Atari in 1884 with the consumer properties becoming part of Trammel Technologies LTD. (renamed Atari Corp) Atari Corp's budget was much smaller and the staff was cut way back (especially the game programmers who were not hired at all) But Warner's random and sloppy management of the sale screwed a lot of things up.
Atari Inc then died (living on as a corporate shell for legal reasons) while TTL (renamed Atari Corp) continued to support their products, but it was a totally different company. And unlike common myth, Tramil was very intersted in continued support of Video games (he needed them to support the company), but it was a conflict with Warner (over ownership and the partnership with GCC)) that delayed the 7800 from continuing its launch and a general lack of funding that limited things thereafter.
And of course you had Commodore toppling the (already unstable) video game industry in 1983 with their price war. (not unlike what happened with Sony on the mid 90s in some respects, but very different in others -most similar due to the price war and vertical integration)
Atari very well could have continued being the market leader in spite of the crash had Warner not panicked, though the crash could have been avoided had Atari started reforming 6 months to a year sooner and/or the computer price war not occurred when it did.
The brand name was a big part of how the managed to sell so many 2600s and 7800s (and established the niche with the ST) in the late 80s and a very limited marketing budget with relatively few new games. (double the market share of Sega in North America, or more so, mostly with the 2600 but an additional 3.77 million 7800s in the US alone)
Sega had dual management issues as well and compounded with culture clash and conflicts of interest. Then there were external issues driving many other problems. (North American market slump from '93 to '97, Jaguar, 3DO, PSX, etc -in Japan you had Nintendo and NEC driving other things as well and general regional conflicts of interest)
Conflicts of interest and international markets were things that Atari didn't deal with as much, though more so for the latter (ET is an obvious example of conflicting interests), but the foreign market was something Atari generally missed out on (screwed up their chance for the 8-bit computer market in Europe and the console market in Japan by not releasing the 2600 until 1983 as the 2800 AFTER the SG-1000 and Famicom...)
The conflicts of interest were almost certainly the biggest issues Sega dealt with in general and intertwined with the complexity of dual management. Sony's presence on the market severely exacerbated those problems and more or less prevented reasonable recovery or compromise of the problems.
Nintendo, of course had different problems entirely and considerably different business practices.
That's just it: if people wanted 2D they stuck more to where cost was low with the 16-bit consoles:
With the nextgen hardware for the money being paid in most cases, they expected nextgen games: ie polygons or advanced pseudo 3D (like Doom, Duke 3D or Amok).
3D was coming one way or another and 2D would have been expected, but only for niche things like arcade ports and a few exclusive games, but mainly 3D. (and any 2D had better be stunning or at least inexpensive)
Sony pushed that further, but that was hardly the main route: if anything they pushed POLYGONS over other forms of "3D" that could have continued emerging (ray casting based height maps and voxel engines combined with scaling "sprites" and polygons), but not 2D over 3D in general.
Thinking otherwise is as unrealistic.
And most of all, "before the N64" was before the 5th gen consoles were even in the mass market: they were pretty mcuh the high-end niche while 16-bit ruled until mid/late '96 and the N64 took off right away and the PSX took off at the same time and even the Saturn's popularity increased significantly. (but market share plummeted due to the drastic rise of competition -in the US/NA at least... in other regions the N64 didn't rise quickly at all)

