The System

You can collect records before you have a stereo system, but there really is no point if you aren't going to listen to it.  Maybe if you had a really rare record worth $15000, that might be different, but I would still listen regardless, just like I would drink a $10000 bottle of wine if I had the chance. 

So the first thing you should probably do, as I did, was decide on a system to listen to your records.  It can be simple - turntable, receiver, speakers.  It can also be more complicated - new or vintage turntable, upgrade the tone arm, the weights, the headshell, the cartridge?  Install a new cartridge yourself and you need the tools - the tracking weight scale, a protractor thingy, tiny screwdrivers and pliers.  Then the complexity of finding the right components to plug your source into.  You can go with an integrated amplifier (pre-amp and amp in one) or receiver (integrated amp plus tuner) or you can go with a dedicated pre-amp and power amp.  Then you also need to decide to go new, or go vintage. Then the same with the speakers, and connectors, and tweaks (turntable mat, anti-vibration feet, etc etc.)

For me, I have a tight budget (for now), so I found a pretty good deal on Kijiji for a bunch of components.  This is what I got:

NAD 2150 Power Amp
NAD 1020A Pre-Amp

I also got a Pioneer 6 CD player, which I kept, as well as an equalizer, NAD tape deck, and tuner.

A week or so later, I found another deal on Kijiji for a turntable.  I wanted a vintage look, at a decent price, that was decent quality.

I ended up getting a Realistic R8000 belt driven turntable.  This unit was made in 1977 by CEC, which is a Japanese company that also manufactured Pioneer and Marantz brands. Although Realistic is a Radio Shack brand name, it still shares a lot of the same components as other brands that would be considered higher quality. The same thing happens today, with Samsung making several iPhone components.

In the end, I think it depends on budget mostly, and what you are looking for in a system and sound.  Every component will add or subtract something from what the true sound of the recording is, and it comes down to personal preference.

A lot like most people don't know what they want to do for a career until they try many out, you probably don't know what sounds best to you until you try various components.

This will be a very expensive and long lasting hobby.