OK, you asked for it! ! ! !
A few saws. I'd suggest the pull style saws which are also commonly called Japanese saws. One of the larger size Ryoba combination rip and cross cut saws is one to have to start. You'll be surprised at what you can do with such a saw used with very little effort. And these saws are common at the lumber yards these days. I suggest the Ryobi because it's all but impossible to buy a western style push saw with rip teeth any more. And you'll need to have some ripping capability.
Next up are the edge tools. Odd as it sounds I strongly suggest a small hatchet for chopping the worst of the waste wood away. Keep it razor sharp and an edge scabbard on it.
Next are a few good quality chisels. The smaller sizes are the more useful sizes. So get a set or buy then so you have 1/4, 3/8, 1/2 and 3/4. The larger sizes for stock work and smaller wood working don't get used that often but it's nice for flattening a larger area to have a 1 inch as well. But most of your work will be with the smaller sizes.
Spokeshave. A small 7 to 9 inch overall length spokeshave is one of the most commonly used tools I've got for shaping things like rifle stocks. And the style with the rounded sole comes in pretty darn handy as well. Try to get this size spokeshave with a nice smaller clean head as it gets into tighter spots than some that are larger and rather clunky with fancy adjustments. The ones I've got are made by Kunz, a German company.
A few fixed blade knives are handy as well. The boxcutter style are OK but for carving the shape and way the blades are sharpened does not let them cut out curved hollows easily. So some fixed blade hand sharpened knives is good to have. And actually they are easy to make from steel you get from old non-carbide tooth circular saw blades.
Sharpening stones. You can't have edge tools without the need to sharpen them. So a good set of stones is critical. But the stones can require a lot more skill to use well. This is one spot where I'd suggest a power tool might be a faster and easier to use option. I've got a Tormeck which I LOVE. But for casual use and a starting out soul I'll grudgingly suggest that the cheaper version of this wet grinder and leather wheel machine from Grizzly is the way to go.
Notice I didn't mention wood planes? For making rifle stocks you really don't need them much. But for any other sort of wood working you very seriously do need a set. So figure out how you want to go on that for yourself. If it's strictly stocks then maybe just a small block plane. But if you wish to do much else you'll need a wider variety.
You'll obviously need a way to drill holes. You likely have a hand drill and bits. But for rifle stocks you are likely to find that a set of forstner bits is something to get. The clean sided and flat bottom holes work exceptionally well for the initial wasting out work on lock mortising jobs.
You also need that all important "third hand" known as a work bench and vise. For the hand tool wood worker (HTWW) this does not entail a simple table and machinist's vise bolted to it. The HTWW's bench needs to be solid, heavy and most of all rigid so that when you hit something the bench doesn't bounce. And when you cut sideways with a saw or chisel the bench doesn't flex or lean. This calls for a thick top and a sturdy connection to legs that don't want to flex.
What you make it from depends on what you have for tools. A bench done from two or three layers of 3/4 fir plywood is superb. But a great work bench can also be made from heavy dimensioned building lumber. It all depends on what you're more comfortable with using. But as long as it's strong and heavy and uses good solid connections that won't wrack you'll have a good bench.
That bench is no good if it can't hold the work pieces in a variety of positions. So things like vises and hold downs are built into the bench to give you that important "third helping hand".
Notice that I haven't said much about some other accessories such as hammers or mallets. That's because much of this is stuff you can make from wood along the way with your tools. And making these tools as you go gives you more skills and confidence along the way. And there's something nice about using the tools you made to make the projects you enjoy. Even to the point of your set squares being shop made from wood so when cluttered on the side table with the edge tools they don't ding the cutting edges like metal tools would do.
I'm going to stop now. But here's a few random video links to whet your appetite and hopefully give you a peek into the hobby of HTWW. Some of them are simple things. Others look like I'm expecting you to perform brain surgery. But they all show how possible it is to attack even a big job in small bites. None of them are intended to get you started but some of them you might find applicable to your beginner's status. The key is that they all have one thing in common, namely very little electric power tool use. I hope you enjoy them. Don't see them as what I'm directly suggesting to you. Instead look at then as how broad the possibilities are and don't be afraid to adapt the techniques into your own present tools and preferences. For example, all those nice laminated lumber bench tops you are going to see are easily replaced with two or three layers of 3/4 ply glued and screwed together. Just remove the screws afterwards as you go so when you drill for bench dogs and other accessories you don't hit the screws.
https://www.youtube.com/watch?v=Z4LohjmskEk
https://www.youtube.com/watch?v=u31Ixu6mSHY
https://www.youtube.com/watch?v=yvhn-PAfEW4
https://www.youtube.com/watch?v=Bmg1gvqZ2mQ
https://www.youtube.com/watch?v=NE_FGdrX0K4
https://www.youtube.com/watch?v=lQWY7Pi4v-M
https://www.youtube.com/watch?v=x5tMPhHQYbA
EDITED TO ADD ONE MORE- A nice pragmatic down to earth discussion on the English or "Nicholson" style of bench. This style gains strength, stiffness and more than most other options SIMPLICITY and LOW COST from the use of deep supporting aprons. Some 2x4's for the lower framing and cross members between the aprons and two or three 2x10's or 2x8's for the top and you're off to the races.
https://www.youtube.com/watch?v=E8nQpYXT8V4