If I were starting up on a budget, here's how I'd do it.
I'd buy a hand-cranked conical burr grinder like Hario's
Skerton or
Mini Mill ($35-40). I think this is probably worth the premium over a blade grinder ($20), and worth the savings over an
inexpensive motorized model (minimum $90). The physical effort is also probably worth because of quality increases over either cheap flat burr grinders ($35-60+) or preground coffee. The only reason I'd do something else is if you're planning on making coffee for a crowd (in which case grinding by hand might take too long). Since it's sounds like you're planning to brew mostly for yourself, this should work find.
I'd also buy an accurate thermometer, if you don't already have one. Depending on your stove, your kettle, how much water you're heating, and other factors, guidelines about how fast water cools can be woefully inaccurate. Since water temperature makes a huge impact in how coffee tastes, this is really an essential tool. Plus, you can use it all around the kitchen, so don't think of it as a coffee-only purchase.
There are plenty of worthy options for brewing equipment, almost all of which are $25 or less: french press, pourover, aeropress, etc.
French press will give you a heavy, full-bodied cup, with the option to make multiple cups at once. Pourovers, which make a clean, lighter-bodied cup, come in two varieties: restricted opening ones, like the
Melitta, and unrestricted openings, like
Hario and
Chemex. The former gives the brewer less control, which limits both the peaks and valleys of coffee quality. With a little practice, the Hario/Chemex brewers can make extraordinary coffee, but since the brewer is in control of the flow rate, making underwhelming coffee is also possible. Most of the pourover methods work best when brewing only a couple cups at a time. The
aeropress is a unique one-cup brewer that makes coffee that's maybe a bit like an Americano (watered down espresso, which isn't a bad thing). If you like espresso drinks at the coffee shop, it may be worth consideration; it gets a lot of love. All of these methods take some technique and have different quirks in terms of how much attention the need, how fast they are, how much coffee they make, etc. Take a look around, and watch some demo videos to get a feel for what it's like to use each of them.
If you're going to drink a few cups over the course of the morning, you might want to get a stainless steel
thermos to store the hot coffee.
As far as beans go, if you can find a good local roaster (local does not always guarantee good), that's probably the best option. Most local roasters will also be able to offer tips on equipment, brewing technique, and bean selection. Buy your beans fresh (within a week of roasting), and store them in an airtight container. Mail-order is find too. Feel free to buy a couple pounds at a time. When they arrive, put all but one of the bags in the freezer. By storing coffee still sealed in the bags, you'll avoid the commonly cited pitfalls of freezer storage (condensation, flavor pickup) and the freezer will keep the coffee fresh, allowing you to buy 2+ lbs at once (don't believe me? Read
this). Just allow the beans to come to room temperature before opening a new bag to avoid condensation. Keep the in-use coffee in an airtight container (not just in it's bag with a clip on it). The "best" coffee these days goes for something in the ballpark of $17-22+ per pound. You don't have to spend that much, but you can use it as a guide--I probably wouldn't spend less than $13/lb or so, but I haven't really explored the offerings in that range.
(Fwiw, I use a
Capresso Infinity grinder and a french press, with beans (~$20/lb) from
Barismo here in Arlington MA. I make about 28oz of coffee each morning for about $0.70 per 12oz cup.)