Audiovox has an affordable MP3 player with a 512MB capacity for only $19.99, losing the frills found in many of the latest media players and providing bare-bones portable tunes.
The Audiovox MP6512 supports MP3 and WMA (and protected WMA) files, has a nice OLED display that shows the track playing, and a fairly small case measuring 2.4 inches long and .8 inches thick.
Transfer the music files via a USB connection to your computer, and I guess I was a little off with the bare-bones comment. The MP6512 does have an built-in equalizer with presets for rock, jazz, classic, and pop.
Throw in 10 more dollars and you can get a 1GB version with an FM tuner with the MP6610FM.