Modern Music is my favorite place to buy new stuff, despite being incredibly expensive. I think it's actually more expensive to buy PSF stuff at their own store than it is to buy it overseas. It's located near Meidai-mae station on the Keio Line.
http://www.psfmm.com/page/2
For used stuff Disk Union is my favorite. There are a couple great Disk Unions around the east side of Shinjuku station. Walk around there long enough and you'll find them. The Disk Union in Kichijoji is pretty excellent too, especially for used experimental stuff.
Some other places to look around are Shimokitazawa (Odakyu or Keio Lines) and Koenji (Chuo Line). I can't recall any names but there are definitely some good stores around there.