Second Layer is the best one I've been to. Pretty far out on the tube (Highgate), but once you get off at the station it's only a 2 minute walk. I usually visit Sister Ray and Sounds of the Universe as well whenever I'm in London. The latter two are all close to each other, on/around Berwick St, which also has a few second hand stores (Music and Video Exchange, Reckless Records). It's also just a few minutes walk to Poland St (mentioned in blunderbuss/Genteel Death's posts) and there's a big corporate HMV close by at Oxford St, which you might as well pay a visit as well.
Another one not yet mentioned is Honest Jons on Portobello Road. Not sure if it would suit your tastes, but have a look at their website to see if it does:
http://honestjons.com/shop.php
(someone told me Damon Albarn of Blur is the owner???
wtf, etc)
It's also just a 5 minute walk from Rough Trade West, which is the less hipster-ish counterpart of Rough Trade East. The East store has a better stock, but it's also the most laughable hipster record store I've ever been to. When you walk in you'll see some ridiculous bicycle rack to your right (bicycles seem to be a trendy thing in London?) and to your left there will be some 18-year old girl wearing a Black Flag shirt standing behind the coffee bar making espressos.