A good place to start is maybe Amon Duul II's Yeti album. Can, Faust and Neu are great too, but you might get a false impression of what most krautrock actually sounds like. Yeti, on the other hand, is just pure Kraut.
Another good way of getting some background to the music, as well as some pointers regarding which albums to go for, is Julian Cope's book Krautrocksampler.