R'Zusha explains (see Menorat Zahav, Parashat VaYigash) that it is because you are sitting in a "moshav leitzim." The imprint left on that place by other Jews who spent time in that place without learning Torah.
It's a basic teaching of the Baal Shem Tov (and if you stop to think about it, Chz"l in general, see Parashat Noach) that our actions have a profound and lasting effect on our environment. When we invest our time and energies in meaningless pursuits, that leaves an impression in that place. For someone else, to occupy that same space and then perform mitzvot is more challenging because of us.
Wow that's a lot of responsibility. But that's what I like best about Hassidut, with great responsibility comes great power. It means the reverse is also true. Want to ensure that it's easier for another Jew to learn Torah somewhere or do mitzvot in some place? Do a few mitzwoth yourself, learn a little Torah there and you've now made their lives a little easier.
Spread the light.