The Noam Elimelech has a great answer for me:
What exactly was Kayin's sin that he offered up fruits of the earth, he was a farmer? The problem is, that HaShem created many many worlds and it is our job to connect them all back to HaShem, to relate to HaShem through them all. It isn't enough to say simply, I know and believe that God exists, for what do I need all these worlds? Kayin wanted to do that, he said, I'm a farmer, I rely on HaShem for everything He provides, and I will relate to Him on that level.
This isn't enough. All the worlds and everything in creation was created for the sole purpose of our connecting to and interacting with HaShem. We need to relate to Him via all available media.