Here is what I believe is the problem with such thinking in nearly ANY setting. If you are viewing YOUR energy conservation measures at an individual scale then you are going about it in the wrong way. Yes, if you look at energy reductions at the INDIVIDUAL level then the reductions are so minuscule that you can probably discount them. You just can't view the situation in such a manner. You have to view your reductions as part of a group effort that includes many other like-minded individuals such as yourself that are ALSO reducing their energy. Taken together, this group of individuals can be making a big dent.
The same concept can be applied in a green buildings setting. One building getting LEED certified and going green might not make much of a difference in a city-wide setting but if a GROUP of buildings do this or if you add up all the green buildings together in a city then you realize how big of a difference that building, as part of the larger group, is making.
What I am trying to get at is that in issues such as these, you have to view your improvements and behavior in context and consider yourself as part of an overall GROUP and community effort. Viewed in this context, your actions DO matter because if everyone thought the way you did then everyone would say "well, it doesn't matter and I personally can't make a large dent" and basically not even try. If this ever happened, environmentally conscious behavior would be non-existent.
For instance, I am starting a drought resistant garden and I am going to remove our house's front lawn and replace it with drought-tolerant plants. This new setup is going to save hundreds of gallons of water per year. If I look at these savings in an individual context, I am basically doing NOTHING because California uses up water in the billions (perhaps trillions) of gallons a year. But, I realize that there are MANY others like myself who have replaced their lawns and are saving hundreds of gallons as well. Add up enough of these people together and you have already reached hundreds of thousands of gallons in savings (or perhaps more).
Another way to consider the situation is to look at the effects your behavior might have on others. Once again, you aren't operating at an individual level. In my example, my neighbor might see our new front lawn and come admire it. Perhaps he never realized that drought-resistant gardens could look so beautiful. At this point, I would share the benefits of having such a garden and how he can conserve so much water every year. I have now hopefully made him consider doing something like this as well. In the future, what if HIS neighbors or relatives who visit his house ask him the same question and are also struck by how appealing such a garden can look? And on and on it goes...
When it comes to making a real difference, YOU are NOT alone. You are part of a group of like-minded individuals that want to do good things. You can also cause a chain reaction and affect other people's behavior and increase the size of your group even further. Viewed in this context, your actions DO matter because if everyone thought in the same defeatist way, then nothing beneficial would ever happen.