St. Theresa of Avila criticizes small talk in her autobiography. She criticizes it on spiritual grounds, but I can't remember at this writing what they were. I assume it basically revolves around the fact that every act of speaking is an assertion of self, small talk takes up time that could be spent more profitably, and talking is always a potential occasion of sin (when you think about it, it's hard to speak 100% truthfully during any extended discussion; reality simply contains too many elements to convey accurately once your talk goes beyond a very limited scope).