Why the narrator of James's story Brooksmith is a complete fool

The Henry James story Brooksmith (1892), in the Crane anthology 50 Great Short Stories, is among the most sorrowful and in many ways horrendous of James's stories; those who find his novels meandering and impenetrable, especially the late novels, should consider or re-consider his stories, where the confines of the form pushed him if not toward concision at least toward building a plot and establishing a character with some economy of phrasing, while maintaining the James acuity and psychological insight. In this story, the unnamed narrator reflects on an "Arcadian" time in the past when he and several others would regularly visit a retired diplomat in his house in London for what was essentially a literary/intellectual salon - and the narrator opines that the eponymous Brooksmith, the butler, was the one who through his acuity and sense of order and whom to sit near (or away from) whom, etc., made the salon pleasant and possible. Eventually, the retired diplomat sickened and died and of course the salon ended; the narrator laments this lost period of his life, but what about Brooksmith? The diplomat left B some 80 pounds; I don't know how to translate that into contemporary $s, but it was obviously a pittance. Brooksmith goes on to work in other houses, then gets ill and is out of work for a while, then takes on a job as a hired hand in a catering crew - a steady downward slope. And none of the well-to-do men in the all-male "salon" is able to do a thing to help this poor man - yet apparently they all look upon this past period of their lives as a shining moment. But a moment w/out a heart or a care for any but themselves, without even for a moment questioning the class structure that keeps B confined to a clearly lower class and without a social structure to support anyone in ill health or unemployed. All of their opining about beauty and art and philosophy does nothing to broaden the scope of their understanding of and compassion for others. The narrator is in a sense a complete fool, without James ever having to say so; he hangs himself w/ his own rope.