Why 19th-century writers were obsessed with madness

Read a few more stories in Library of America "American Fantastic Tales" (or fantastic American tales?), these by some of the 19th-century giants: Irving, Hawthorne, Poe, Melville. The Melville choice (The Maids of Artemis, or something like that) a bit odd, as it isn't truly a horror/fantasy story. The narrator/protagonist takes a long ride through isolated and dismal woods to visit a paper mill and factory. Freezing cold when he arrives (by horse-drawn carriage). He shelters the horse, tours the factory, sees the miserable workers. From Melville's POV, I would say he saw this is social realism. Perhaps he had visit Crane paper, near where he lived in Lenox, and was moved to write about the workers' conditions. The other three stories, including the famous Young Goodman Brown, a truly in the tradition. Amazing, the early 19th-century fascination with various conditions of madness - that's the main trope in these early horror stories. Madness is generally a freefloating and inexplicable malady, in this preFreudian area. Nobody knew the causes, much less the treatment - so madness seemed much scarier, like a visitation or possession. Poe's story in the collection, "Berenice," while still a genre piece, is far superior to others of its type - the main character a truly compulsive obsessive, unable to control himself, and he becomes fixated on his cousin/fiancee's teeth. Extremely wierd and disturbing. Hawthorne's Young Goodman Brown is rightly famous - strange and unsettling in a different way, as the young Salem man, on a night journey (a dream?) sees that the upstanding citizens of his time are secretly communicants with the devil - a vision that destroys his life. A lot of dimensions to this story: psychological, allegorical, sociological, historical, theological - could easily teach a whole seminar this one story I think.