The next three chapters, entitled "Men and Brothers," "Men and Masters," and "Fading Away," focus upon Stephen and his relationship with his fellow workers, his encounter with his employer, and his loss of employment.
In Chapter 4, Dickens pictures the workers seeking to lessen the burdens of their lives. The labor-union agitator, Slackbridge, is the supposed "saviour" for the workers as they make their voices of protest heard. Dickens shows that the labor leaders may be as corrupt as the employers; he depicts the laboring class grasping at straws and led by a Judas or a false prophet. Of the laboring group, Stephen is the only one who cannot agree with Slackbridge's ideas; consequently, Slackbridge uses him as an example and turns the other workers against Stephen. When Stephen announces his decision not to join the union, the workers are convinced that "private feeling must yield to the common cause." Ostracized by his fellow workers, Stephen walks alone, afraid even to see his beloved Rachael. At the conclusion of the chapter, Bitzer comes to him and tells him that Bounderby wishes to see him.
In "Men and Masters," Stephen defends the workers against Bounderby, who calls them the "pests of the earth." Stephen says that he has not refused to join the union because of his loyalty to Bounderby but because he has made a promise. Although his own fellow workers distrust him, he is faithful to them and gives his reasons for needed reform, thus infuriating Bounderby, who dismisses him from his job in the factory. Dickens' philosophy is expressed in the conversation as Stephen tells Bounderby that men are not machines, that they do have souls. After Bounderby, who cannot bear to hear any truth except his own, fires him, Stephen leaves the large "brick castle" saying, "Heaven help us in this world." The discussion between Bounderby and Stephen has made a deep impression upon Louisa.
In the sixth chapter, "Fading Away," many threads of the plot appear. Upon leaving Bounderby's house, Stephen meets Rachael and the old woman whom he had met some time before standing outside Bounderby's house. The old woman questions Stephen carefully about Bounderby's wife. When she hears that Louisa is young and handsome, she seems delighted. Again, Stephen wonders little about the woman's curiosity concerning Bounderby. He tells Rachael that he has been fired and that he plans to leave Coketown to seek employment elsewhere. He tries to make her understand that it would be better for her if she were not seen with him anymore. Later at his room, where he and Rachael are talking with the old woman, who calls herself Mrs. Pegler, Louisa and her brother Tom come to see Stephen. For the first time, Louisa has come to the home of one of the workers. She knows well the facts of supply and demand, the percentage of pauperism and the percentage of crime, and the results of the changes in wheat prices, but she knows nothing of the workers who make up these statistics. Indeed, to her they have been just so many units producing a given amount of goods in a given amount of time and space. For the first time, she realizes that these people are not mere statistics; they have pride; they struggle to exist. She learns, too, that if a worker is fired from his job, he will not be able to find another one in the same town.
As she talks with Stephen and Rachael, she feels compassion for them and offers Stephen money to help him find employment away from Coketown. When Stephen accepts two pounds from her, Louisa is impressed with his self-command.
Tom remains quiet while Louisa converses with Rachael and Stephen. When he sees his sister ready to depart, he asks Stephen to step out on the stairs with him while Louisa remains inside the room talking with Rachael. Tom persuades Stephen that he may be able to do something for the discharged worker during the few days remaining before his departure from Coketown in search of work. Tom hints strongly about a job as a light porter at the bank. Stephen wonders about, but does not question, the strange request made of him to wait outside the bank for a while each evening. Stephen agrees to grant the request. During his three days of fruitless waiting, Stephen is probably observed by Mrs. Sparsit and Bitzer. At the end of that period, having completed the work on his loom, Stephen takes leave of Rachael and departs from Coketown.
Dickens weaves into this chapter some third-person narration concerning the fate of the workers. He says, "Utilitarian economists, skeletons of schoolmasters, Commissioners of Fact, genteel and used-up infidels, gabblers of many little dog's-eared creeds, the poor you will have always with you." He urges these people to give the poor some consideration, lest they — when nothing is left except a bare existence — rise up and destroy their oppressors.