Both men know their town very well. Unbeknownst to the Finch children, Boo has watched them grow up. The reader can fairly assume that Boo is also familiar with the Ewells, and probably doesn't think much more of them than the rest of Maycomb. Boo and Tom have had minor skirmishes with the law, but that past doesn't tarnish the kindness they show to others in the story. The moment that Mayella makes a pass at Tom, he inherently knows that he's in serious danger. Truthfully, he probably knew that helping her without pay was not the safest thing for him to do, but the compassion of one human being for another won out over societal expectations.
The children treat Boo with as much prejudice as the town shows Tom Robinson. They assign characteristics to Boo without validation; they want to see Boo, not as their neighbor, but as a carnival-freak-show-type curiosity. Ironically, watching the injustice that Tom suffers helps the children understand why Boo may choose to be a recluse: "'it's because he wants to stay inside.'"
