What does my brother mean when he says he's too ensconced in his studies to look for a girlfriend?

Sounds like you have a devoted student among your family members! Your brother's choice of the word ensconced suggests that he's settled in with a study routine that requires lots of his attention. To be ensconced means to be planted firmly in a position that's safe, hidden, or just plain comfortable.

From Mark Twain's The Adventures of Tom Sawyer:

They found the sharp new heap they were seeking, and ensconced themselves within the protection of three great elms that grew in a bunch within a few feet of the grave.

And in Charlotte Brontë's Jane Eyre:

And then they had called her to a sofa, where she now sat, ensconced between them, chattering alternately in French and broken English; absorbing not only the young ladies' attention, but that of Mrs. Eshton and Lady Lynn, and getting spoilt to her heart's content.

From Incidents in the Life of a Slave Girl, by Harriett A. Jacobs:

Ensconced in a pleasant room, with my dear little charge, I laid my head on my pillow, for the first time, with the delightful consciousness of pure, unadulterated freedom.

Shakespeare's character Falstaff has this to say in The Merry Wives of Windsor:

I, I, I myself sometimes,
leaving the fear of God on the left hand, and hiding mine honour in
my necessity, am fain to shuffle, to hedge, and to lurch; and yet
you, rogue, will ensconce your rags, your cat-a-mountain looks,
your red-lattice phrases, and your bold-beating oaths, under the
shelter of your honour!