Experimental theatre is a vague, catch-all term for a number of theatrical styles and movements that began in the 1900s. At that time, the acceptable conventions for the writing and production of plays was pretty narrow and leaned heavily towards naturalism, which strives to mirror reality in the style of acting, dialogue, costuming, and sets.
The term "experimental theatre" has changed over time, as mainstream theatre has accepted many forms that were once considered radical. It is pretty much used interchangeably with the term avant-garde theatre.
Still confused? Consider this: A production of Tennessee Williams's Cat On a Hot Tin Roof would be naturalist, while Blue Man Group puts on something more experimental. As Blue Man Group grows in popularity and their shows influence more theatrical troupes and writers, however, something even more experimental eventually will come along.