Factory Butte is the most recognizable feature of a large area of stark, barren land either side of the Fremont River known as the Upper Blue Hills, bordered by Capitol Reef to the west, the Henry Mountains to the south, San Rafael Swell to the north and the San Rafael Desert to the east. The topography is characterized by mud flats bearing sparse grass and occasional bushes, rising a little to extensive, undulating grey badlands and sharp ridges completely devoid of any vegetation, surrounding a few flat-topped hills of orange-brown sandstone, of which Factory Butte is a prominent example.