Settled in 1634, Port Tobacco was once Maryland's second largest seaport and was listed on early World Maps. This settlement was originally the site of the Indian Village of Potopaco. Port Tobacco was the first county seat, but after the river began silting up and after a shift of occupations from tobacco farming to other trades and industry people moved to the town of La Plata where the new railroad was being built. The county seat was eventually moved to La Plata.
The first Charles County Courthouse was completed in 1729. However, Port Tobacco was not the site of the first court house. The exact location is unknown but it is thought to be near the present-day fairgrounds. A second courthouse was built in 1819. In 1892, the Courthouse burned. The present building is a reconstruction of the third structure. During tours, costumed docents tell the story of Port Tobacco, from 1620 to present day.