The Malvern Council House, between Church Street and Avenue Road, was the site of a large private house from 1834. Called "The Priory". The house was built by Robert Thompson. Dr Gully, one of the famous water cure doctors, bought the house in 1847 for a sum equivalent to about £1m today.

In 1873 Gully sold it to Alfred Miles Speer, a South American Merchant, who effectively demolished the house and constructed a new building to be a "first class, gentleman's residence" which he called The Priory Mansion. This Mansion is the building which still stands today. Speer subsequently bought the land which is now Priory Park and Swan Pool, combining his magnificent mansion with 7 acres of beautiful gardens.

After Speer and his wife's death, his son leased the building in 1911 with a restriction that the property must be used as a school to educate the sons of noblemen and gentlemen. The school was duly formed, a gymnasium built (Priory Lodge Hall, the one story building between the Council House and Avenue Road) and also an open-air swimming pool (the site of the current Splash swimming pool).

In 1925, the whole property was sold to Malvern Urban District Council and has remained in public hands ever since. The building serves as the Malvern Hills District Council offices, the Priory Lodge (formally gymnasium) accommodates public meetings, the beautiful gardens remain as Priory Park and the open air swimming pool is the site of Malvern Splash, a modern leisure centre.