John Rutherford died on 21 June 1781 aged 40 and was removed from his original burial place at Olne in Germany to the north cloister of Westminster Abbey for burial on 11th October 1782. His grave is marked just with the initials J.R. He was the only son of John Rutherford, who originally came from Kelso in Scotland and then moved to London, and his wife Jane (Kerr). He was a captain in the Middlesex militia and part owner of Covent Garden theatre. In 1781 he published a translation of Cicero's Orations. While endeavouring to establish his claim to the ancient barony of Rutherford he was killed by a fall from his horse. His wife was Ann, daughter of Shuckburgh Broughton and sister of Sir Edward Broughton.
John's sister Margaret lived in a house near the Abbey and was buried on 6th January 1816, aged 68. She was the widow of Captain John Shortland of the Royal Navy and their chirldren were Peggy (who married William Peace), Captain Thomas Shortland (later Commissioner of the Navy at Port Royal, Jamaica), Captain John Shortland who died of wounds fighting the French, and Jane who married Mr Wilkinson.