Irvington is a town on the Rappahannock River in Lancaster County, Virginia, United States. The population was 432 at the 2010 census. Located on a peninsula known as the Northern Neck, it ha…Irvington is a town on the Rappahannock River in Lancaster County, Virginia, United States. The population was 432 at the 2010 census. Located on a peninsula known as the Northern Neck, it has been designated a historic district.