In the mid-eighteenth century the surrounding area was known for its various necks of land along the Great South Bay. After a 1758 land lottery, some of these necks evolved into the villages of Blue Point and Patchogue, while the Swan River Neck and Pine Neck became part of East Patchogue. Farming, maritime trade and lumbering were major occupations which engaged early settlers such as the Avery, Smith, Robinson and Roe families. Cordwood being in high demand, a sawmill was placed at Swan Creek, convenient for loading vessels bound for New York City.
In 1813, Brookhaven Town was divided into 23 school districts. Another was formed in 1857 for East Patchogue, with Norton Robinson as sole Trustee. At that time Stephen S. Roe and his wife Huldah owned most of the Swan River area. Roe Avenue was then known as Pine Neck Road. Robinson purchased from them, for $25, a parcel of land which the deed stipulated was "for the purpose of building and maintaining a School House thereon for the benefit of said district." The next year the school was ready, snugly constructed, with a bell tower and two doorways - separate entrances for girls and boys. At first parents paid a fee per child, but Swan River was soon incorporated into the general public school system. Its district remained intact until 1936, when it was absorbed into the Union Free School District #24 of Patchogue.
In 1962, the Patchogue School District voted to transfer the Swan River School to the Town of Brookhaven for use as a museum. For the next ten years it held displays on local history in times of war and peace. Subsequently it has resumed its principal function: old-fashioned schoolhouse.