All seven rescued from the water were taken to area hospitals, authorities said.
The victims suffered various injuries. Further details were not immediately available.
The Boston Fire Department assisted with the rescue and is assisting in the recovery operation, along with the Boston Police Department, State Police, Massport and Coast Guard.
This is the second time in only a few weeks that a tragic incident occurred off the coast of Castle Island. A swimmer was reported missing on July 1. His body was recovered later that day.
The 19-year-old victim was seen in the water with his brother, but he did not return to the surface, Boston.com reported.