Small-scale gold mining is said to have started at Toi around 1370 during the period of the Ashikaga Bakufu. The gold mine was operated on a large scale from the time of Tokugawa Ieyasu in the late 16th century. Several mines were open in 1577, but Tokugawa Ieyasu endeavored to their development from 1601. He put the exploitation of the mine under the responsibility of Gold Mine Minister (金山奉行) Ōkubo.
Toi was one of several goldmines of the Izu Peninsula, such as Yugashima or Nawaji, totaling about 60 gold mines in Izu alone. The gold and silver produced by these mines permitted the production of Tokugawa coinage, and allowed for the prosperity of the Tokugawa. The city of Toi itself became highly prosperous, with numerous trades flooding in to service the workers and the administration at the gold mine, so that Toi became known as "Toi Sengen" (土肥千軒, "Toi of the 1,000 shops").
The mine became less productive as it became flooded. Workers were killed because of the exhausting conditions due to seeping hot springs, and poor oxygen content of air, leading to the installation of water pumps and ventilators at numerous intervals.
In 1917, gold was again discovered at the mine, and exploitation continued under the company Toi Kinzan KK. In 1931, the mine entered Sumitomo Group, and passed under Toi Kōgyō KK in 1942. The mine was ultimately closed in 1965 and then reopened for tourism.
Toi was the second most productive gold mine in Japan, after the gold mine of Sado in Niigata Prefecture. During its period of exploitation, it produced in total 40 tons of gold and 400 tons of silver, whereas Sado produced as much as 80 tons of gold. One ton of rock would produce in average 5 to 10 grams of gold, although 30 grams ore was common, and some rock has yielded as much as 600 g of gold per ton.
The galleries of the mine total about 100 kilometers in length, over a surface of 37 hectares, and go as deep as 180 meters below sea level. The area visible for tourism is about 350m long, and goes about 150 meters deep into mountain rock.
The mine is now partially open for visits, and has become a tourist attraction. A "Shrine of the mine gods" (山神社) is visible inside the galleries.
The Toi Gold Museum (土肥黄金館) has also been built nearby, which describes the history of the mine and gold mining in Japan. The museum received some fame for housing the world's largest gold bar, weighing 250 kg, and representing a value of about $14.7 million in January 2012. The bar obtained an official Guinness record certificate for "The largest manufactured pure gold bar":
— Guinness World Records Certificate.