Screening using PSA blood tests is controversial because cancer is missed in some men while others get false positive results.
An Icelandic-led team said they had discovered several DNA variations that affect background blood levels of PSA.
In the future, testing for these gene variations may be able to create much more accurate individual PSA thresholds.
"By using these markers ... we believe we have made the PSA test both more specific and more sensitive," Kari Stefansson said. However, Kate Holmes, research manager at Britain's Prostate Cancer Charity, said more research on a large number of men was needed to investigate the accuracy of the new approach.