According to the Henley & Partners Visa Restrictions List (pdf), Hong Kong passport holders are able to travel to more countries in the world without worrying about a visa than any other citizen in China or Taiwan.
Vanilla Chinese passport holders are the worse off, with only 44 countries allowing them to cross the border without sorting out a visa beforehand. The full breakdown for greater China:
People’s Republic of China: 44
Republic of China (Taiwan): 130
Macao Special Administrative Region: 118
Hong Kong Special Administrative Region: 152
Hong Kongers ability to travel the world (mostly) hassle free is likely a welcome hangover from their colonial past, Brits (along with Swedes and Finns) are able to travel to the most countries without a visa, with 173 countries willing to accept them. Strangely enough, that includes most of the countries Britain invaded over the years.