Unless they were in line, they tended to keep a fair way apart (partly due to risk of explosion), and so would pass without each other seeing. A 60 gun frigate would have only been about 100ft long and so could have easily hidden in plain sight.
Also, there is always the possibility that he downed his flag and pretended to be part of the enemy fleet: would have been very easy had Centurion been a prize from another country (often fleets of any country were a hotch potch of other countries' vessels that had been taken as prize...very few had the money to build completely new fleets from scratch). Apart from officers, the crews would have been dressed remarkably similarly and so would not have stood out.
But, if it really was thick fog he was lucky he didn't hit anything.....
EDIT: Looks like it was just good luck:
http://www.archive.org/stream/lifeofadmira...00anso_djvu.txt
He turned up at Spithead (on the Solent) and was told he'd come through a fleet on the way there!
EDIT 2: This is a very good 18th century account (published in 1901) of the voyage which has a large section about how the squadron communicated in fog, etc.
'The next day we had very squally weather, attended with rain, lightning,
and thunder; but it soon became fair again, with light breezes, and
continued thus till Wednesday evening, when it blew fresh again; and
increasing all night, by eight the next morning it became a most violent
storm, and we had with it so thick a fog that it was impossible to see at
the distance of two ships' lengths, so that the whole squadron
disappeared.* On this a signal was made by firing guns, to bring to with
the larboard tacks, the wind being then due east. We ourselves lay to
under a reefed mizzen till noon, when the fog dispersed; and we soon
discovered all the ships of the squadron, except the Pearl, which did not
join us till near a month afterwards. The Trial sloop was a great way to
leeward, having lost her mainmast in this squall, and having been
obliged, for fear of bilging, to cut away the wreck. We bore down with
the squadron to her relief, and the Gloucester was ordered to take her in
tow, for the weather did not entirely abate until the day after, and even
then a great swell continued from the eastward in consequence of the
preceding storm.'
http://infomotions.com/etexts/gutenberg/di...16611/16611.htm