Two shells fell near the South Korean ship, which was sailing near the front-line island of Yeonpyeong, the South's defense ministry said.
"North Korea fired shells which fell near our ship, but it did not cause any damage to our ship," a ministry spokesman told AFP.
The North's move began at 6:00 pm (0900 GMT), prompting a response from the South Korean vessel which fired several rounds into waters near a North Korean vessel, he said.
The exchange of fire sparked a tense confrontation between warships from the two sides, but there was no additional provocation from North Korea, the spokesman said.
"The situation is now stable but we are closely watching the movement of North Korean troops," another ministry official said.
The defense ministry sent a message of protest to North Korea through a military hotline, he said.
The North's military threatened Wednesday to attack South Korean warships "without any warning" if there was even a "trifle" violation of the maritime border.
The threat came a day after a South Korean naval ship fired warning shots to stop an incursion by three North Korean patrol boats across the sea.
The South's navy urged the North to stop "absurd threats" and warned: "We will mercilessly punish any provocative actions by North Korea".
The North does not recognize the Yellow Sea border, the scene of brief but bloody naval clashes in 1999, 2002 and 2009.
In addition, in November 2010, North Korea shelled Yeonpyeong island, killing four South Koreans and briefly triggering concerns of a full-scale conflict.
In March the North fired hundreds of shells in a live exercise near the sea boundary. About 100 shells dropped into South Korean territorial waters, and the South responded with volleys of shells into North Korean waters.
Cross-border tension has been high for months, amid signs that the North may be preparing to conduct a fourth nuclear test.
NJF/NJF