The map has a large 'X' marked in the middle of it. Near the marking, you can see several dotted lines forming a square. Contour lines drawn along the map indicate that the destination is located near sea level, and swamp grass is drawn around the 'X'. Straight lines have been drawn from the center, pointing west. From other markings on the map, you guess that the destination is directly south of here.
The treasure is directly south when I'm on the gondola and directly west of Fang Cove, so I believe it's around shard. I spend most of my time north of Crossing, though, so I have no idea where I would find swamp grass around Shard.