According to Garnet and Peridot, Homeworld’s galaxy can be seen from Earth. Assuming the Gems have the same visual capabilities as humans and they didn’t mean the Milky Way itself (which would be a bit of a silly way to refer to it), that very likely means the Gems came from Andromeda. It’s the only galaxy you can see fairly easily from the northern hemisphere. Triangulum is just barely visible with the naked eye too, but only under very good conditions. Between the clouds, the fact that it’s not entirely dark, and the both of them wearing goggles, I really don’t think they could see it. Finally there’s the two Magellanic Clouds. These are only visible from the southern hemisphere on real!Earth, but back in It Could’ve Been Great, we saw the Earth’s axis may be different, which would also show the northern hemisphere different parts of the sky. So my guess is the Gems came from Andromeda, but there are different possibilities.