Can anyone tell me how to know what number to put in RA and DEC?
The book in the tent lists the detail, but to short-hand it here; choose 0-24 for Right Ascension (RA), choose 0-90 for Declination (DEC). It lets you know settings beyond those bounds are futile anyway. Just try random combinations, basically, but as others have said, just go through it sequentially, using a telescope from between in-game time of 5pm-4am (optionally buy a clock from the tinkers, also a scribe pen and blank scrolls). I suspect as not to spoil the mechanics of the system, the Dev's have been "opaque" about revealing the answers of how it works, so it really is just... try random combinations, apparently best done so across different facets.
I've scored about 10 undiscovered constellations in the space of one IRL hour, although that was when only about 250 had been discovered on my shard. About one fifth of my discoveries had already been discovered, thus were useless.
It's a shame when you hand in a Star Chart and it's accepted as undiscovered, you don't get a Star Chart back that is... permanently inscribed. For some reason you can still manipulate the name, which is kinda weak. Shame.
Finally, I cannot believe the Dev's actually thought it was a good idea for people to name their own Constellations. They have put their censor filter in place at least though, I tried to call a Constellation "Opus Clavicembalisticum", but because the second word contains ***, it was not allowed. Much like the time, many years ago, I tried to engrave a chest with the word "Documents" and it didn't warn me it was an issue, and it wrote instead for me "Do%*"ents".
*Sighs* There is nothing wrong with *** in a word, good grief.