I would avoid using conjunctions within compounds; Dor-Cúarthol seems to be aberrant in this respect, and I wouldn’t use it as an example.
That being said, the form Dor-Cýlestel should work, simply combining cŷl+estel. Alternatively you can separate the elements: Dor-Cŷl-ah-Estel or Dor-Estel-a-Chŷl.
As for ŷ vs ý - it is not correct to assume that ý is a shortened form of ŷ. Sindarin does not phonemically distinguish long vowels from overlong, only short from long/overlong, and the overlong vowels (with rare exceptions) only appear in monosyllabic words. In other words, a monosyllabic word with ŷ will use ý in a compound.
In native orthography (Tengwar or Cirth) the two are not even distinguished in writing.
As for whether it should be long or short - in this case the length should be retained because it was long primitively and is not followed by a consonant cluster, nor is it the final syllable.
In cases where the primitive form had a short vowel (that was subsequently lengthened in a monosyllabic word), the vowel should become short in a compound. When a long vowel is followed by a consonant cluster, it should in most cases shorten. The final syllable of a polysyllabic word, for the purpose of constructing new words, is always shortened regardless of origin.
