There are two cardinal mistakes in your reasoning here: (1) As I mentioned in my other reply, Sindarin words do not derive from Quenya words. Instead you should, in a case like this, find the primitive Elvish word from which the Quenya word was derived, and apply the Sindarin phonological rules to it. (2) Phonological rules apply chronologically in sequence; you can see the order of the rules in the lists of rules on Eldamo, first applying Ancient Telerin (AT) rules, then Old Sindarin (OS) rules, and then Sindarin (S) rules, in the order that they are numbered.
If we ignore point (1) and some other considerations for a moment and pretend that *luksur or some such was a viable form that could then be developed into a Sindarin word, the development would be as follows:
*luksur > [OS32] luxxuro > [S23] luxxur > [S67] luxur, which in normal Sindarin orthography is luchur.
(Note: in the phonological derivations I am using IPA notation, hence [x] is not the same as the English letter 'x', which stands of [ks], but rather as the 'ch' in Loch.)
(Also note that the short [u] does not become [o] because of the other [u] present, cf. S. curu etc.)
However, if we do consider what the correct form from which we should be deriving would be, *luksur would not be it.
The word luxor, as you noted, is ᴱQ, i.e. Early Qenya, in the sense that it comes from the oldest drafts of the Quenya language (it has nothing to do with the in-universe evolution of the language, but everything to do with Tolkien's out-of-universe conceptual evolution). As such it should be treated with great care, since it is highly unlikely to be suitable in the late conception of Quenya. As Paul Strack notes in his write-up about this word, the root from which it derives ᴱ√LUKU in the later conception would be √LUK, except that root, in turn, has a very different meaning in the late conception. As such, this word can only be used in Quenya if a new etymology could be found for it, that would convey a suitable meaning whilst retaining its form.
This process - re-etymologising and updating words from early conceptual periods - is often highly controversial. But if we put the controversy aside and accept Paul's solution, i.e. updating ᴱQ. luxor to ᴺQ. loxor by assuming it derives from √LOG > log·so·ro, then we have a primitive form from which we can try to derive a Sindarin word. This, I believe, would develop thus:
*logsoro > [S9] loɣsoro > [S23] loɣsor > [S30] losor.
That being said, I am not convinced that such a primitive form is truly viable to begin with.
Final thing to note:
The -gth- cluster should be permissible in Sindarin, I think. There is an attested (archaic) example, †Ægthelion. What you should pay attention to however is that the cluster, which is spelled <gth>, is pronounced as [gθ] - which is two consonants, not three; it cannot be reduced to [gh] by removing the middle consonant, because there is no middle consonant - orthographic <th> is phonetic [θ], a single consonant, not [th].
