Quite difficult for them indeed. Later on, an intergalactic alliance will be formed to stand up against them, with several hi-tech alien races taking part as well. But even so, the alliance will still be at a great disadvantage.
Lemuria's only downfall is it's low population. Meaning, the ratio is usually 1-20 for a lemurian soldier.