I have noticed that the lzma compression is quite slow when compressing even a fairly small installer. It nice and fast when uncompressing just the compression seems a little sluggish. Note sure if its a problem or just th design of the lzma algorithm.
Why is the speed while building the installer a problem? You only build it a few times before you ship it. As long as it's not slow to decompress, I don't see any trouble, really. Correct me if I've missed something, though.