It's hard to ask questions that don't suck when you're running a study, so the best advice is "don't do it." But you are going to anyway, so make sure your questions are grounded in what the user has done, not what you want them to do.
Even with a good content management system it can be hard to re-arrange stuff after you've gone live. Take the time to test out your proposed navigation with a reverse card sort to quickly iterate to a working model.
Users are happiest when your site's structure - its information architecture - matches the way they think about the problem space. Get insight into their thoughts using a card sorting task. You'll be surprised how different their perspective is from yours.
There are times when "ecological validity" is important, and there are other times when you just want to know whether users can make their way through your interface. Save money by choosing the right location to run each type of study.
Labels: planning user studies