My 2cents...
Dont go for site if you are interested in design. Once you are on site, you will get stuck there, and even if you try to come back to design you will be considered as fresh in the field as others pointed out.
Stick to the design, yes you need to have actual on site understanding of things, which you can learn at any stage (may be after 5 yrs of design experience)? You can learn it. But after 5 years of experience at site, you cannt come back to design.
And dont worry about that first 5 years period in design office. You can have visits to the site to coorelate your design with actual site execution.