You can do this:
DATEDIFF(dd,startDate,EndDate) - DATEDIFF(wk,startDate,endDate);
It should work correctly regardless of your datefirst setting, but check the boundary conditions (when start date or enddate is on a weekend) to see if it is off by 1 and make adjustments as necessary.