I would suggest taking a look at the liquidation market as a possible source. Someone like via trading, located in LA, do carry a pretty good range of men's clothing, and they ship worldwide.
Check them out, but just beware of the type of stock you are buying when dealing with any liquidations. You can find anything from salvage to new overstocks, and everything in between.
Via have excellent customer service, and are quite happy to answer any questions you have about their stock, and that's really important in that market.
To try and save a few dollars on shipping, because it won't be cheap, you could also try to find another Australian member who is interested in splitting a load with you.