Ran across this one recently and was surprised by SharePoint’s behaviour. If you create your own custom master page and apply it to a parent Publishing site and have all your subsites set to inherit the custom master page, all will seem fine. Then if you create a new Team based subsite off of your Publishing parent site, you will notice that it does NOT inherit your custom master page and instead uses default.master. If you then go to the master page setting screen for the subsite you will see this message:
“The site master page setting currently applied to this site is invalid. Please select a new master page and apply
it.”
This is beyond frustrating if you have a site that needs many subsites with the same branding. Luckily you can manually correct the problem by creating all your subsites and THEN going back to the master page settings screen and set all the subsites to inherit the custom master page and click OK. This will set them all to your custom master.
There is a KB on this topic located here: http://support.microsoft.com/kb/936908/en-us
Essentially this has to do with the Team sites not having publishing enabled, which is frustrating because you will run into this even if you aren’t using any publishing specific items in your master page. The only way they list for automatically correcting is by using Feature Stapling.