This one was undocumented in SharePoint 2007 / WSS 3.0 and is still undocumented in SharePoint 2010 / SPF. I posted about this back in 2007 for the previous version, but was asked the same question in the SharePoint Support Forums. A little digging and firing off a test confirmed that the command has stayed intact for the latest version as well. Using STSADM from a command windows:
STSADM -o setproperty -pn max-template-document-size -pv 524288000
Let it run and upon completion, you should see this:
Trackback from your site.