If I don't tell osquery.conf to load packs, does it load all of them or none?

It won't load any pack you don't specify in your config. So you can have no packs and just custom individual queries, I think I did this till I started using fleet.