every once and a while I get to have some fun, I spent some time in the airport yesterday to see if I could convert osquery table implementations into a Python "generator" like feel. The goal being to prevent ballooning of memory when we hold 2 copies of table results in memory as we copy into SQLite's representation. While I was successful in smoothing the private memory requirements over time it was only about 10% and at the cost of significant context switching leading to 3% more CPU needed for each table except for

launchd

which results in 7% less CPU: https://github.com/theopolis/osquery/commit/557acbff69c360eeb3b249a74aca7a8810e6fa19 To get an idea of what this change does consider: https://github.com/theopolis/osquery/blob/557acbff69c360eeb3b249a74aca7a8810e6fa19/osquery/tables/system/darwin/packages.cpp#L274

void genPackageReceipt(const std::string& path, RowYield& yield) {
  auto receipt = SQL::selectAllFrom("preferences", "path", EQUALS, path);
  if (receipt.size() == 0) {
    // Fail if the file could not be plist-parsed.
    return;
  }

  Row r;
  r["path"] = path;
  for (const auto& row : receipt) {
    if (kPkgReceiptKeys.count(<http://row.at|row.at>("key")) > 0) {
      r[<http://kPkgReceiptKeys.at|kPkgReceiptKeys.at>(<http://row.at|row.at>("key"))] = <http://row.at|row.at>("value");
    }
  }

  if (!r["package_id"].empty()) {
    yield(r);
  }
}

looks like Python!