ebpf
  • a

    alessandrogario

    04/26/2022, 11:59 AM
  • zwass

    zwass

    05/24/2022, 6:07 AM
    Seeing this (https://doublepulsar.com/bpfdoor-an-active-chinese-global-surveillance-tool-54b078f1a896?gi=82b009d3a9dd) makes me wonder if anyone has thought about building a
    bpf_programs
    table to list the loaded bpf programs? In any case, very clever use of BPF!
  • s

    seph

    06/15/2022, 12:51 AM
    Somewhat related… https://arstechnica.com/information-technology/2022/06/novel-techniques-in-never-before-seen-linux-backdoor-make-it-ultra-stealthy/ is interesting. Malware that uses BPF to filter out it’s traffic from packet capture.
    “When an administrator starts any packet capture tool on the infected machine, BPF bytecode is injected into the kernel that defines which packets should be captured,” the researchers wrote. “In this process, Symbiote adds its bytecode first so it can filter out network traffic that it doesn’t want the packet-capturing software to see.”
  • s

    seph

    06/15/2022, 12:51 AM
    I think you’re right — BPF is an interesting vector
  • Stefano Bonicatti

    Stefano Bonicatti

    06/15/2022, 9:15 AM
    Yeah there were also some LPEs due to bugs in the BPF verifier via unprivileged BPF. One can also check
    system_controls
    to see if unprivileged BPF is enabled.
  • zwass

    zwass

    07/25/2022, 6:29 PM
    What's the status of container support for the bpf evented tables?
  • zwass

    zwass

    07/26/2022, 6:15 PM
    Are the pids returned by
    bpf_(process|socket)_events
    the global pid or in-container pid?
  • zwass

    zwass

    08/09/2022, 5:03 PM
    Bringing some things out of DM from @alessandrogario
    alessandrogario  [9:52 AM]
    The procfs-based approach can be a really useful addition to the processes table; it can't be used for events if the containers or the processes are shortlived. in that case, we have to update ebpfpub. To do that, we just have to
    detect if btf can be constructed
    pass a constructed btf object to FunctionTracer (osquery/events/linux/bpf/bpfeventpublisher.cpp), ideally only for fork/vfork/clone so that we don't add 64 bytes of overhead to all the probes
    update the sytem state tracker (osquery/events/linux/bpf/systemstatetracker.h) to propagate the cgroup names
    
    alessandrogario  [9:52 AM]
    I think having both approaches would be really cool
    
    alessandrogario  [9:55 AM]
    I've merged an example
    Initializing btf: <https://github.com/trailofbits/ebpfpub/blob/main/examples/execsnoop/src/main.cpp#L167>
    Reading the cgroup names: <https://github.com/trailofbits/ebpfpub/blob/main/examples/execsnoop/src/main.cpp#L79>
  • a

    alessandrogario

    08/09/2022, 5:04 PM
    I think this should cover the original feature request i got on the library; if there's anything else I can help with let me know (here or in private is perfectly fine!)
  • a

    Artemis Tosini

    08/09/2022, 5:10 PM
    I'll get an AWS server to test it since I've been having weird issues with my VM on top of macOS. I definitely do see the utility in both, getting things from procfs is a pain, especially since the readfile syscall isn't upstream
  • a

    alessandrogario

    08/09/2022, 5:18 PM
    I think the procfs approach to improve the processes table is even better, it's a great idea and definitely useful on all kind of deployments
  • a

    Artemis Tosini

    08/11/2022, 4:20 PM
    Not using eBPF but I have a draft PR for adding cgroups to the processes table: https://github.com/osquery/osquery/pull/7728
  • a

    Artemis Tosini

    08/11/2022, 4:22 PM
    I'm not sure how to deal with cgroups v1 since that would blow up the schema (There are 13 different controllers and each can get their own hierarchy)
  • a

    alessandrogario

    08/11/2022, 5:06 PM
    This is going to be really useful!
  • a

    Artemis Tosini

    08/11/2022, 7:48 PM
    I fixed some test issues, I'd appreciate any testing you can do
  • a

    alessandrogario

    08/17/2022, 10:51 AM
    I experimented a bit with a sql function to extract the container name from it, and it's so useful. Draft code below:
  • a

    alessandrogario

    08/17/2022, 10:51 AM
    /**
     * Copyright (c) 2014-present, The osquery authors
     *
     * This source code is licensed as defined by the LICENSE file found in the
     * root directory of this source tree.
     *
     * SPDX-License-Identifier: (Apache-2.0 OR GPL-2.0-only)
     */
    
    #include <string_view>
    #include <vector>
    
    #include <osquery/logger/logger.h>
    
    #include <sqlite3.h>
    
    namespace osquery {
    
    namespace {
    
    const std::vector<std::string> kPrefixList{
        "libpod-conmon-",
        "libpod-",
        "docker-",
    };
    
    const std::string kSeparatorList{"./"};
    
    void cgroupContainerName(sqlite3_context* context,
                             int argc,
                             sqlite3_value** argv) {
      auto buffer_ptr = reinterpret_cast<const char*>(sqlite3_value_text(argv[0]));
      if (buffer_ptr == nullptr) {
        sqlite3_result_null(context);
        return;
      }
    
      auto buffer_size = std::strlen(buffer_ptr);
      std::string_view cgroup_path(buffer_ptr, buffer_size);
    
      for (const auto& prefix : kPrefixList) {
        auto start_index = cgroup_path.find(prefix);
        if (start_index == std::string::npos) {
          continue;
        }
    
        start_index += prefix.size();
    
        auto end_index = cgroup_path.find_first_of(kSeparatorList, start_index);
        auto substr_size = (end_index != std::string::npos)
                               ? end_index - start_index
                               : std::string::npos;
    
        auto container_name = cgroup_path.substr(start_index, substr_size);
    
        sqlite3_result_text(context,
                            container_name.data(),
                            container_name.size(),
                            SQLITE_TRANSIENT);
        return;
      }
    
      sqlite3_result_null(context);
    }
    
    } // namespace
    
    void registerCgroupUtilsExtensions(sqlite3* db) {
      sqlite3_create_function(db,
                              "cgroup_container_name",
                              -1,
                              SQLITE_UTF8 | SQLITE_DETERMINISTIC,
                              nullptr,
                              cgroupContainerName,
                              nullptr,
                              nullptr);
    }
    } // namespace osquery
  • a

    alessandrogario

    08/17/2022, 10:52 AM
    osqueryd -S 'SELECT name, cgroup_path, cgroup_container_name(cgroup_path) AS container FROM processes WHERE container <> "";' 
    +--------+--------------------------------------------------------------------------------------------------------+------------------------------------------------------------------+
    | name   | cgroup_path                                                                                            | container                                                        |
    +--------+--------------------------------------------------------------------------------------------------------+------------------------------------------------------------------+
    | conmon | /machine.slice/libpod-conmon-b8c8b2f3d8dcffed531ea0557e59e740264a1b2a3d6b76fd86b2c258309e3254.scope    | b8c8b2f3d8dcffed531ea0557e59e740264a1b2a3d6b76fd86b2c258309e3254 |
    | bash   | /machine.slice/libpod-b8c8b2f3d8dcffed531ea0557e59e740264a1b2a3d6b76fd86b2c258309e3254.scope/container | b8c8b2f3d8dcffed531ea0557e59e740264a1b2a3d6b76fd86b2c258309e3254 |
    +--------+--------------------------------------------------------------------------------------------------------+------------------------------------------------------------------+
  • a

    alessandrogario

    08/17/2022, 10:53 AM
    Another thing that could be useful is something to get the backend name, like Docker, podman, etc
  • s

    seph

    08/17/2022, 12:24 PM
    I suspect you can do those with
    REGEX_MATCH
    but
    cgroup_container_name
    seems less fiddly 🙂
  • a

    alessandrogario

    08/17/2022, 4:17 PM
    @Artemis Tosini would something similar to this functionality be interesting for your future enhancements roadmap?
  • a

    alessandrogario

    09/27/2022, 1:34 AM
    We have a new BPF PoC here: https://github.com/osquery/osquery/pull/7773 Test packages: https://github.com/osquery/osquery/actions/runs/3132085206 TLDR: With BTF (built-in kernel debug symbols, in kernels >= ~5.3) we can now read kernel structures while also respecting the osquery guidelines of having no external dependencies. This means we no longer need to trace a large amount of system calls, which was solved by auto-generating the BPF probes using LLVM IR. The new probes are written in C! We have some additional advantages compared to other solutions, like being able to inspect private kernel types (thanks to the debug info we are parsing). If there is interest and some spare time during the next office hours, I'll briefly walk over this PR and explain what it contains, how it was built and why. It still requires some work, and it's currently marked as an experiment and not a stable feature (more on this in the PR description). As previously stated it is going to require newer kernels, so we may still have to keep the current implementation for a while
  • zwass

    zwass

    09/28/2022, 10:10 PM
    @alessandrogario do you have some time Monday that we could meet with @Artemis Tosini and talk about joining in on the eBPF work? Our schedules are fairly open. Perhaps 5PM your time?
  • a

    alessandrogario

    09/29/2022, 3:04 PM
    Totally unrelated, live streaming for the second day of the (free) ebpf-summit event https://ebpf.io/summit-2022/day-2
  • a

    ag4ve

    09/29/2022, 8:35 PM
    so, that talk where the guy showed a ./query-bpf (which I searched for and couldn’t find) that created and deployed a program and filter and showed the events - that’s possible in osquery?
  • a

    alessandrogario

    10/01/2022, 3:56 PM
    Pushed a new update to PR7773, fixing support for Ubuntu 20.04 LTS (kernel 5.15.0-48-generic) The new packages can be downloaded from here: https://github.com/osquery/osquery/actions/runs/3164969151 (files are only visible for logged in users)
  • a

    alessandrogario

    10/04/2022, 8:33 AM
    @Zander Mackie && @Matt Uebel Would it be possible for you to try out these packages? This is from the following PR: https://github.com/osquery/osquery/pull/7773 (packages are from this runner: https://github.com/osquery/osquery/actions/runs/3177333118)
  • a

    Artemis Tosini

    10/14/2022, 6:58 PM
    I think there's a TOCTOU in the execveat example of ebpfpub, though there shouldn't be an issue if you use the tracepoint like in linuxevents
  • l

    Lili

    11/25/2022, 9:15 AM
    Hello Everyone! I am testing usage ebpf with osquery, but sometimes get this errors: 1.
    orbit[24125]: I1125 11:47:48.859995 24164 rocksdb.cpp:67] RocksDB: [ERROR] [table/block_based/block_based_table_reader.cc:1090] Encountered error while reading data from properties block IO error: While pread offset 4655206 len 40: /opt/orbit/osquery.db/004706.sst: Is a directory
    
    orbit[24125]: I1125 11:47:48.955194 24164 rocksdb.cpp:67] RocksDB: [ERROR] [db/db_impl/db_impl_compaction_flush.cc:2624] Waiting after background flush error: IO error: While pread offset 4655206 len 40: /opt/orbit/osquery.db/004706.sst: Is a directoryAccumulated background error counts: 1
    
    orbit[24125]: E1125 11:49:16.361975 25035 shutdown.cpp:79] Error adding new results to database for query pack/Global/bpf_socket_events (test)-1: IOError: Is a directory
    2.
    orbit[13777]: I1125 11:43:09.381443 13824 rocksdb.cpp:67] RocksDB: [WARN] [db/error_handler.cc:387] Background IO error IO error: While appending to file: /opt/orbit/osquery.db/005943.sst: Bad file descriptor
    
    orbit[13777]: I1125 11:43:09.381582 13824 rocksdb.cpp:67] RocksDB: [ERROR] [db/db_impl/db_impl_compaction_flush.cc:2624] Waiting after background flush error: IO error: While appending to file: /opt/orbit/osquery.db/005943.sst: Bad file descriptorAccumulated background error counts: 1
    
    orbit[13777]: E1125 11:44:52.375362 19350 shutdown.cpp:79] Error adding new results to database for query pack/Global/ebpf proccess events (test): IOError: Bad file descriptor
    After that agent is restarted.Why this errors occurs? Maybe I can tune some flags for exclude this error? Query: 1.
    SELECT uid, gid, local_address, local_port, path, pid, remote_address, remote_port, ntime, duration, family, exit_code FROM bpf_socket_events WHERE path NOT IN ('/usr/bin/hostname') AND remote_address NOT IN ('127.0.0.1', '169.254.169.254', '', '0000:0000:0000:0000:0000:0000:0000:0001', '::1', '0000:0000:0000:0000:0000:ffff:7f00:0001', 'unknown', '0.0.0.0', '0000:0000:0000:0000:0000:0000:0000:0000');
    2.
    SELECT cmdline, ntime, cwd, gid, uid, parent, path, pid FROM bpf_process_events;
    Flags used:
    table_delay: 200
      augeas_lenses: /usr/share/augeas/lenses/dist
      logger_plugin: tls
      disable_events: false
      watchdog_delay: 120
      buffered_log_max: 3000000
      disable_watchdog: false
      enable_bpf_events: true
      logger_min_stderr: 1
      logger_tls_period: 1
      logger_tls_max_lines: 8192
      watchdog_memory_limit: 2048
      bpf_buffer_storage_size: 4096
    Fleet version: 4.23.0 Agent version: 1.3.0 Osquery version: 5.5.1 VM has 8 GB RAM, 4 CPU.