Extending Applications Safely and Efficiently Yusheng Zheng, UC Santa Cruz; Tong Yu, eunomia-bpf Community; Yiwei Yang, UC Santa Cruz; Yanpeng Hu, ShanghaiTech University; Xiaozheng Lai, South China University of Technology; Dan Williams, Virginia Tech; Andi Quinn, UC Santa Cruz https://www.usenix.org/conference/osdi25/presentation/zheng-yusheng This paper is included in the Proceedings of the 19th USENIX Symposium on Operating Systems Design and Implementation. July 7–9, 2025 • Boston, MA, USA ISBN 978-1-939133-47-2 Open access to the Proceedings of the 19th USENIX Symposium on Operating Systems Design and Implementation is sponsored by Extending Applications Safely and Efficiently Yusheng Zheng UC Santa Cruz Tong Yu eunomia-bpf Community Yiwei Yang UC Santa Cruz Yanpeng Hu ShanghaiTech University Xiaozheng Lai South China University of Technology Dan Williams Virginia Tech Andi Quinn UC Santa Cruz Abstract This paper presents the Extension Interface Model (EIM) and bpftime, which together enable safer and more efficient extension of userspace applications than the current state-of-the-art. EIM is a new model that treats each required feature of an extension as a resource, including concrete hardware resources (e.g., memory) and abstract ones (e.g., the ability to invoke a function from the extended application). An extension manager, i.e., the person who manages a deployment, uses EIM to specify only the resources an extension needs to perform its task. bpftime is a new extension framework that enforces an EIM specification. Compared to prior systems, bpftime is efficient because it uses extended Berkeley Packet Filter (eBPF)-style verification, hardware-supported isolation features (e.g., Intel MPK), and dynamic binary rewriting. Moreover, bpftime is easy to adopt into existing workflows since it is compatible with the current eBPF ecosystem. We demonstrate the usefulness of EIM and bpftime across 6 use cases that improve security, monitor and enhance performance, and explore configuration trade-offs. 1 Introduction Developers extend their software to customize it for the needs of a particular deployment. For example, extensions can improve application performance [2, 40], add custom features [48, 50], enhance security [46, 62], and enable application observability for performance monitoring [61, 77] and debugging [39, 45]. Software extensions do not require modifying the original application, which enables customization while ensuring that the deployment can easily integrate maintenance updates from upstream repositories. Many software applications support extensibility, including web browsers [21, 43], HTTP servers [17, 25], text editors [24, 64], and databases [49, 59]. To extend software, a developer defines new logic for the program as a set of extensions, and associates each extension with a specific location in the host application, called an extension entry. When a user invokes the application, the USENIX Association system loads the host application and the user’s configured extensions. Each time an application thread reaches an extension entry, the thread jumps to the associated extension. It executes the extension in the extension runtime context; once the extension completes, the thread returns to the host at the point immediately after the extension entry. This paper presents a new approach for specifying the interface between an extension and a host, called the Extension Interface Model (EIM), and a new extension runtime system, called bpftime, that together provide safer and more efficient software extensions. EIM and bpftime are motivated by the challenges current extension frameworks face in balancing key extensibility features. First, current extensibility frameworks struggle to navigate the tradeoff between extension interconnectedness and safety. On the one hand, enabling extensions that perform meaningful work requires interconnectedness, i.e., the ability to observe or modify the host application’s state and to execute functions defined by the host. On the other hand, application deployments require that their extensions are safe, i.e., that an extension failure cannot harm the health of the host application and the underlying system. Achieving safety requires restricting an extension’s behavior to limit both the system resources it consumes (e.g., the memory it uses, the files it opens) and the host interactions it performs (e.g., the host state it reads). Interconnectedness and safety are in tension because interconnectedness often necessitates allowing an extension to modify the host in potentially unsafe ways. We introduce the term extension manager to describe the person who is responsible for configuring the extensions on a given deployment and thus must navigate the tradeoff between interconnectedness and safety. To maximize system safety, an extension manager should follow the principle of least privilege, granting extensions only the minimal set of features their use cases require. For example, a deployment that supports observability extensions for debugging and monitoring may have different interconnectedness/safety needs than one that supports extensions for customizing application behavior. Unfortunately, current 19th USENIX Symposium on Operating Systems Design and Implementation 557 extension frameworks poorly support specifying and enforcing such deployment-specific interconnectedness/safety tradeoffs. Many frameworks (e.g., safe language runtimes [23, 38], NaCl [75]) cannot express an interconnectedness/safety tradeoff. Instead, they rely on applications to enforce their own safety, which is ad hoc and error-prone (see §2). Other frameworks (e.g., lwC [37], RLBox [44], Shreds [11]) do not support fine-grained limits on extensions. They either cannot restrict certain unsafe extension behaviors or cannot do so on a per-extension-entry basis. Finally, some frameworks (e.g., Orbit [31], Wedge [7]) do support fine-grained interconnectedness/safety tradeoffs, but they are not designed for extensibility and require modifying host application source code to impose different tradeoffs. Our first contribution, EIM, supports fine-grained interconnectedness/safety tradeoffs. EIM’s key idea is to represent the extension features needed for interconnectedness or restricted for safety through a single abstraction called a resource. For example, a resource can represent an extension’s ability to call a host function or read a host variable. EIM represents the ability to use resources with capabilities [55, 60, 71]. An EIM specification is produced by two parties: the original application developer and the extension manager. First, developers define capabilities that represent the resources the host application can provide to extensions, essentially enumerating the extension interconnectedness that the host application supports. Then, during deployment, the extension manager creates extension classes that specify the set of capabilities allowed at a particular extension entry, essentially choosing the interconnectedness/safety tradeoff for each extension entry. In sum, EIM specializes interfaces for fine-grained protection, such as those for access control of OS objects (e.g., SELinux [54]) or browser manifest files [22], to support the fine-grained tradeoffs necessary for software extensions. EIM specifications are runtime-agnostic, and we could enhance an existing extension framework to enforce them. However, current extension frameworks poorly navigate the tradeoff between three properties: extension safety, as specified in an interconnectedness/safety tradeoff; extension isolation, which prevents a host application from harming an extension and is necessary for security monitoring extensions; and extension efficiency, which requires that extensions execute at near-native speed. Current frameworks are inefficient because they employ heavyweight techniques for isolation and safety. For example, many frameworks (e.g., Orbit [31], lwC [37], Wedge [7]) provide new operating system-level isolation abstractions and require context-switch-like overhead to switch between a host and an extension. Other frameworks (e.g., Wasm [23], NaCl [75]) enforce safety and isolation using software fault isolation (SFI), which is much slower than native execution [29]. Our second contribution is bpftime, a new extension runtime that efficiently supports EIM and extension isolation using two design principles. First, the system uses lightweight 558 approaches to provide extension safety and isolation. It enforces the safety in an EIM specification without any runtime overhead using extended Berkeley Packet Filter (eBPF)style verification, and it enforces isolation with minimal overhead using ERIM-style intraprocess hardware-supported isolation [66, 72]. Second, bpftime introduces concealed extension entries, which improve efficiency by eliminating runtime overhead from extension entries that are not in use by a running process. Concealed extension entries use binary rewriting [10, 14, 18, 70] to inject an extension entry into a host only when a user loads an associated extension. While prior work uses techniques similar to bpftime’s verification, isolation, and rewriting techniques, bpftime is the first system combine them to satisfy EIM’s requirements. Additionally, bpftime is fully compatible with eBPF, streamlining the system’s path to adoption. With eBPF compatibility, not only can current users of eBPF extensions (e.g., uprobes) seamlessly adopt bpftime1 , but bpftime extensions can also share state and interact closely with eBPF kernel extensions, thereby supporting extensibility use cases that require extending both the kernel and applications. We maintain bpftime as an open source project 2 ; bpftime has 1,000 stars on GitHub, more than 20 contributors, and several PRs per month. Our users currently rely on bpftime and EIM for many use cases, including observability, fault injection, hot patching, and other application customizations. Inspired by these users, we present 6 use cases that highlight the benefits of bpftime and EIM. These use cases explore design tradeoffs, improve security, monitor performance, and enhance system efficiency. In particular, we use bpftime to monitor a microservice application, create new durability configurations in Redis, cache metadata operations in FUSE, implement an SSL-supporting distributed tracing tool, monitor system calls for performance analysis, and enhance webserver security. We evaluate bpftime’s performance on the aforementioned 6 use cases. We find that bpftime improves the throughput of profiling microservices by a factor of 1.5 compared to eBPF. bpftime enables a Redis durability configuration that loses orders of magnitude less data in a crash while decreasing throughput by only about 10%. bpftime enables FUSE caching that accelerates operations by orders of magnitude. The system adds only 2% overhead when extending Nginx, which is up to 5× lower than state-of-the-art alternatives such as WebAssembly, Lua, ERIM [66], and RLBox [44]. bpftime reduces the overhead of SSL traffic monitoring by a factor of 3.79 compared to native eBPF. Moreover, bpftime offers configurations that prevent monitoring overhead from affecting unmonitored processes, a feature eBPF cannot provide. We also use microbenchmarks to illustrate the key features that enable bpftime’s performance advantages and demonstrate bpftime’s compatibility with eBPF. 1 Note: eBPF uprobes do not support all bpftime features. 2 available at https://github.com/eunomia-bpf/bpftime. 19th USENIX Symposium on Operating Systems Design and Implementation USENIX Association 2.1 What extensions should be allowed? Extension Manger Host Application uses User Extension Runtime Entry 1 ext1 Entry 2 ext2 writes Application Developer(s) Extension Program writes Extension Developer(s) Figure 1: A process extended with two extensions, ext1 and ext2, with associated extension entries, entry1 and entry2, respectively. The application developer(s) and extension developer(s) write the host application and extension program, respectively. The User uses the host application. The Extension Manager decides which extensions to allow and use in the deployment. Roles The system extension usage model considers four key principals. The application developers are a group of trusted developers who write the original application, while the extension developers are a group of trusted developers who create the extensions. System extensions assume that both the application developer(s) and extension developer(s) are trusted but fallible, so applications and extensions might be exploitable but are not intentionally malicious. Next, the system extension model includes an extension manager, a trusted individual that installs and manages the extensions; the model relies on the manager to be both trusted and infallible. Finally, users are untrusted individuals who interact with the extended application; users can be malicious and may try to craft inputs that would trigger vulnerabilities in otherwise benign code. Figure 1 provides a representation of an extended application and shows the role of each principal. The application developers write the host application. The extension developer creates the extension program, which can read and write application state and execute application-defined functions. The extension manager is responsible for deciding which extensions to use at each extension entry. Finally, users produce input that interacts with the host application and, indirectly, the extension program. In sum, our contributions are: • EIM, which allows users to specify fine-grained interconnectedness/safety tradeoffs. • bpftime, an efficient extension runtime system that enforces EIM specifications and isolation through two separate verification techniques and concealed extension entries. • An evaluation of EIM and bpftime demonstrating their usefulness and efficiency in 6 use cases. In the rest of the paper, we motivate EIM and bpftime (§2), describe EIM (§3), explain the design and implementation of bpftime (§4), discuss 6 use-cases (§5), evaluate bpftime(§6), describe related work (§7), and conclude (§8). 2 Motivation System extensions augment an application without modifying its source code to customize behavior, enhance security, add custom features, and observe behavior. By supporting application modifications without requiring source code changes, extensions allow a customized deployment to integrate maintenance updates from upstream repositories easily and can provide assurances of security and safety. The rest of this section discusses the principal roles of system extensions (2.1), provides an example web-server use-case (2.2), articulates the key properties of extension frameworks (2.3), discusses limitations of the current state-of-the-art (2.4), and articulates the threat model (2.5). USENIX Association 2.2 Web-Server Example Consider an instance of Nginx deployed as a reverse proxy. The application developers write the server, while the extension developers provide a suite of possible extensions to deploy on the system for monitoring, firewalls, and load balancing. The extension manager determines the extensions for the deployment and the privileges to provide each extension. First, the manager uses an extension program that monitors traffic to detect reliability issues [27]. Second, the manager deploys an extension program that implements a firewall that returns a 404 response for URLs that are indicative of SQL injection and cross-site scripting attack. Finally, the manager deploys an extension program to perform load balancing across the possible servers downstream from the proxy by periodically contacting downstream servers to measure system load [26]. 2.3 Key Extension Framework Features Extension use-cases require three key features: Fine-grained Safety/Interconnectedness tradeoffs. Extensions must be interconnected, i.e., able to interact with the host application. Host interactions include reading/writing host state and executing host-defined functions. The Nginx example extension programs highlights both of these features: extensions in the observability use-case require read access to host state, while the firewall, load balancing, and statistics 19th USENIX Symposium on Operating Systems Design and Implementation 559 Bug Bilibili [73] Software Nginx CVE-2021-44790 [47] Apache CVE-2024-31449 [42] Redis Summary Livelock (infinite loop) in an extension caused production outage. Buffer overflow in httpd’s lua module causes application to crash. Stack overflow in Lua script leads to arbitrary remote code execution. Table 1: Example issues caused by extension safety violations. extensions call Nginx functions to parse user requests and return responses. At the same time, an extension manager wishes to ensure that extensions are safe, i.e., do not harm the health of the deployment. Safety does not aim to thwart malicious extensions, but rather to ensure that a bug in the extension cannot harm the reliability or security of the host. Extension bugs have lead to catastrophic consequences in production, including livelock, system crashes, and remote code execution; Table 1 provides examples. There is no single definition of safety appropriate for all extension use-cases, since safety and interconnectedness are in tension. Instead, the manager should be able to follow the principle of least privilege and allow each extension to perform only the actions necessary for the interconnectedness required for its task. For example, the Nginx observability extensions only need to read from specific host states, but would be unsafe if allowed to write to them. In contrast, the Nginx firewall extension needs to read/write to a different set of host states, but would be unsafe if it were allowed to read the observability states. Safety rules that can be applied to each individual extension are fine-grained safety/interconnectedness tradeoffs. We note two desirable properties. First, the same extension entry may need to support different safety/interconnectedness tradeoffs since it may be useful for separate extension use-cases. Second, while the universe of possible interconnectedness features depends on the host application and thus requires a knowledgeable application developer to modify the host source code, the manager should be able to modify the safety/interconnectedness tradeoffs for their extensions without changing the original application. Isolation. Extensions must be isolated, i.e., not be harmed by the host application. Isolation does not thwart a malicious host application, but rather ensures that attackers cannot circumvent extension-based security by exploiting bugs in the host application. Isolation requires ensuring that host applications cannot modify extension states. The Nginx firewall extension is an example that relies upon isolation. Efficiency. Extensions should be efficient, i.e., execute at near-native speed, since they may be deployed on the hot path of production systems. For example, the Nginx load balancer 560 extension is invoked on the hot path for all user requests. 2.4 The Limitations of State-of-the-Art We describe the limitations of existing extension frameworks. Native Execution. A number of extensibility approaches execute the application and extension in the same execution context, essentially treating the extension as a component of the original program. Such approaches include those that use dynamic loading (e.g., LD_PRELOAD) and dynamic binary instrumentation [8, 39, 45]). These systems provide efficiency but neither isolation nor support fine-grained safety/interconnectedness tradeoffs. SFI-based tools. Many extension frameworks use SFI [69] to provide isolation, including XFI [16], NaCL [75], RLBox [44], and language-based sandboxes such as WebAssembly [23] and Lua [38]. Some of these tools (e.g., Lua, WebAssembly, and NaCL) do not provide an interface for safety/interconnectedness tradeoffs. Instead, they rely on the host application to check for safety violations, an approach that has proven buggy (see Table 1). Other tools (e.g., RLBox, XFI) lack fine-granularity or the ability to limit certain extension behaviors. Additionally, SFI-based tools are typically inefficient since they validate extension behavior at runtime [29]. Subprocess Isolation. Subprocess isolation systems, such as Wedge [7], Shreds [11], lwC [37], and Orbit [31], separate extensions from the host application through operating system isolation abstractions. Such systems ensure isolation. However, some lack fine-grained interconnectedness/safety tradeoffs (Lwc and Shreds), while others (Orbit and Wedge) could provide such tradeoffs, but only after code changes to the host application, because they are not designed for extensibility. Finally, such systems struggle to be efficient for frequently-executed extension use-cases, since they require context-switch-like overheads when switching between the host application and its extensions. eBPF uprobes. While it is usually used for kernel extensions, the extended Berkeley Packet Filter (eBPF) framework provides userspace extensions through the uprobe interface. eBPF uprobes are isolated from the host application, but do not support fine-grained interconnectedness/safety tradeoffs. Moreover, eBPF uprobes are not efficient. eBPF uprobes place a software breakpoint on every extension entry, causing the system to trap into the kernel to execute each extension. Aspect-oriented programming.. Aspect-oriented programming allows extensions, but existing aspect-oriented languages do not support safety/interconnectedness tradeoffs. For example, if an AspectJ extension were exploited [34], the attacker would have unrestricted ability to observe and modify the original host application. 19th USENIX Symposium on Operating Systems Design and Implementation USENIX Association 2.5 Threat Model State_Capability( name = " readPid " , 3 operation = read ( ngx_pid )) 1 2 The system extension threat model is as follows. First, it assumes that the extension manager accurately and completely identifies the correct safety/interconnectedness tradeoff for each extension entry. This means that a buggy extension cannot corrupt, crash, or hang an application through the interface that the extension manager allows. Second, the model assumes that the control-flow of the application cannot be modified or corrupted even if an application is compromised—essentially equivalent to control flow integrity [1]—since an attacker could otherwise circumvent extension execution. Given these limitations, the threat model considers two key threats. First, it considers buggy extensions that accidentally crash or hang the application through errant pointers, infinite loops, or stack corruption that are outside of their allowed safety/interconnectedness tradeoff. Second, it considers compromised applications that modify or corrupt the state used by extensions to alter the extension’s behavior. 3 Extension Interface Model (EIM) The Extension Interface Model (EIM) is a new model for specifying fine-grained interconnectedness/safety tradeoffs. Using the model, an extension manager can follow the principle of least privilege and specify a tradeoff that would enable extensions to perform their tasks (i.e., sufficiently interconnected) with minimal potential harm to the system (i.e., are sufficiently safe). The model is sufficient to prevent past extension bugs from harming production; e.g., an extension manager can use EIM to prevent each of the bugs in Table 1. An extension framework (e.g., bpftime) can later ensure that the extensions loaded into an application follow their EIM specification, thereby ensuring their safety. EIM’s key idea is to represent the extension features that might be necessary for interconnectedness or restricted for safety as a resource. Such resources include both classical systems resources, such as compute cycles, and host application interactions, such as the ability to call a host function or read/write to a host variable. EIM models the ability to use a resource as a capability [55, 60, 71]. An EIM specification encodes fine-grained interconnectedness/safety tradeoffs by specifying the set of capabilities that a loaded extension is allowed to use when configured for a given extension entry. An EIM specification consists of two separate components. First, the EIM specification includes a development-time configuration, prepared by an application developer, that specifies the set of possible safety and interconnectedness features (§3.1). Second, an EIM specification includes a deploymenttime configuration, prepared by an extension manager, that specifies precise interconnectedness/safety tradeoffs (§3.2). 3.1 Development-time EIM Specification USENIX Association 4 Function_Capability( name = " nginxTime " , 7 prototype = ( void ) -> time_t , 8 constraints = { rtn > 0}) 5 6 9 Extension_Entry( name =" processBegin " 12 extension_entry = " ngx_http_process_request " , 13 prototype = ( Request *r) -> int )) 14 Extension_Entry( 15 name =" updateResponseContent " 16 extension_entry = " ngx_http_content_phase " , 17 prototype = ( Request *r) -> int *) 10 11 Figure 2: An EIM development-time specification for a simplified version of the Nginx observability use-case. The development-time EIM specification encodes the possible interconnectedness features and extension entries of the host application. Since these features are tightly coupled with the host application, the development-time EIM is created by an application developer while developing the host application. A development-time EIM specification defines three sets of entries: state capabilities, function capabilities, and extension entries. Figure 2 provides an example of a development-time EIM specification for the Nginx observability use-case. State Capability. A state capability expresses the ability to read or write to a global state located in the host. A state capability includes a name and an operation of the form read(var) or write(var), which specify the capability to dereference and read or write to a variable, var, respectively. Figure 2 includes a state capability, readPid, that enables reading the global variable in Nginx that stores the process id of the current worker process (Lines 1– 3). Function Capabilities. A function capability expresses the ability to execute a host-provided function. An application developer defines a new function capability by providing the capability a name, the function prototype of the function, and a set of constraints that provide pre- and post-conditions for the function. Figure 2 includes a function capability in Lines 5–8. The capability specifies the ability to call Nginx’s function for getting a timestamp. The function constraints provide the post-condition that its return value is positive. Constraints. Constraints ensure that extensions safely use host-provided functions and can encode two things. First, constraints can encode binary relationships between arguments and return values, high-level semantic facts, and boolean operators over other constraints. Second, constraints can encode high-level fats about the function that an extension framework could support. Currently, EIM supports allocation facts indicating that a function’s return was allocated, IO facts in- 19th USENIX Symposium on Operating Systems Design and Implementation 561 Extension_Class( name = " observeProcessBegin " , 3 extension_entry = " processBegin " , 4 allowed = { instructions