Hardening Blockchain Infrastructure with Papora
In the past three years, we have witnessed hundreds of blockchain hacks resulting in losses of billions of dollars. Although many of them are related to smart contracts, the underlying blockchain client software is the crucial component to the achievement of consensus and the protection of the network.
According to the statistics provided by ethernodes.org, over 95% of Ethereum mainnet nodes are running Ethereum client software on Linux. It turns out that the underlying operating system is actually a substantial part of the attack surface for Ethereum and many other blockchains. To scrutinize critical bugs at the operating system level, Amber Group has conducted a joint research with Peking University since early 2022. Now, we are thrilled to share some preliminary results and present you the Papora fuzzer.
Papora
The Papora fuzzer, named after the ancient people from the hometown of the main contributor of this project, successfully identified double-digit bugs or vulnerabilities in the Linux kernel. By the time of writing this blog, we have nine patches going through the v6.2 release candidate process which will be finalized in a couple weeks. Besides, we have three patches (1, 2, 3) which have been accepted by the subsystem maintainer. In particular, one of the critical vulnerabilities we identified could be exploited to perform local privilege escalation attacks. It means if a victim happens to execute an executable file crafted by a bad actor, the whole machine would be compromised with the highest privilege. If that machine is running a hot wallet, the bad actor could even get the private key and walk away with crypto assets depending on how the victim strengthens the key management mechanism.
Inspired by Janus and Hydra, Papora generates the context-aware workload for fuzzing filesystems of the Linux kernel. The first thing we tried was setting up Janus on Linux kernel v5.19.x. However, we soon realized that the Linux kernel library (LKL) was not up-to-date. Therefore, porting LKL to the latest kernel turned out to be the first dependency we need to resolve. Besides, we found the must-have feature for detecting memory corruption bugs was not supported by LKL such that we spent a few more weeks integrating KASAN support into LKL before bootstrapping the fuzzer for the first time. Now, we are able to maintain the whole thing, keep it up-to-date, and keep contributing valuable patches to the community.
NTFS3
We chose NTFS as our first target to evaluate the feasibility of Papora. In particular, NTFS3 is a full-fledged NTFS read-write driver firstly upstreamed into the Linux kernel in late 2021, which makes it a good target to start with. In addition, NTFS3 is the open source version of the closed-source NTFS implementation by Microsoft. We believe that there are some hidden or undocumented things we could find out and fix them.
Similar to what had been done in Janus, we built our own NTFS3 parser to greatly improve the efficiency of Papora. The parser extracts important metadata for mutation and maintains the integrity of the mutated metadata for skipping the invaluable paths. Specifically, the NTFS3 parser extracts and parses two important parts of the metadata: “Partition Boot Sector” (PBS) and “Master File Table” (MFT).
Partition Boot Sector
The “Partition Boot Sector” located at the start of the volume holds some important boot information. The parser retrieves the following key fields from it for better mutating other parts in the metadata.
OEM ID
The magic number which indicates that this is an NTFS file system. Since the only valid value of this field is “NTFS” (“NTFS” with four spaces), the NTFS parser always fills this field with a correct value after each mutation. Otherwise, the NTFS3 driver checks this field and refuses to mount if the content of this field is randomly mutated.
Bytes Per Sector
The number of bytes in a disk sector. This size should always be larger or equal to 256. And, it should be the power of 2 (e.g., 256, 512, 1024, etc). The parser modifies it to the nearest valid value if it is not a legit number.
Sectors per Cluster
The number of sectors in a cluster. If the value N is greater than 0x80, it means each cluster has 2^(256-N) sectors.
Bytes or Clusters Per MFT Entry
The size of the MFT entry. If the value is between 0 and 127, it represents the number of clusters per MFT entry. If the value N is between 128 and 255, it means the size of the MFT entry is 2^(256-N) bytes.
MFT Cluster Number
The cluster that contains the MFT. This field is crucial to parse the next important part of the metadata — the Master File Table (MFT).
Master File Table
The following code snippets illustrate how the parser optimizes the fuzzer based on the understanding of MFT data structure.
void ntfs_fuzzer::ntfs_parse_mft_rec(RECORD_NUM idx) {
// TODO: remove it for next new fuzz
//return;
FileRecordHeader *mft = (FileRecordHeader *)malloc(bytes_per_mft_rec);
if (mft == NULL)
FATAL("[-] malloc for mft record failed.");
uint8_t *end = U8P(mft) + bytes_per_mft_rec;
// skip parsing if AFL mutation cause OOB read for MFT records
if (mft_offset + idx * bytes_per_mft_rec > image_size_ - bytes_per_mft_rec)
return;
// copy idx record from MFT
memcpy(mft, U8P(image_buffer_) + mft_offset + idx * bytes_per_mft_rec,
bytes_per_mft_rec);
// save the record
save_meta_rec(mft_offset + idx * bytes_per_mft_rec, bytes_per_mft_rec,
TYPE_MFT, idx);
switch (idx) {
// case range: GNU C extension
case MFT_REC_MFT:
case MFT_REC_LOG:
case MFT_REC_BOOT:
case MFT_REC_BADCLUST:
case MFT_REC_UPCASE:
// skip these data
return;
default:
break;
}
...The MFT keeps the metadata for all files and directories, including names, creation / modification date, access permissions, even the metadata itself. In particular, the first 27 entries (i.e., metafiles) of the MFT are used as the metadata of metadata. Some of them define and organize the whole file system. For example, entry 0 ($MFT) describes all files on the volume, including file names, timestamps, stream names, and lists of cluster numbers where data streams reside. Some other entries are reserved for extension.
As mentioned earlier, the size of an MFT entry is defined in the Partition Boot Sector. In most cases, it would be 1024 bytes. Each MFT entry has the following fields that the parser handles more carefully:
MFT Entry Header
The signature to indicate the type of this entry (e.g., FILE, INDX, etc.) is stored in the header. Besides, the offset and size of the fix-up array could be found here.
Fix-up Array
Used as the checksum for data integrity. The parser would re-compute the checksum to skip the invaluable paths due to checksum failures.
Array of MFT Attributes
Attributes that describe the MFT entry.
Padding
The parser would fix up all non-zero padding bytes.
Slab Out-of-Bounds Write
As mentioned earlier, Papora successfully identified a critical vulnerability which could be exploited to launch privilege escalation attacks. In fact, it is a vulnerability related to out-of-bounds memory write. We crafted a malformed NTFS image and implemented a PoC to demonstrate that the vulnerability could be exploited that way. The following KASAN log is thrown by the kernel once we run the PoC:
[ 259.209031] BUG: KASAN: slab-out-of-bounds in ni_create_attr_list+0x1e1/0x850
[ 259.210770] Write of size 426 at addr ffff88800632f2b2 by task exp/255
[ 259.211551]
[ 259.212035] CPU: 0 PID: 255 Comm: exp Not tainted 6.0.0-rc6 #37
[ 259.212955] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
[ 259.214387] Call Trace:
[ 259.214640] <TASK>
[ 259.214895] dump_stack_lvl+0x49/0x63
[ 259.215284] print_report.cold+0xf5/0x689
[ 259.215565] ? kasan_poison+0x3c/0x50
[ 259.215778] ? kasan_unpoison+0x28/0x60
[--------------------------------------------------------------------- skipped –-----------------------------------------------------------------]
[ 259.240715] do_syscall_64+0x3b/0x90
[ 259.240934] entry_SYSCALL_64_after_hwframe+0x63/0xcd
[ 259.241697] RIP: 0033:0x7fc6b26e4469
[ 259.242647] Code: 00 f3 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 088
[ 259.244512] RSP: 002b:00007ffc3c7841f8 EFLAGS: 00000217 ORIG_RAX: 00000000000000bc
[ 259.245086] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fc6b26e4469
[ 259.246025] RDX: 00007ffc3c784380 RSI: 00007ffc3c7842e0 RDI: 00007ffc3c784238
[ 259.246961] RBP: 00007ffc3c788410 R08: 0000000000000001 R09: 00007ffc3c7884f8
[ 259.247775] R10: 000000000000007f R11: 0000000000000217 R12: 00000000004004e0
[ 259.248534] R13: 00007ffc3c7884f0 R14: 0000000000000000 R15: 0000000000000000
[ 259.249368] </TASK>As stated in the log, a ‘slab-out-of-bounds’ action is triggered while the kernel is accessing an NTFS object, which is a heap overflow that happens in the kernel space. Also in the KASAN log, we can see the out-of-bounds write enables us to overflow 426 bytes into the memory chunk followed by the NTFS object.
To exploit this vulnerability, we need to grasp some basic knowledge of how the Linux kernel performs kernel space memory management. Linux kernel uses the slab allocator to allocate and release memory chunks. To allocate a memory object, slab firstly asks the kernel’s page allocator for free memory pages. Slab then splits these pages into fixed size chunks. For example, if two 64-bytes chunks are allocated in a short period of time, they are likely to be in the same memory page.
Back to the out-of-bounds write vulnerability. We can firstly allocate an NTFS object. And then use heap spray technique to place another type of objects with the same size right after the NTFS object. Therefore, we can trigger the out-of-bounds write to manipulate the target object we prepared. But what target object should we manipulate to perform the privilege escalation attack? Based on our experience, we have three choices:
Object with Function Pointers
If the target object has a function pointer, we can overwrite the function pointer with a controlled address. Then, we can hijack the control flow and escalate the privilege of the current process.
Object with Reference Count Fields
If the target object has a reference count field, we can convert the vulnerability into a use-after-free vulnerability by manipulating the reference counter as we have plenty of ways to exploit an UAF loophole.
“struct iovec”
Back in 2016, an out-of-bounds write vulnerability CVE-2015–1805 was actively exploited in the wild. Those exploits used the “struct iovec” object. What’s special about the object is that the object can be placed into a chosen sized slab chunk by controlling the object allocation count. And the object has only two fields, “iov_base” and “iov_len”. “iov_base” is a pointer type that points to a kernel buffer that will be filled by IO operations and “iov_len” is the buffer length. By overwriting these two fields using out-of-bounds write, we can make “iov_base” points to any kernel address. Then, we can write arbitary data to that address, for example, overwriting a function pointer for hijacking the control flow.
Take-Aways
- We have implemented a fuzzer, dubbed as Papora, which identified dozens of vulnerabilities in the NTFS file system of a Linux kernel. So far, nine among them were merged into the upstream Linux kernel repository.
- We have constructed a PoC which demonstrates that malicious users can exploit an out-of-bound write on heap discovered by Papora to conduct local privilege escalation attacks.
- We have integrated KASAN into the Linux kernel library (LKL), and ported the LKL to the latest Linux kernel (v5.19.x). More security analysis on the latest Linux kernel can be performed based on the LKL.
- We have implemented the first NTFS3 parser, which could extract only metadata from a given NTFS3 image and dynamically maintain the integrity of the metadata. After parsing, Papora could efficiently and effectively conduct mutation and fuzzing.
- We plan to migrate the core of Papora from AFL to AFL++ and/or other AFL forks to leverage the improvements done by talented researchers. Last but not least, we are simplifying the parser development process by extracting the logic automatically from the file system spec, which could help Papora to easily support new subsystems in the future.
Disclaimer
The information contained in this post (the “Information”) has been prepared solely for informational purposes, is in summary form, and does not purport to be complete. The Information is not, and is not intended to be, an offer to sell, or a solicitation of an offer to purchase, any securities. The Information does not provide and should not be treated as giving investment advice. The Information does not take into account specific investment objectives, financial situation or the particular needs of any prospective investor. No representation or warranty is made, expressed or implied, with respect to the fairness, correctness, accuracy, reasonableness or completeness of the Information. We do not undertake to update the Information. It should not be regarded by prospective investors as a substitute for the exercise of their own judgment or research. Prospective investors should consult with their own legal, regulatory, tax, business, investment, financial and accounting advisers to the extent that they deem it necessary, and make any investment decisions based upon their own judgment and advice from such advisers as they deem necessary and not upon any view expressed herein.
