Inside the Windows Cache Manager

Introduction

The cache is an integral part of the operating system and its hybrid kernel. Roughly speaking, it's just a virtual memory region in the kernel address space, on which the Cache Manager maps file data to provide quick access to them in the future. This access is frequently used by the File System Driver (FSD) or the Windows Memory Manager (VMM). Instead of reading file data from disk every time a user or system needs to access to it, the OS kernel calls the Cache Manager in an attempt to get this data from memory. In turn, the Cache Manager is a set of function in the kernel executable file ntoskrnl.exe, which starts with a prefix Cc. These functions are private, so to get to their names, you need to configure the symbol server settings in WinDbg or IDA.

Learning the Windows Cache Manager is quite a difficult task for beginners. This Windows kernel subsystem is closely related to the VMM, so if you don't have enough knowledge in it, try to understand the basic concepts without going into complicated technical aspects. In addition, you should have some knowledge in the field of file system drivers (FSD), because they are the most frequent clients of the Cache Manager. It's worth to note that the cache concept exists only at the level of file system, lower drivers on the device stack like the volume manager, partition manager, disk driver, and disk port driver don't use it.

This blog post is dedicated to the technical aspects of the Windows Cache Manager and designed for the skilled Windows Internals readers. If you lack knowledge on this topic, read the corresponding chapter in the Windows Internals book and then get back to this post. I would say that this blog post is some kind of technical addition to the chapter about the cache in the book (or I hope it claims..).

Let's take a look at some terms for newbie. 

Working Set (WS) - the set of pages in the user mode or kernel mode address space that are currently resident in physical memory. The kernel mode working set called System Working Set.

PTE (Page Table Entry) - a structure that is used by the CPU and VMM to translate virtual addresses to physical ones.

Proto-PTE (Prototype PTE, PPTE) - a special type of so called Software PTE that is used only by the VMM (not CPU) to work with section objects (memory-mapped files) and serves as an intermediate level for the translation mapped section pages to the real hardware PTE. PPTE is a key structure for understanding the section objects.

Segment Control Area (or just Control Area, CA) - a structure that contains information required for performing I/O operations with file data in or from the mapped file. It's stored in the non-paged pool. With the help of CA the VMM can address the same file as binary and as executable.

The basic concepts

The memory region in the kernel mode address space occupied by the cache starts with the value of the VMM variable MmSystemCacheStart and ends with the value of MmSystemCacheEnd. Thus, if X - is a pointer to the memory region that belongs to the cache, then MmSystemCacheStart<=X<=MmSystemCacheEnd. File data in this region are mapped into slots, 256KB blocks of data. The cache has two features, which are a consequence of the fact that the VMM is responsible for its internal implementation.

• The section objects maintained by the VMM are used to map file data into slots. Thus, the VMM is responsible for paging file data.
• Cache virtual pages can be unloaded to the page file.

These features emphasize the fact that the Cache Manager doesn't know for sure whether the file data is in physical memory or not. Undocumented structure called Virtual Address Control Block (VACB) is used to describe the cache slots, which are reserved in the paged pool. The control blocks are addressed from CcVacbs variable. Each of these blocks controls a specific slot. The variable CcNumberVacbs stores the number of slots.

• CcVacbFreeList. It's a list of free VACBs, i e those VACBs that are ready for use.
• CcVacbLru. A list of all other structures. A VACB has free status if its .ActiveCount field is zero. When reused, the slot address is re-mapped. The following WinDbg command confirms these facts.

r eax=0; !list "-t ntdll!_LIST_ENTRY.Flink -x \"r eax=@eax+1;? @eax;? @$extret-10; dt nt!_VACB @$extret-10\" nt!CcVacbFreeList "

We can use it to print free VACBs and their numbers, for example.

Next from the first - 0x81954fb0=81954fa0 + 10.

We can do the same for the remaining (CcVacbLru).

r eax=0; !list "-t ntdll!_LIST_ENTRY.Flink -x \"r eax=@eax+1;? @eax;? @$extret-10; dt nt!_VACB @$extret-10\" nt!CcVacbLru"

Most of these structures have initialized shared maps and are mapped to the cache. If sum up the last VACB numbers from both lists, u get something like this.

14b+6b3 = 7fe

dd CcNumberVacbs l1

8055f670 000007fe

The virtual address of a specific slot will refer to the PTE pointing to the PPTE, the latter is linked to the subsection that describes the file (usually there's a one subsection that linked to the shared map and maps the file as binary, look at MmMapViewInSystemCache). You can learn more about PPTEs from my blog post here.

The cached file is described by two important structures called a shared cache map and a private cache map. Unlike the shared cache map, the private cache map isn't so interesting for exploring, because it's used for so-called intelligence ahead-read. Let's take a look at the shared cache map. It represents a structure that the Cache Manager maintains for caching a specific file. As in the case of control areas, which are unique for disk files (one is used to map the file as binary and another one to map it as an executable), the shared cache maps are unique as well and are addressed with SECTION_OBJECT_POINTERS structure, the latter is held by the FSD in the FCB structure of a specific file. Thus the Cache Manager knows what exactly slot describes a specific file via VACB, which stores a pointer to the shared cache map.

The cache manager can find this structure for each opened FileObject, because it points to SECTION_OBJECT_POINTERS (FileObject->SectionObjectPointer). The shared cache map is described by the following structure.

The Cache Manager can find out quickly which of the specific files are already mapped (i e have used slots), the shared cache map points to the VACB index array. The first element of the array points to the first 256KB of the file, the second to the next 256KB and so on. In case if the file has size not more than 1MB, i e can fit in four slots, the array InitialVacbs from the shared cache map acts as an index array, otherwise the array is allocated from the paged pool. In any case, the pointer to it is stored in the Vacbs field. All shared cache maps linked into a list with the head in PrivateList (&SharedCacheMap->PrivateList, &PrivateCacheMap->PrivateLinks). Moreover, all shared cache maps are also linked into lists with SharedCacheMapLinks. There's a special function of the Cache Manager CcInitializeCacheMap, which is called by the FSD, and is responsible for initializing a shared cache map (if it hasn't been created yet), creating a section object and creating a private cache map.

VOID CcInitializeCacheMap (__in PFILE_OBJECT FileObject, __in PCC_FILE_SIZES FileSizes, __in BOOLEAN PinAccess, __in PCACHE_MANAGER_CALLBACKS Callbacks, __in PVOID LazyWriteContext) 

This function is responsible for.

• It creates and initializes the shared cache map if it doesn't exist yet (FileObject->SectionObjectPointer->SharedCacheMap is zeroed), SharedCacheMap->FileObject is initialized by the first file object for which the map is created.
• It creates the section object with MmCreateSection. Further, this section will be used to map file data into cache slots.
• Creates a VACB index array with CcCreateVacbArray. This function initializes fields .Vacbs and .SectionSize.

If the FSD needs to read data from the cache, it calls CcCopyRead.

Internally, the Cache Manager maps parts of file data with help of CcGetVirtualAddress, this function returns the base address of the data in memory. The function operates only with one VACB and one slot.

The Cache Manager uses the following function to map file data.

PVACB CcGetVacbMiss (IN PSHARED_CACHE_MAP SharedCacheMap, IN LARGE_INTEGER FileOffset, IN OUT PKLOCK_QUEUE_HANDLE LockHandle, IN LOGICAL HasBcbListHeads)

The function searches for a VACB to map file data into cache slots and maps it using MmMapViewInSystemCache (the value for the file mapping is taken from &Vacb->BaseAddress).

The following WinDbg script explores the cache.

Take a look at some printed data from my system.

Therefore, the $Mft file is cached at 0xd90c0000 with an offset 0x00ac0000 from its beginning. Take a look at it.

Let's ask the question how does the kernel maps sections into the cache. The answer is located in the MmMapViewInSystemCache function. Before analyzing it, point out some facts.

• The cache PTEs start from address that stores in MmSystemCachePteBase (usually it matches the address of the beginning of the page table, 0xC0000000).
• Free cache slots are linked to MMPTE_LIST list to provide quick access to them (see WRK for more info about this structure). The pointer to the head of the list is stored in MmFirstFreeSystemCache. The field .NextEntry in MMPTE_LIST stores a value that points to the next field (next block of PTEs). This value is relative to MmSystemCachePteBase. The MiInitializeSystemCache function is responsible for initializing of the PTEs cache list. The PTEs for the cache are reserved by adjacent blocks, i e to cover 256KB, the block is included 64 PTEs, see MiInitializeSystemCache.

MmMapViewInSystemCache maps only one cache slot, i e CapturedViewSize argument can contain a value-size of no more than 256KB. Below you can see is a pseudocode for the typical behavior of MmMapViewInSystemCache. Take a look at the comments, they explain the operations to be performed.

#define GetVirtualAddressByPte(PTE) ((PVOID)((ULONG)(PTE) << 10))