
All right folks, I've been procrastinating a lot because of the workload and have been laying low, enjoying my comfort zone. I've been learning Linux kernel internals for a while, and if you're like me, you might have the urge to debug the kernel to see how it actually works and what magic **Linus Torvalds** and others at the Linux Foundation have crafted to get our system running. I'll try to keep this post as simple and beginner-friendly as possible, so if you're just starting to look into the kernel, you don't have to spend a month properly understanding and setting up the environment. Let's jump in and configure our cool Linux kernel debugging environment!

## Resonating with Linux Spirit

First things first, in the old-fashioned Linux way, we need to update our system and install tons of dependencies. You can do this using the following commands:

<Code language="sh">
{` 
   #This will take time depending on your internet connection
   sudo apt update && sudo apt-get upgrade -y
   sudo apt install gcc make bc build-essential libelf-dev libssl-dev 
   sudo apt install bison flex initramfs-tools qemu-system-x86 cpio
   sudo apt install git-core libncurses5-dev dwarves zstd

`}
</Code>

## Downloading the Source

Once you're done installing the dependencies, the first thing you'll need is the kernel you want to debug. You can download the source code for the Linux kernel from the [kernel.org](https://kernel.org) website. If you're interested in older kernel versions, you can visit the [mirror site](https://mirrors.edge.kernel.org/pub/linux/kernel/). And if you're a true Linux kernel enthusiast, you can use the `wget` command to download the source code.


<Notice type="tip">
   If you decided to learn linux internals, I suggest to pickup an kernel version and stick with it for consistent learning.
</Notice>

<Code language="sh">
{` 
   set $KERNEL_VERSION=6.5
   set $BASE=6.x
   wget -cv https://mirrors.edge.kernel.org/pub/linux/kernel/v$BASE/linux-$KERNEL_VERSION.tar.gz

`}
</Code>

<Notice type="info">
  You can set the `KERNEL_VERSION` and `BASE` environment variables accordingly here the BASE is set as **6.x** because the kernel version i want to use is **6.5** if you are interested in let's say **5.15.80** then you have to set the `BASE` as **5.x** and the `KERNEL_VERSION` as **5.15.80**
</Notice>

Once, the **wget** completes the downloading you have tarball of the linux kernel source code uncompress it using the following command

<Code language="sh">
{` 
   tar -xvf linux-$KERNEL_VERSION.tar.gz

`}
</Code>

## Configure the Kernel

Contrary to popular belief, compiling the kernel is not difficult. In fact, it's quite similar to compiling other utilities from source code. In true Linux tradition, we'll configure the Linux kernel by editing the `config` file.

<Code language="sh">
{`
   make defconfig  
   vim .config

`}
</Code>

### Important configuration

Here are the few configurations you must enable to make your debugging process easier and more enjoyable. After all, who wants to set breakpoints at machine code offsets? We want to be able to set them like this: `break fs/pipe.c:496`.

<Code language="sh">
{`
   # Make sure to have these configs on 
   "CONFIG_NET_9P=y" 
   "CONFIG_NET_9P_DEBUG=n" 
   "CONFIG_9P_FS=y" 
   "CONFIG_9P_FS_POSIX_ACL=y" 
   "CONFIG_9P_FS_SECURITY=y" 
   "CONFIG_NET_9P_VIRTIO=y" 
   "CONFIG_VIRTIO_PCI=y" 
   "CONFIG_VIRTIO_BLK=y" 
   "CONFIG_VIRTIO_BLK_SCSI=y" 
   "CONFIG_VIRTIO_NET=y" 
   "CONFIG_VIRTIO_CONSOLE=y" 
   "CONFIG_HW_RANDOM_VIRTIO=y" 
   "CONFIG_DRM_VIRTIO_GPU=y" 
   "CONFIG_VIRTIO_PCI_LEGACY=y" 
   "CONFIG_VIRTIO_BALLOON=y" 
   "CONFIG_VIRTIO_INPUT=y" 
   "CONFIG_CRYPTO_DEV_VIRTIO=y" 
   "CONFIG_BALLOON_COMPACTION=y" 
   "CONFIG_PCI=y" 
   "CONFIG_PCI_HOST_GENERIC=y" 
   "CONFIG_GDB_SCRIPTS=y" 
   "CONFIG_DEBUG_INFO=y" 
   "CONFIG_DEBUG_INFO_REDUCED=n" 
   "CONFIG_DEBUG_INFO_SPLIT=n" 
   "CONFIG_DEBUG_FS=y" 
   "CONFIG_DEBUG_INFO_DWARF4=y" 
   "CONFIG_DEBUG_INFO_BTF=y" 
   "CONFIG_FRAME_POINTER=y"

`}
</Code>

<Notice type="note">
 The above configuration list is not exhaustive. The kernel has thousands of configuration options, and they are set based on your specific needs. If you plan to debug a specific module or file system, make sure to enable the relevant configuration options for it. 
</Notice>

## Happy Make time

With the configuration complete, compilation is straightforward. Use the `make` command to build the Linux kernel. This process might take some time depending on your system resources.

<Code language="sh">
{`
   make -j $(nproc)

`}
</Code>

Tada! You're done with the compilation process. I decided to keep things simple and ignore many details. If you want to explore this process in detail, you can check out the blog written by my friend Ali [Linux Kernel Compilation](https://medium.com/@alirazamumtaz/linux-kernel-compilation-and-addling-a-custom-system-call-d6060baff429) and the handout by **Dr. Arif Butt** [Linux Kernel Compilation Handouts](http://www.arifbutt.me/wp-content/uploads/2022/04/Handout-Linux-Kernel-Compilation-1.pdf).


## Setting up the GDB


![image](/images/post/kernel_debugging_gdb.jpeg)

If you're used to debugging with GDB, you might be tempted to skip this section. But I highly recommend reading it! While I won't waste time explaining how to install GDB (you can use your package manager), I do want to emphasize the importance of using specific tools for the job.

While userland debuggers like GDB wrappers (GEF, pwndbg, Peda) are fantastic, they're not ideal for debugging the kernel. That's why [bata24 ](https://github.com/bata24) created an enhanced version of [GEF](https://github.com/bata24/gef/), specifically designed for kernel debugging. 

If you find this wrapper helpful, be sure to express your gratitude to both [bata24](https://github.com/bata24) and [Dawar Zaman](https://www.linkedin.com/in/hafiz-dawar-zaman-abbasi-55b674146/). Dawar introduced me to this enhanced GEF version.

The installation process for the enhanced GEF is quite straightforward. You can follow the instructions provided on the GitHub repository.

## Setting up the Qemu Environment 

![image](/images/post/kernel_debugging_qemu.jpeg)

Qemu is open source emulator which will run over linux the best thing about the Qemu is that it provides you option to debug the kernel over the socket connection.

### Setting up the Kernel

1. The first step for the qemu environment setup is to clone the repository from [github](https://github.com/DeathNet123/Kernel_Debugging_Env/)

Let's look at the directory structure of the repository. We have two scripts and three directories.
<Code language="sh">
{`
   $ tree -L 1
   .
   ├── copy2vm
   ├── img
   ├── kernel
   ├── poc
   ├── README.md
   └── startvm

`}
</Code>

The `startvm` script will run our QEMU machine and load the kernel present in the `kernel` directory. This is where you should place the `bzImage` file built by compiling the kernel.

2. Place the `bzImage` into the kernel directory by using the following command

<Code language="sh">
{`
   #Make sure to adjust your path accordingly because it depends where you compiled the kernel.
   cp ../linux-$KERNEL_VERSION/arch/x86/boot/bzImage ./kernel/arch/x86/boot/

`}
</Code>


<Notice type="info">
  The `bzImage` is present in linux-$KERNEL_VERSION/arch/x86/boot/
</Notice>

### Setting up filesystem

Now that we have the kernel set up, let's configure the filesystem! After all, we can't boot without it. Thankfully, a script provided in the Syzkaller Environment Setup makes building the filesystem a breeze. Shoutout to the anonymous hero who made this process easier!


<Code language="sh">
{`
   ./img/create_image.sh -d bookworm -f full

`}
</Code>

- `-d` flag is used to specify the distribution we want to use
- `-f` flag here stands for feature i opted for the full

### POC directory

Inside the `poc` directory, you'll find `poc.c` and a `Makefile`. This is where you can place the code which will trigger the kernel APIs or system calls you're interested in examining. Compile your code using the `Makefile`. 

When you run the `create_img.sh` script, it will automatically copy the compiled ELF binary into your filesystem. Alternatively, you can use the `copy2vm` script to copy the binary manually.

### Installation of third party libraries

Although, the -f flag we used will install pretty much everything but you might end up in situations where you have to install someother dependencies or other libraries you can do this by 
mounting the filesystem img file and then chroot into it. 

1. Mount the filesystem

<Code language="sh">
{`
   cd img && mkdir filesys
   sudo mount bookworm.img filesys/

`}
</Code>

2. Chroot into the filesystem

<Code language="sh">
{`
   cd filesys
   sudo chroot .

`}
</Code>

3. Install the dependencies, utilities or library

<Code language="sh">
{`
   apt install libfuse3-dev

`}
</Code>

4. Exit the terminal and you are done

### Play Time

I know you are now impatient and perhaps tired because after all you have done a lot of hardwork since setting debugging environment for the kernel is not piece of cake.
Without any further details, let's run the env and start debugging.

1. run the startvm script
2. load the `$KERNEL_VERSION/vmlinux` file in gdb
3. log in to the your system
   ![image](/images/post/kernel_debugging_startvm.png)
 - if you want to log in as root just type the root as username
 - for non privileged users type user as username and user as password
4. set breakpoint for example `break pipe_write` => this will set break pipe_write function in the kernel which is called everytime we write to the pipe
5. Attach the gdb to qemu using the following command `target remote :1234` once it will be attached it will hit breakpoint type `contine`
   ![image](/images/post/kernel_debugging_continue.png)
6. Run the example poc i provided which creates a pipe and writes to it inside the qemu machine 
7. In gdb the breakpoint at pipe_write will be hit
   ![image](/images/post/kernel_debugging_pipe_write.png)


<Notice type="info">
  The `vmlinux` is present in linux-$KERNEL_VERSION/ => it's present at root of the kernel source code once you have compiled it.
</Notice>

## End 

I hope you enjoy this writeup it took me whole day to write this down i hope this helps you out. Happy Debugging...

~Αρσλάν


Kernel Debugging Setup


Our society may seem scarce in this vast universe, yet it holds many untold stories. Observed from afar, it could present a humorous illusion of a well-defined, interconnected, and streamlined system. Imagine two people walking towards each other: they adjust their paths seamlessly, unless one is blind or harbors ill intentions, But zoom in closer, and the picture isn't so harmonious. The smiles mask hidden scars wanting desperately to be healed, her charming words are nothing but a deceitful play, the confident strides and promises are mere an effort designed to mislead. The promises of standing by your side, don't these thoughts ultimately makes you feel more alone? It's ironic how this society of seemingly connected components ultimately feels like a collection of isolated components. Each component, driven by its own desires, but it requires fuel to function. Fuel? you may ask, Perhaps Blood? Not literally, ofcourse but the energy extracted from ourselves and others. 

Ironically, despite the underlying isolation, our society still finds ways to interconnect these isolated components. It seems like a paradox, doesn't it? Our desires have taught us to present lies in the form of truth. Some of us have found ways to connect with each other as followers of some religion or as citizens of a nation. Some have mastered the art of bringing these components together to form a powerful yet dull system to act as their weapon and shield. The wise have likened society to chess or checkers, but it's more complex than them, yet it can fall like dominoes if the cards are played correctly.

Every single one of us has a story. Every action performed, every sentence, every word spoken is an opportunity to learn about their stories, to understand their desires, and to see through their foul play. While there might be satisfaction in exposing someone's blatant lies, some of us have the lens to see beyond that. Should I take you on an adventure where you will look at the world through a lens, the lens of Excaliats 

![image](/images/post/placeholder.jpeg)

The Untold Tale Of Your Life


I began my bachelor's journey in 2019, like many others, filled with excitement to learn and accomplish great things during my university experience. However, reality hit me hard, and I soon realized that university is less about teaching technical subjects and more about imparting life lessons and prompting personal growth.

## Initial Days

Like every other innocent student, I considered CS as a practical program where students were given the opportunity to learn by doing hands-on experience. But oh boy, I was ever wrong about it. My first semester was full of theoretical stuff, which seemed like it was from the era of my grandparents, with math as the main course and physics as the side dish. Programming was absent, like ketchup in Italian restaurants.

Honestly, I didn't have much time to dwell on the reality shock I just experienced because I was busy coping with all the math stuff. The fact that I joined a few days before the midterm meant I had to grind my way through all these boring and, in some cases, interesting subjects that I wasn't initially smart enough to grasp. I am an introvert, but during my initial days at university, I made a friend whom I didn't initially like. However, as time passed, he became one of my good friends.

Having him didn't help me much because, let's say, he wasn't very bright when it comes to studies, so I had to actually grind through it. As time passed, I made a new friend during the mid-exams, and he was the bright one. He actually helped me in my first semester. Through him, I made another friend who was delusional and had some irrational and impractical ideas, but hey, who is complaining? I'm not the most practical person either.

I am an introvert, but I am not rude. I tend to approach people nicely with a smile, yet some out there consider me rude. However, I really just mind my own business. My unique and somewhat displeasing image still managed to win one more friend and build a good relationship with a few of my classmates by the end of first semester.

In a quick time montage, the start of a new semester had me motivated and ready to achieve straight A's and maintain 100% attendance. However, someone, somewhere in China, decided to cook and have a bat as their meal.

## Covid-19 and Me

I really do not like the blame game, but I think COVID played an important role. As I write this line down, I feel like COVID might not be as I initially thought. You can clearly see a shift in tone from being critical to being somewhat less critical and more favorable.

> Placing the blame for your own mistakes on others might be a good move in the realm of power, but never emotionally support it. Learn from your mistakes; you cannot play the same drama twice.
> -*Αρσλάν*

Well, COVID-19, which was a horror for many people, never really made me worried. In fact, like many other students, I was amused by the fact that, like other educational institutes, our university had also been closed, and we might not have to go to university for a month. But obviously, I was not looking forward to the one and a half years of lockdown.

We never really got a one-month vacation because our university started conducting online classes within a week. But it didn't bother me much because I was just happy that I didn't have to use public transport anymore. Also, I used to sleep throughout the classes.

Remember when I said that I was shocked because there was no programming course? In the second semester, there was a course called Programming Fundamentals, but I slept throughout its classes too. I can swear to God I really never attended a single class, but I still managed to get an A in it. The fact that Programming Fundamentals was actually focused on the fundamental constructs of programming gave me leverage because I had already mastered them in my college days.

My second semester seemed like heaven to me. I used to sleep throughout the classes and listen to songs in my remaining time of the day, which is really awkward because I hardly used to listen to music. I am an introvert, but deep inside, it was hard for me to accept that I am locked inside my home without knowing when it will be over.

With little to no effort, I managed to get straight As in the second semester, which gave me this false positive encouragement to continue what I was doing. I didn't change anything during the whole pandemic, slept throughout the classes in my following semesters, and didn't put any effort into some of the important and core courses. Although I was better than many, I should have approached and aimed for further heights.

> Ignorant and delusional individuals may fancy themselves living a life akin to 
  The Sun King, but in reality, they might end up facing a fate similar to Ch'in Shih Huang Ti.
> -*Αρσλάν*

## Back to the University

With COVID phasing out of the country, Our University started our fifth semester
with good news of physical classes. I enjoyed the first two weeks because It was
really good to actually meet with friends with whom my bounds got stronger during
the lockdown.

The fifth semester not only reopened our university but also introduced two of my favorite courses: Operating Systems and Algorithms. I have always been fascinated by Operating Systems, and during my Data Structures course, I developed an appreciation for Algorithms. A well-crafted algorithm is a work of art, a testament to the intelligence of its creator.

Even though not much of it was boring in university, I lost the sense of competition and the drive to work hard, improve, and achieve. A few days before midterms, I realized the mistake I had committed. After the realization of my mistake, I once again grinded through it.

After the midterms, I decided to participate in the ICPC, and I think it was a good idea because it gave me the opportunity to realize how much I still had to learn. After the competition was over, I began to learn and truly started putting effort into Algorithms, which led me to score a perfect score in the final exam of the Algorithms. 

I am not proficient in algorithms; in fact, recursion still baffles me, and there's much more to learn. Despite my struggles, I still have a fondness for them. However, at the start of my sixth semester, I discovered something even more interesting than algorithms that kept me captivated.

## The Shift, The Rise 

The sixth semester offered us the course "System Programming," which was already of my interest since it is related to the OS, and the fact that it was being taught by one of my favorite teachers excited me.

During the first week of the semester, I realized what I wanted to do in my life, and I felt like I could really spend some time doing low-level binary exploitation. I decided to pay a visit to my professor and told him that I really didn't want a boring developer job, as I had realized that development was not something I fancied.

I had a very interesting interaction with him because at first, he said I should not spend time in binary exploitation since it is dry and I would not like it. But in the end, he said, "If you are really willing to do it, try to complete the System Programming (SP) course content, then we can discuss this further."

I took it as a challenge and completed the entire content of the course during the summers, which concluded with exploiting basic buffer overflow and popping up the shell. I was so immersed in it that I bypassed the NX bit mitigation on my own. I know it's not a big deal, but I can remember staying up the whole night and making sense out of the weirdly written blogs, somehow managing to bypass the NX bit on my own.

After completing the course content and bypassing the NX bit, I emailed my professor, and he asked me to give him a demo on Google Meet. The next day, I presented the exploit which exploits the web server with NX bit mitigation enabled. After the demo, he asked me to read the paper he had just published, which was on Stack Smashing with all mitigations in place.

If you are wondering about the paper, here is the link [Bufferoverflow](https://www.mdpi.com/2076-3417/12/13/6702). If you are also getting started in binary exploitation, I highly recommend reading it.

I managed to complete the paper with practical exploits in my hands within a week. A few days later, I decided on the Final Year Project. I opted for conducting N-Day research on CVE-2019-2215, but before I could start this, I still had to learn a lot of prerequisites.

For the next few months, I learned the basics of binary exploitation like a samurai skillfully slicing through a forest filled with bamboo.

## September of 2022

As I learned and enjoyed the process, an unfortunate event occurred in my life. It was challenging to accept what had just happened, and though I continued learning, it affected me both mentally and physically. Despite the desire to scream out loud, I found solace in laughter with my friends, choosing not to burden anyone with my struggles.

The thrill and excitement in what I was doing started to fade. I persevered because I had no other choice. Coordinating with my FYP group became difficult, especially since I was the group leader with the responsibility to manage things efficiently. Additionally, this situation not only impacted my coordination but also slowed down my learning process. Managing my emotions became crucial, as it eventually hindered my ability to learn things I otherwise would have.

## Back to sanity 

They say the new year brings joys and a sense of motivation, and oh boy, was I ever in more need of motivation. I decided to move on and started to pick up the pace again. By this time, I managed to somewhat have enough overview of the userland and was ready to delve into kernel land exploitation.

As I was absolutely excelling in my exploration of the kernel security module at Pwn College, I saw an opening at Ebryx and decided to give it a shot. To my surprise, I was hired and on-boarded a week before the start of my eight semester.

I completed most of FYP research during the probation period at Ebryx and by the end of August I graduated from the university.

#### Getting Lost

Working at Ebryx was the first time I ever had a real job, and although I enjoyed it, soon it became hard to manage. Despite my job being related to kernel security, my learning curve declined in the past few months.

#### Root Cause

A month ago, I decided to reflect on my life once again and came to the conclusion that I was not managing my time properly, and I was not able to dedicate much time to my personal growth. I decided to allocate a fixed amount of time to learning about the Linux kernel and different techniques used in kernel exploits, but I failed to do so multiple times.

## What is sparklers about?

I know this was supposed to be the main topic of this blog, but the background was important. Not only is it full of lessons, but it also makes the need for the blog more clear.

During my initial days of FYP research, I realized that there is a need for content that explains the initial background of Low-Level Exploitation. Without a doubt, there are many great resources out there, but they lack the initial background and journey of how to get started.

I am lazy, but I am somewhat responsible with this blog. I might get some sense of responsibility to actually come up with some content for it. In order to have content, I'll actually learn new things, which will eventually put my learning curve back on track.

- My blog will mostly be related to Low-Level Exploitation on both userland and kernel-land.
- I like algorithms, so eventually, I will be writing about them too.
- I might write some stories as well.
- I will also write stuff related to self-improvement, as this blog is an effort on my end.

## Conclusion

I think I ended up writing more than I wanted to, but I hope you can learn the lessons from the mistakes I have made in the past because regrets are not easy to deal with. I really like to enjoy the privilege of being better than others, but such privilege cannot be achieved easily; we have to strive for it. If you are just getting started, let it be a lesson not to waste time and never stop improving. If you are like me, indulged in regrets, let it be motivation because you are what you create, and consider yourself.

> Reimagine and recreate yourself, and consistently set the bar high. You will reach heights that others cannot.
> -*Αρσλάν*

Showing posts fromdebugging

Kernel Debugging Setup