Building custom (real-time) kernel for Atom (X86)

I found the sound quality really improved by using real time kernel after building real time kernel for Cubieboard2. Therefore I started working on real time kernel for my Atom PC.

I have a small form factor (about size of a 3.5" hard disk) Atom motherboard, the

EPIC-N28 VER:2.1

Some highlights:

• Atom MT2800 with fanless heatsink  
• DDR3 SODIMM slot x 1
• VGA, HDMI, LVDS
• RTL8111E Gigabit Ethernet Adaptor x 2
• Realtek ALC662 6ch Hi-def Audio on-board 
• USB 2.0 x 6

When building custom kernel, I aim to build only those I needed for Voyage MPD, so I would like to build the following features:

• CPU architecture select "Intel Atom" (which means some of the advance instruction sets not appeared in Atom will not be supported)
• Real time support (Fully preemptive)
• USB 2.0 controller and USB audio, no other USB device drivers will be built (such as USB network card, USB mouse, TV card, scanner, printer, etc)
• No AGP, only PCI and PCIe controller
• VGA only, no framebuffer, no 3D support.
• SATA and SCSI (no legacy support like PATA)
• IP v4 only without advance features like tunneling, multicast, etc
• Driver (module) for realtek R8169 only (this is compatable with RTL8111E)
• No virtualization support
• No debugging / verbose support for every hardware!

I think that today's Linux kernel is very mature in the sense that building all the features in modules doesn't hurt the performance obviously. However, I believe that the system will still load those modules if it detects a hardware is supported, which will introduce some unnecessary latency inevitability. So the best way is not to provide those modules and not to build into the kernel core. Of course, the baseline is to have a kernel build that is bootable and be able to play music flawlessly. So I did compiling and testing the same kernel version with different combination of options on and off.

What I am trying is say is that, when you build your own kernel for a x86 PC, bare in mind that it may not success at once. You may need to turn the options on and off and try until you find one workable configuration with least number of modules loaded / built into the kernel.

Ok, let's start building the kernel!

First, get a disk drive (USB drive or SATA hard disk) with 4G (8G or more preferred) available. Load it with Voyage current version. You may refer to the "readme" of Voyage MPD, section 2 - "Installation" for the official setup guide. I used version 0.9.2:
http://svn.voyage.hk/repos/voyage/branches/voyage-live/0.9.2/config/includes.chroot/README

The official installation guide assumes you run on a Linux environment. If you don't have another Voyage MPD ready, the easiest way is to download Ubuntu 12.04 32bit live CD and run in Ubuntu Desktop.

When you have the Voyage MPD disk ready, try to boot it in your target machine and make sure it can boot up flawlessly. After that, you can start the following steps on the same machine, or you may boot the Voyage MPD on another faster PC (say an Intel i7) so that you can build kernel much faster.

1. Connect a LAN cable (with internet connection), power up. After boot up, login (username=root, password=voyage). Remount as "read/write" mode:
%remountrw

2. Set correct date/time (optional, do it if necessary)
For example, set new data to 2 Oct 2006 18:00:00, type the following command:
# date -s "2 OCT 2006 18:00:00"
3. Update application package database:
%apt-get update
(this would take a minute or two, depends on your Internet connection)

%apt-get upgrade
(this would take another 2 to 10 minutes, depends on your Internet connection)

4. Install the required packages:
%apt-get install wget build-essential libncurses5-dev openssl bc
 (this would take about 10 minutes or more, be patient)

5. Download a kernel version with RT project. Check here for the available real-time kernel patch versions first.
%cd (go to root's home directory)
https://www.kernel.org/pub/linux/kernel/projects/rt/

Then check the link for the corresponding kernel version in here:
https://www.kernel.org/pub/linux/kernel/v3.x/

In my case, I selected kernel version 3.10.27-rt25. Highlight and copy the URL for the kernel patch and paste the URL with wget command to download:
%wget https://www.kernel.org/pub/linux/kernel/projects/rt/3.10/patch-3.10.27-rt25.patch.xz
Extract it:
%unxz patch-3.10.27-rt25.patch.xz

And download the kernel source:
%wget --no-check-certificate https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.10.27.tar.gz
Extract it:
%tar xvzf linux-3.10.27.tar.gz
(a folder called "linux-3.10.27" under "/root" will be created)



6. Applies the real time kernel patch:
%cp patch-3.10.27-rt25.patch linux-3.10.27
%cd linux-3.10.27
%patch -p1 < patch-3.10.27-rt25.patch
(it would patch the kernel source in few seconds...)

7. Optional, but recommended, edit the "Makefile"
%nano Makefile
See the line starting with "EXTRAVERSION" (it should be the forth line), go to the end, add some text, say "-atom" (without quotes), the "extra version" just appended after the minor version.

Remember not to add a space "after" the text you added. You can have white space between the "=" and the first character of the text

Also optional, but recommended, copy the kernel configuration file that comes with Voyage MPD:
%cd /boot
%ls config*
You shall see a file named "config-3.x.x-voyage", just copy this file to the linux kernel source folder that you just made above and rename as ".config" (note that there is a dot before "config")
%cp config-3.10.11-voyage /root/linux-3.10.27/.config


8. Start kernel configuration menu:
%make menuconfig

You shall see a text mode menu with blue background. You (must) do the following changes in order to enjoy the real-time kernel:

Kernel Features -> Preemption Model
Change to "Fully Preemptible Kernel (RT)"

General setup -> RCU Subsystem, enables:
-Enable RCU priority boosting
-Offload RCU callback processing from boot-selected CPUs

Optional: As mentioned above, remove all the drivers and features that are not need. This could make the kernel and driver size smaller and minimize the change of CPU get bothered.

(Tips #1: Check the inline help page for explaination of the feature and the suggestion of keep or not

Tips #2: I found a web tutorial written by DevynCJohnson which is very comprehensive and useful. You may get more idea about what to include to kernel build after reading it:
http://www.linux.org/threads/%EF%BB%BFthe-linux-kernel-introduction.4203/)

Update (Aug-4-2014): Please select 32-bit kernel if you are building for Voyage Linux. 64-bit kernel doesn't work on Voyage Linux.

Exit and save the configuration. (The configuration would be saved as ".config", a hidden file, under the root folder of the kernel source, i.e. under "/root/linux-3.10.27" in my case)

9. Start building kernel

%make -j2 all

this step will compile the kernel and create a compressed binary image of the kernel and also the loadable kernel modules. After the step, the kernel image can be found at arch/i386/boot/bzImage (for a x86 based processor) and kernel modules (i.e. those files ended with .ko) will be built.

Remember to use "-j" option if you are building on a multiple core CPU. "-j2" means running the build in two threads. You may use "-j4" if you have 4 core/virtual threads in total.

(this would take at 10 minutes up to 2 hours, depends on what you have selected to build and the CPU power)

10. Install to Voyage MPD disk
%make -j2 modules_install
The above command will make those kernel modules being copied to "/lib/modules/3.10.27-rt25")
%make -j2 install
The above command will build the SystemMap, vmlinuz, initrd and config files that appear in "/boot". It will also update the GRUB (the boot loader) to include the newly built kernel as option in the boot menu.


11. You may want to touch up the boot menu to make it more readable and remove unuse menu items. Just:
%nano /boot/grub/menu.lst

Important: The default kernel menu links the kernel images in the root folder. (/vmlinuz and /initrd.img), this two links were being redirected to the newly built kernel images after executing the "make install" command. So you should update the menu.lst file to make the link points to the ACTUAL files instead of the symbolic links.

12. Now you can reboot and you shall see the newly built kernel version as one of the boot options. Select it and see what happen. It may not boot successfully, especially if you didn't select most of the items. If it just can't boot up or you want to further fine tune, do the following:

13. Reboot with the stock kernel version selected in boot menu, it shall boot up normally.

14. Login as root, remount as read/write:
%remountrw

15. Goto /root/linux-{version}, if your recent kernel built works fine, I suggest making a backup of the ".config" in case the new kernel built doesn't work.
%cp .config config.bak
Run make menuconfig again.

16. Based on the error message (you have to "guess" what blocks the machine bootup!), enable it this time.

Or, if it could boot up but some hardware is not working, just enable that particular hardware. You can choose to build that device driver into kernel core or build as module. I think it doesn't have noticible performance hurt for building the device driver as kernel module but it certainly would be more flexible. (For me, I built the USB keyboard, USB wi-fi driver as kernel modules because I may not always need them.)

17. Save and exit the kernel configuration menu, delete the files created previously.
%rm /boot/*3.10.27*
This will remove those files with the kernel version as part of the file name.
%rm /vmlinuz /vmlinuz.old /initrd.img /initrd.img.old

repeat steps 9-12. Good luck!

Building custom (real-time) kernel for Cubieboard 2

Thanks benngcp in review33 for providing a detail guide for building real time kernel:
http://www.review33.com/avforum/forum_message.php?topic=70130726214918

Below is a recap of his guide, I use two SD cards in this guide (first card with 8G or more space to store the Cubian and the kernel source, the other card with 4G or more to store the Mubox image and the compiled kernel files)

1. Follow the guide in https://github.com/cubieplayer/cubian/wiki/Install-Cubian, download Cubian (Cubian-base-r4-arm-a20.img.7z) in here: http://cubian.org/downloads/

2. Load to a micro SD card using Win32 DiskImager:
http://sourceforge.net/projects/win32diskimager/

3. Insert the micro SD card, LAN cable and power up.

4. Login, the default username and password are both "cubie" (no double quotes)


5. After login, you can reset the root password:
#sudo passwd root
(enter new password twice)

6. Then su to root and set correct time & timezone:
#su -
(enter the new password that you just assigned in step 5)

#dpkg-reconfigure tzdata


select the correct region and country, it will show you the local and UTC time.
If the date is incorrect, set it like:
date -s "7 FEB 2014 18:00:00"


7. Refresh package database:
#apt-get update --fix-missing

Note that:

• It would take about 5-10 minutes, depends on your network bandwidth. 
• Press "enter" key for all the questions.

Below step is optional:
#apt-get upgrade

8. Get required softwares
#apt-get install libnl1 libnl-dev libssl-dev wpasupplicant bridge-utils curl wget dkms build-essential unzip ncurses-dev uboot-mkimage libusb-1.0 libusb-dev pkg-config

• It would take about 5-10 minutes, depends on your network bandwidth. 
• Press "enter" key for all the questions.
• Additional 150MB disk spaced is required.

9. Get the sunxi kernel source
#cd /root
#git clone git://github.com/cubieboard/linux-sunxi -b cubie/sunxi-3.4

• It would take an hour or longer, depends on network condition. 
• Additional 2GB disk spaced is required. (1GB download size, 1.7GB after decompression)

10. Modify kernel source (avoid compile error related to power management)
#cd /root/linux-sunxi
#nano arch/arm/mach-sun7i/pm/standby/mem_printk.c
(Use Ctrl+W to search by text, use Ctrl+C to show current row number)

10.1 goto line 168:
change: char digit_string[] = "0123456789ABCDEF";
to: char* digit_string = "0123456789ABCDEF";(add "*" and remove [])

Before:

 
After:

10.2 goto line 213:
change: char digit_string[] = "0123456789ABCDEF";
to: char* digit_string = "0123456789ABCDEF";(add "*" and remove [])

10.3 goto line 369:
change: char fill_ch[] = "       ";
to: char* fill_ch = "       ";
(add "*" and remove [])

Save and exit (Ctrl+X, enter Y and press enter key again)

 11. Download Realtime Patch 3.4.61
#wget https://www.kernel.org/pub/linux/kernel/projects/rt/3.4/older/patch-3.4.61-rt77.patch.xz
#unxz patch*

12. Apply the patch:
#patch -p1 < patch*
It shall take less than 10 seconds to complete.


13. Modify "fs/timerfd.c"
#nano fs/timerfd.c
goto line 388, change

hrtimer_wait_for_timer(&ctx->tmr);

to:

hrtimer_wait_for_timer(&ctx->t.tmr);

14. Get the cubieboard specific kernel configuration default values and open the kernel configuration menu page:
#git clone git://github.com/cubieboard/cubie_configs
#cp cubie_configs/kernel-configs/3.4/cubieboard2_defconfig .config
#make menuconfig

15. Do the following changes inside the menu:
15.1 Kernel Features -> Memory Split -> (change to) 1G/3G user/kernel split

15.2 Kernel Features -> Preemption Model -> (change to) Fully Preemptible Kernel (RT)

15.3 CPU power management -＞ CPU frequency scaling -＞ Default CPUFreq governor
(change to "performance")

(Exit and save the configuration)



16. Make the kernel
#alias make='make -j2'
#make
("-j2" means running the process in two threads, this could better utilize the CPU if your computer has multiple CPU/cores. Cubieboard2 is dual core, so using j2 will fully utilize the two cores)

The make process would take over an hour if there is no error, be patient...

Make sure the make process complete without error. If there is error, try to look at the cause of error, if it is something look familar that you've modified before, may be there was typo. If there is something new, try copy and paste the cause (file name, line number, error description) and search. Someone may have faced the same problem before.



(optional) Do a disk cache write back to avoid data loss due to accident:
#sync

17. Make the boot image#make ARCH=arm -j2 uImage

The make process would take about 30 minutes if there is no error, be patient

(optional) Do a disk cache write back to avoid data loss due to accident:
#sync

18. Now we can work on the Mubox. Take another micro SD card (4G or larger), put it into a card reader and connect to Cubieboard2. It should be mounted as /dev/sdb1 if there is a partition.

19. Setup Mubox on this SD card (in card reader), just follow the official setup guide in here:
http://mubox.voyage.hk/cubieboard2
The SD card should be in /dev/sda (before partitioning) and /dev/sda1 (after partitioning and formating). So just replace those "/dev/sdX" with "/dev/sda" and "/dev/sdX1" with "/dev/sda1".

20. Now you should have this SD card in card reader loaded with Mubox, mount it in /media/voyage-mubox. Just unplug the card reader and re-insert it. The system shall auto mount it into /media/voyage-mubox. If not, run the following command manually
#mkdir /media/voyage-mubox
#mount /dev/sdb1 /media/voyage-mubox

21.  Copy uImage & kernel modules

(optional, backup before copying)
#cp /media/voyage-mubox/boot/uImage /media/voyage-mubox/boot/uImage.old
#cd /root/linux-sunxi
#cp arch/arm/boot/uImage /media/voyage-mubox/boot
#make -j2 modules_install INSTALL_MOD_PATH=/media/voyage-mubox

24. Generate the boot up script  #cd /root
#git clone git://github.com/cubieboard/sunxi-tools
#cd /root/sunxi-tools
#make
#cd
#cp /root/linux-sunxi/cubie_configs/sysconfig/linux/cubieboard2.fex .
#nano cubieboard2.fex

(modify the usb detect type from 1 to 0 to solve the CPU overload problem, search "usb_detect_type=1")
[usbc0]
usb_detect_type=0

Save and exit


#./fex2bin cubieboard2.fex /media/voyage-mubox/boot/script.bin 

25. Done, power off and put the SD card in card reader to the SD card slot of Cubieboard2:
#sync
#poweroff
Then power on again, you shall see Mubox login, enter username=root, password=voyage to login. You may want to set the correct timezone again (on this SD card):
#dpkg-reconfigure tzdata
select the correct region and country, it will show you the local and UTC time.
If the date is incorrect, set it like:
date -s "7 FEB 2014 18:00:00"







If you want to install the Mubox onto NAND drive, do the below steps.
101. Boot up Mubox SD-card and login (username: root, password: voyage)
 (now run the below using the "root" account)

102. Add Cubian APT source:
#wget -O cubian.key http://packages.cubian.org/cubian.gpg.key
#apt-key add cubian.key

Edit /etc/apt/sources.list
#nano /etc/apt/sources.list
and add this line (anywhere)

deb http://packages.cubian.org/ wheezy main

Save and exit

103. Install required tools:
#apt-get install cubian-nandinstall dosfstools cubian-resizefs

104. Fix the "cubian-nandinstall" script:
#nano /usr/sbin/cubian-nandinstall
Find the line

echoRed "!!! This tool must be run on SD-card system"

Remove the "exit 2" line below that line.

Save and exit.


105. Run the copy script, it will erase your NAND drive and copy the content on SD card (mubox) to the NAND drive.
#cubian-nandinstall

poweroff, remove the SD card and power on again. It should boot from the NAND now.