Linux Device Driver Development Cookbook

Installing the Development System

In this chapter, we will present and set up our working platform. In fact, even if we write and then test our own device drivers on our working PC, it is recommended to use a second device to test the code. This is because we're going to work in the kernel space where even a little bug can cause severe malfunctioning! Also, using a platform where several kinds of peripherals are available allows us to test a large variety of devices that are not always available on a PC. Of course, you are free to use your own system to write and test your drivers but, in this case, you should take care of the modifications needed to fit your board specifications.

In this book, I'm going to use the Marvell ESPRESSObin system, which is a powerful Advanced RISC Machines (ARM) 64-bit machine with a lot of interesting features. In the following figure, you can see the ESPRESSObin alongside a credit card and can gain an idea about the real dimensions of the board:

My board is the v5 release of ESPRESSObin while the latest version at the time of writing (announced on September 2018) is v7, so the reader should be able to get this new release by the time this book is published. The new ESPRESSObin v7 will feature 1GB DDR4 and 2GB DDR4 configurations (while v5 has DDR3 RAM chips), and a new 1.2GHz chipset will replace the currently sold configurations, which sports 800MHz and 1GHz CPU frequency limits. Even by taking a quick look at the new board layout, we see that a single SATA connector has taken the place of the existing two-pieces combination of SATA power and interface, the LED layout is now rearranged in a row, and an on-board eMMC is now in place. Moreover, this new revision will ship with an optional 802.11ac + Bluetooth 4.2 mini PCIe Wi-Fi card, which is sold separately.

Lastly, you will now have the option to order your v7 ESPRESSObin with a complete enclosure. This product has FCC and CE certifications to help to enable mass deployment. Further information regarding the revision v7 (and v5) can be found at http://wiki.espressobin.net/tiki-index.php?page=Quick+User+Guide.

In order to test our new drivers, we will cover the following recipes in this first chapter:

Setting up the host machine
Working with the serial console
Configuring and building the kernel
Setting up the target machine
Doing native compiling on foreign hardware

Technical requirements

Following are some interesting URLs where we can get useful technical information regarding the board:

The home page: http://espressobin.net/
The documentation wiki: http://wiki.espressobin.net/tiki-index.php
Forums: http://espressobin.net/forums/

Taking a look at the technical specifications at http://espressobin.net/tech-spec/, we get the following information where we can see what the ESPRESSObin v5 can offer in terms of computational power, storage, networking, and expandability:

System on chip (SoC)	Marvell Armada 3700LP (88F3720) dual core ARM Cortex A53 processor up to 1.2GHz
System memory	1 GB DDR3 or optional 2GB DDR3
Storage	1x SATA interface 1x micro SD card slot with footprint for an optional 4GB EMMC
Network connectivity	1x Topaz Networking Switch 2x GbE Ethernet LAN 1x Ethernet WAN 1x MiniPCIe slot for wireless/BLE peripherals
USB	1x USB 3.0 1x USB 2.0 1x micro USB port
Expansion	2x 46-pin GPIO headers for accessories and shields with I2C, GPIOs, PWM, UART, SPI, MMC, and so on.
Misc	Reset button and JTAG interface
Power supply	12V DC jack or 5V via micro USB port
Power consumption	Less than 1W thermal dissipation at 1 GHz

In particular, the following screenshot shows the top view of the Marvell ESPRESSObin v5 (from now on, please take into account that I'm not going to explicitly add "v5" anymore):

In the preceding screenshot, we can see the following components:

The power connector (12V DC jack)
The reset switch
The micro USB device port (serial console)
The Ethernet ports
The USB host ports

The next screenshot shows the bottom view of the board where the microSD slot is located; this is where we should plug in the microSD we're going to create later on in this chapter:

In this book, we'll see how we can manage (and reinstall) a complete Debian distribution, something that will allow us to have a wide set of ready-to-run software packages, as in a normal PC (in fact, the Debian ARM64 version is equivalent to the Debian x86 version). Afterward, we will develop our device drivers for the board, and then, when possible, we will test them with real devices connected to the ESPRESSObin itself. A little tutorial about how to set up the host system is also present in this chapter, and you can use it to set up a GNU/Linux-based working machine or a dedicated virtual one.

The code and other files used in this chapter can be downloaded from GitHub at https://github.com/giometti/linux_device_driver_development_cookbook/tree/master/chapter_01.

Setting up the host machine

As every good device driver developer knows, a host machine is absolutely necessary.
Even if the embedded devices are getting more powerful nowadays (and the ESPRESSObin
is one of these), there are some resource-consuming tasks where a host machine can help.
That's why, in this section, we're going to show how to set up our host machine.

The host machine we decide to use could be a normal PC or a virtualized one—they are equivalent—but the important thing is that it must run a GNU/Linux-based OS.

Getting ready

In this book, I will use an Ubuntu 18.04 LTS based system but you can decide to try to replicate some settings and installation commands that we will use during the course of this book into another major Linux distribution, with little effort for a Debian derivative, or in a bit more of a complicated manner in the case of non-Debian derivative distributions.

I'm not going to show how to install a fresh Ubuntu system on a PC nor on a virtualized machine since it's a really easy task for a real programmer; however, as the last step of this chapter (the Doing native compiling on foreign hardware recipe), I will introduce, with detailed steps about how to install it, an interesting cross-platform environment that proved useful to compile foreign target code on the host machine as we were on the target. This procedure is very useful when we need several different OSes running on your development PC.

So, at this point, the reader should have their own PC running (natively or virtualized) a fresh installed Ubuntu 18.04 LTS OS.

The main usage of a host PC is to edit and then cross-compile our new device drivers and to manage our target device via the serial console, to create its root filesystem, and so on.

In order to do it properly, we need some basic tools; some of them are general while others depend on the specific platform onto which we are going to write our drivers.

General tools are surely an editor, a version control system, and a compiler and its related components, while specific platform tools are essentially the cross-compiler and its related components (on some platforms we may need additional tools but our mileage may vary and, in any case, each manufacturer will give us all of the needed requirements for a comfortable compilation environment).

About the editor: I'm not going to spend any words on it because the reader can use whatever they want (regarding myself, for example, I'm still programming with vi editor) but regarding others tools, I'll have to be more specific.

How to do it...

Now that our GNU/Linux distribution is up and running on our host PC we can start to install some programs we're going to use in this book:

First of all, let's install the basic compiling tools:

$ sudo apt install gcc make pkg-config \
           bison flex ncurses-dev libssl-dev \
           qemu-user-static debootstrap

As you know already, the sudo command is used to execute a command as a privileged user. It should be already present in your system, otherwise you can install it by using the apt install sudo command as the root user.

Next, we have to test the compiling tools. We should be able to compile a C program. As a simple test, let's use the following standard Hello World code stored in the helloworld.c file:

#include <stdio.h>

int main(int argc, char *argv[])
{
    printf("Hello World!\n");

    return 0;
}

Remember that code can be downloaded from our GitHub repository.

Now, we should be able to compile it by using the following command:

$ make CFLAGS="-Wall -O2" helloworld
cc -Wall -O2 helloworld.c -o helloworld

In the preceding command, we used both the compiler and the make tool, which is required to compile every Linux driver in a comfortable and reliable manner.

You can get more information regarding make by taking a look at https://www.gnu.org/software/make/, and for gcc, you can go to https://www.gnu.org/software/gcc/.

Finally, we can test it on the host PC, as follows:

$ ./helloworld 
Hello World!

The next step is to install the cross-compiler. Since we're going to work with an ARM64 system, we need a cross-compiler and its related tools. To install them, we simply use the following command:

$ sudo apt install gcc-7-aarch64-linux-gnu

Note that we can also use an external toolchain as reported in the ESPRESSObin wiki at http://wiki.espressobin.net/tiki-index.php?page=Build+From+Source+-+Toolchain; however, the Ubuntu toolchain works perfectly!

When the installation is complete, test our new cross-compiler by using the preceding Hello World program, as follows:

$ sudo ln -s /usr/bin/aarch64-linux-gnu-gcc-7 /usr/bin/aarch64-linux-gnu-gcc
$ make CC=aarch64-linux-gnu-gcc CFLAGS="-Wall -O2" helloworld
aarch64-linux-gnu-gcc-7 -Wall -O2 helloworld.c -o helloworld

Note that I've removed the previously compiled helloworld program in order to be able to correctly compile this new version. To do so, I used the mv helloworld helloworld.x86_64 command due to the fact I'll need the x86 version again.

Also, note that since Ubuntu doesn't automatically create the standard cross-compiler name, aarch64-linux-gnu-gcc, we have to do it manually by using the preceding ln command before executing make.

OK, now we can verify that newly created version of the helloworld program for ARM64 by using the following file command. This will point out which platform the program is compiled for:

$ file helloworld
helloworld: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=c0d6e9ab89057e8f9101f51ad517a253e5fc4f10, not stripped

If we again use the file command on the previously renamed version, helloworld.x86_64, we get the following:

$ file helloworld.x86_64 
helloworld.x86_64: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=cf932fab45d36f89c30889df98ed382f6f648203, not stripped

To test whether this new release is really for the ARM64 platform, we can use QEMU, which is an open source and generic machine emulator and virtualizer that is able to execute foreign code on the running platform. To install it, we can use apt command as in the preceding code, specifying the qemu-user-static package:

$ sudo apt install qemu-user-static

Then, we can execute our ARM64 program:

$ qemu-aarch64-static -L /usr/aarch64-linux-gnu/ ./helloworld
Hello World!

To get further information about QEMU, a good staring point is its home page at https://www.qemu.org/.

The next step is to install the version control system. We must install the version control system used for the Linux project, that is, git. To install it, we can use the following command in a similar manner as before:

$ sudo apt install git

If everything works well, we should be able to execute it as follows:

$ git --help
usage: git [--version] [--help] [-C <path>] [-c <name>=<value>]
           [--exec-path[=<path>]] [--html-path] [--man-path]
           [--info-path] [-p | --paginate | --no-pager]
           [--no-replace-objects] [--bare] [--git-dir=<path>]
           [--work-tree=<path>] [--namespace=<name>]
           <command> [<args>]

These are common Git commands used in various situations:

start a working area (see also: git help tutorial)
   clone Clone a repository into a new directory
   init Create an empty Git repository or reinitialise an existing one
...

In this book, I'm going to explain every git command used but for complete knowledge of this powerful tool, I suggest you start reading https://git-scm.com/.

Working with the serial console

As already stated (and as any real programmer of embedded devices knows), the serial console is a must-have during the device drivers development stages! So, let's see how we can get access to our ESPRESSObin through its serial console.

Getting ready

As shown in the screenshot in the Technical requirements section, a micro USB connector is available and it's directly connected with ESPRESSObin's serial console. So, using a proper USB cable, we can connect it to our host PC.

If all connections are OK, we can execute any serial Terminal emulator to see data from the serial console. Regarding this tool, I have to state that, as editor program, we can use whatever we prefer. However, I'm going to show how to install two of the more used Terminal emulation programs—minicom and screen.

Note that this tool is not strictly required and its usage depends on the platform you're going to work on; however, in my humble opinion, this is the most powerful development and debugging tool ever! So, you definitely need it.

To install minicom, use the following command:

$ sudo apt install minicom

Now, to install the Terminal emulator named screen, we just have to replace minicom string with the screen packet name, as shown in the following:

$ sudo apt install screen

Both of them need a serial port to work on and the invocation command is quite similar. For brevity, I'm going to report their usage to get connected with the ESPRESSObin only; however, for further information about them, you should refer to their man pages (use man minicom and man screen to show them).

How to do it...

To test the serial connection with our target system we can do the following steps:

First of all, we have to locate the right serial port. Since the ESPRESSObin uses an USB emulated serial port (at 115,200 baud rate), usually our target port is named ttyUSB0 (but your mileage may vary, so let's verify it before continuing) so the minicom command we have to use to get connected with the ESPRESSObin serial console is the following:

$ minicom -o -D /dev/ttyUSB0

To correctly get access to the serial console, we may need proper privileges. In fact, we may try to execute the preceding minicom command, and we don't get an output! This is because the minicom command silently exits if we don't have enough privileges to get access to the port. We can verify our access to privileges by simply using another command on it, as shown here:
$ cat /dev/ttyUSB0
cat: /dev/ttyUSB0: Permission denied
In this case, the cat command perfectly tells us what's wrong so we can fix this issue using sudo or, even better, by properly adding our system's user to the right group as shown here:
$ ls -l /dev/ttyUSB0 crw-rw---- 1 root dialout 188, 0 Jan 12 23:06 /dev /ttyUSB0
$ sudo adduser $LOGNAME dialout
Then, we log out and log in again, and we can access the serial devices without any problem.

The equivalent command by using screen is reported as follows:

$ screen /dev/ttyUSB0 115200

Note that, on minicom, I didn't specify the serial communication options (baud rate, parity, and so on) while, for screen, I've added the baud rate on the command line; this is because my default minicom configuration automatically uses correct communication options while screen uses 9,600 baud as a default baud rate. Please refer to the program man pages for further information about how to do this setting in order to fit your needs.

If everything works well, after executing your Terminal emulator on the right serial port, turn on our ESPRESSObin (simply by plugging in the power). We should see the following output on our Terminal:

NOTICE: Booting Trusted Firmware
NOTICE: BL1: v1.3(release):armada-17.06.2:a37c108
NOTICE: BL1: Built : 14:31:03, Jul 5 2NOTICE: BL2: v1.3(release):armada-17.06.2:a37c108
NOTICE: BL2: Built : 14:31:04, Jul 5 201NOTICE: BL31: v1.3(release):armada-17.06.2:a37c108
NOTICE: BL31:

U-Boot 2017.03-armada-17.06.3-ga33ecb8 (Jul 05 2017 - 14:30:47 +0800)

Model: Marvell Armada 3720 Community Board ESPRESSOBin
       CPU @ 1000 [MHz]
       L2 @ 800 [MHz]
       TClock @ 200 [MHz]
       DDR @ 800 [MHz]
DRAM: 2 GiB
U-Boot DComphy-0: USB3 5 Gbps 
Comphy-1: PEX0 2.5 Gbps 
Comphy-2: SATA0 6 Gbps 
SATA link 0 timeout.
AHCI 0001.0300 32 slots 1 ports 6 Gbps 0x1 impl SATA mode
flags: ncq led only pmp fbss pio slum part sxs 
PCIE-0: Link down
MMC: sdhci@d0000: 0
SF: Detected w25q32dw with page size 256 Bytes, erase size 4 KiB, total 4 MiB
Net: eth0: neta@30000 [PRIME]
Hit any key to stop autoboot: 2

Configuring and building the kernel

Now, it's time to download the kernel sources and then configure and build them. This step is needed for several reasons: the first one is that we need a kernel for our ESPRESSObin in order to boot an OS, and the second one is that we need a configured kernel sources tree to compile our drivers into.

Getting ready

Since our ESPRESSObin is now supported into vanilla kernel since the 4.11 release, we can get Linux sources by using the following git command:

$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

This command will take a lot of time to finish so I would suggest you take a break by drinking your favorite cup of coffee (as real programmers should do).

When finished, we can enter into the linux directory to see the Linux sources:

$ cd linux/
$ ls
arch CREDITS firmware ipc lib mm scripts usr
block crypto fs Kbuild LICENSES net security virt
certs Documentation include Kconfig MAINTAINERS README sound
COPYING drivers init kernel Makefile samples tools

These sources are related to the latest kernel release that could be unstable, so to be sure that we're using a stable kernel release (or a long-term release), let's extract release 4.18, which is the current stable release at time of writing this chapter, as follows:

$ git checkout -b v4.18 v4.18

How to do it...

Before starting the compilation, we have to configure the kernel and our compiling environment.

The last task is quite easy and it consists of executing the following environment variables assignments:

$ export ARCH=arm64
$ export CROSS_COMPILE=aarch64-linux-gnu-

Then, we can select the ESPRESSObin standard kernel configuration by simply using the following command:

$ make defconfig

Depending on the kernel release you're using, the default configuration file may be also called mvebu_defconfig or either mvebu_v5_defconfig or mvebu_v7_defconfig. So, please take a look into the linux/arch/arm64/configs/ directory in order to see which file is present that best suits your needs.
In my system, I have the following:
$ ls linux/arch/arm64/configs/
defconfig

If we wish to modify this default configuration, we can execute the make menuconfig command, which will show us a pretty menu where we can enter our modifications in order to fit our needs. The following screenshot shows how the kernel configuration menu appears on the Terminal:

Before continuing, we must be sure that the Distributed Switch Architecture (DSA) is enabled into the kernel, otherwise we're not able to use the Ethernet ports at all! This is because the ESPRESSObin has a complex (and really powerful) internal network switch that must be managed by using this special support.

For further information regarding the DSA, you can start reading the linux/Documentation/networking/dsa/dsa.txt file, located in the kernel sources we're currently working on.

To enable DSA support, just navigate into the kernel menu at Networking support. Go to Networking options and, at the end, enable the entry Distributed Switch Architecture support. After that, we have to go back to the menu's top level and then select these entries: Device Drivers | Network device support | Distributed Switch Architecture drivers and then enable Marvell 88E6xxx Ethernet switch fabric support, which is the ESPRESSObin's on-board switch chip.

Remember that, to enable a kernel feature as a module or a built-in, you need to highlight the desired feature and then press the spacebar until the character inside the <> characters changes to * (which means built-in, that is, <*>) or to M (which means module, that is, <M>).

Note that, to enable DSA as a built-in instead of as a module, we have to disable 802.1d Ethernet Bridging support (that is, the entry just above).

Well, after all kernel settings are in place, we can start the kernel compilation by using the following make command:

$ make Image dtbs modules

Again, as the downloading command, this command will need a lot of time to finish, so let me suggest you take another break. However, in order to speed up the compilation process, you may try using the -j option argument in order to tell make to use several simultaneous process to compile the code. For example, on my machine, having eight CPU threads, I use the following command:
$ make -j8 Image dtbs modules
So, let's try using the following lscpu command to get how many CPUs your system has:
lscpu | grep '^CPU(s):'
CPU(s): 8
Alternatively, on Ubuntu/Debian, there's also the pre-installed nproc utility, so the following command also does the trick:
$ make -j$(nproc)

When finished, we should have the kernel image into the arch/arm64/boot/Image file and the device tree binary into the arch/arm64/boot/dts/marvell/armada-3720-espressobin.dtb file, which are ready to be transferred into the microSD we're going to build in the next recipe, Setting up the target machine.

Setting up the target machine

Now, it's time to install whatever we needed on our target system; since the ESPRESSObin is sold with just the bootloader, we have to do some work in order to get a fully functional system with a proper OS.

In this book, I'm going to use a Debian OS for the ESPRESSObin but you may use other OSes as reported at http://wiki.espressobin.net/tiki-index.php?page=Software+HowTo. On this site, you can get more detailed information about how to properly set up your ESPRESSObin to fit your needs.

Getting ready

Even if the ESPRESSObin can boot from different media, we're going to use a microSD since it's the most easy and useful way to set up the system. For different media, please refer to the ESPRESSObin's wiki pages—see http://wiki.espressobin.net/tiki-index.php?page=Boot+from+removable+storage+-+Ubuntu for some examples.

How to do it...

To set up the microSD, we have to use our host PC, so plug it in and then locate the corresponding device.

If we're using an SD/microSD slot, as soon as we plug the media in, we'll get something like this in the kernel messages:

mmc0: cannot verify signal voltage switch
mmc0: new ultra high speed SDR50 SDHC card at address aaaa
mmcblk0: mmc0:aaaa SL08G 7.40 GiB 
 mmcblk0: p1

To get kernel messages on the Terminal, we can use the dmesg command.

However, if we're going to use a microSD to USB adapter kernel, messages will look like the following:

usb 1-6: new high-speed USB device number 5 using xhci_hcd
usb 1-6: New USB device found, idVendor=05e3, idProduct=0736
usb 1-6: New USB device strings: Mfr=3, Product=4, SerialNumber=2
usb 1-6: Product: USB Storage
usb 1-6: Manufacturer: Generic
usb 1-6: SerialNumber: 000000000272
usb-storage 1-6:1.0: USB Mass Storage device detected
scsi host4: usb-storage 1-6:1.0
usbcore: registered new interface driver usb-storage
usbcore: registered new interface driver uas
scsi 4:0:0:0: Direct-Access Generic STORAGE DEVICE 0272 PQ: 0 ANSI: 0
sd 4:0:0:0: Attached scsi generic sg3 type 0
sd 4:0:0:0: [sdc] 15523840 512-byte logical blocks: (7.95 GB/7.40 GiB)
sd 4:0:0:0: [sdc] Write Protect is off
sd 4:0:0:0: [sdc] Mode Sense: 0b 00 00 08
sd 4:0:0:0: [sdc] No Caching mode page found
sd 4:0:0:0: [sdc] Assuming drive cache: write through
 sdc: sdc1
sd 4:0:0:0: [sdc] Attached SCSI removable disk

Another easy way to locate the media is by using the lsblk command, as follows:

$ lsblk 
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 5M 1 loop /snap/gedit/66
loop1 7:1 0 4.9M 1 loop /snap/canonical-livepatch/50
...
sdb 8:16 0 931.5G 0 disk 
└─sdb1 8:17 0 931.5G 0 part /run/schroot/mount/ubuntu-xenial-amd64-f72c490
sr0 11:0 1 1024M 0 rom 
mmcblk0 179:0 0 7.4G 0 disk 
└─mmcblk0p1
        179:1 0 7.4G 0 part /media/giometti/5C60-6750

It's now obvious that our microSD card is here listed as /dev/mmcblk0 but it is not empty. Since we want to clear everything from it, we have to clear it first by using the following command:

$ sudo dd if=/dev/zero of=/dev/mmcblk0 bs=1M count=100

You may need to unmount the device before proceeding with the clearing in order to work safely on the media device, so let's unmount all of the device's partitions by using the umount command on all of them as I will do in the following with the only defined partition on my microSD:

$ sudo umount /dev/mmcblk0p1

You have to just repeat this command for each defined partition on your microSD.

Now, we will create a new partition, /dev/mmcblk0p1, on the empty SD card with the next command:

$ (echo n; echo p; echo 1; echo ''; echo ''; echo w) | sudo fdisk /dev/mmcblk0

If everything works well, our microSD media should appear formatted, as in the following:

$ sudo fdisk -l /dev/mmcblk0
Disk /dev/mmcblk0: 7.4 GiB, 7948206080 bytes, 15523840 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x34f32673

Device Boot Start End Sectors Size Id Type
/dev/mmcblk0p1 2048 15523839 15521792 7.4G 83 Linux

Then, we have to format it as EXT4 with the following command:

$ sudo mkfs.ext4 -O ^metadata_csum,^64bit -L root /dev/mmcblk0p1

Note that this command line works for the e2fsprogs version >=1.43 only! If you're using an older release, you should use the following command:
$ sudo mkfs.ext4 -L root /dev/mmcblk0p1

Next, mount this partition on your local Linux machine:

$ sudo mount /dev/mmcblk0p1 /mnt/

Note that, on some OSes (and especially on Ubuntu), as soon as we unplug and then we plug in the media device again, it is mounted automatically into /media/$USER/root where $USER is an environment variable holding your username. For instance, on my machine, I have the following:
$ ls -ld /media/$USER/root
drwxr-xr-x 3 root root 4096 Jan 10 14:28 /media/giometti/root/

Adding Debian files

I decided to use Debian as the target OS since it's my favorite distribution for development (and, when possible, for production) systems:

To install it, we use QEMU software again, using the following command:

$ sudo qemu-debootstrap \
       --arch=arm64 \
       --include="sudo,file,openssh-server" \
       --exclude="debfoster" \
       stretch ./debian-stretch-arm64 http://deb.debian.org/debian

You could see warnings about keyring here; they are harmless and they can be safely ignored:
W: Cannot check Release signature;

I suppose this is another coffee-break command.

Once finished, we should find, in debian-stretch-arm64, a clean Debian root filesystem for the ESPRESSObin but, before transferring it into the microSD, we should fix the hostname file contents as shown here:

$ sudo bash -c 'echo espressobin | cat > ./debian-stretch-arm64/etc/hostname'

Then, we have to add the serial device ttyMV0 to the /etc/securetty file in order to be able to log in as the root user through the serial device, /dev/ttyMV0. Use the following command:

$ sudo bash -c 'echo -e "\n# Marvell serial ports\nttyMV0" | \
                cat >> ./debian-stretch-arm64/etc/securetty'

Use man securetty for further information about the root login through a serial connection.

And, as a last step, we have to set up a root password:

$ sudo chroot debian-stretch-arm64/ passwd
Enter new UNIX password: 
Retype new UNIX password: 
passwd: password updated successfully

Here, I used the root string as password for the root user (it is up to you to choose yours).

In order to have further information regarding this usage of the chroot command, you can use the man chroot command or continue reading till the end of this chapter where I'm going to explain a bit better how it works.

Now, we can safely copy all files into our microSD using the following command:

$ sudo cp -a debian-stretch-arm64/* /media/$USER/root/

Here is what the microSD content should look like:

$ ls /media/$USER/root/
bin   dev  home  lost+found  mnt  proc  run   srv  tmp  var
boot  etc  lib   media       opt  root  sbin  sys  usr

Adding the kernel

After OS files, we need also kernel images to get a running kernel and, in the previous section, we got the kernel image into the arch/arm64/boot/Image file and the device tree binary into the arch/arm64/boot/dts/marvell/armada-3720-espressobin.dtb file, which are ready to be transferred into our freshly created microSD:

Let's copy them into the /boot directory as done here:

$ sudo cp arch/arm64/boot/Image \
       arch/arm64/boot/dts/marvell/armada-3720-espressobin.dtb \
           /media/$USER/root/boot/

If the /boot directory was not present in the microSD and the preceding command returned an error, you can recover by using the following command and rerun the preceding cp command:
$ sudo mkdir /media/$USER/root/boot

Then, the /boot directory should look like this:

$ ls /media/$USER/root/boot/
armada-3720-espressobin.dtb  Image

The preceding files are sufficient to boot the system; however, to also install kernel modules and headers files, which are useful for compiling new software, we can use the next commands after all Debian files have been installed into the microSD (to avoid overwriting with Debian files):

$ sudo -E make modules_install INSTALL_MOD_PATH=/media/$USER/root/
$ sudo -E make headers_install INSTALL_HDR_PATH=/media/$USER/root/usr/

Well, now we are finally ready to tie it all up and run our new Debian system, so let's unmount the microSD and plug it into the ESPRESSObin.

Setting up the booting variables

After powering up, we should get the bootloader's messages from the serial console and then we should see a timeout running to 0 before doing the autoboot:

Quickly stop the countdown by hitting the Enter key on the keyboard to get the bootloader's prompt, as follows:

Model: Marvell Armada 3720 Community Board ESPRESSOBin
       CPU @ 1000 [MHz]
       L2 @ 800 [MHz]
       TClock @ 200 [MHz]
       DDR @ 800 [MHz]
DRAM: 2 GiB
U-Boot DComphy-0: USB3 5 Gbps 
Comphy-1: PEX0 2.5 Gbps 
Comphy-2: SATA0 6 Gbps 
SATA link 0 timeout.
AHCI 0001.0300 32 slots 1 ports 6 Gbps 0x1 impl SATA mode
flags: ncq led only pmp fbss pio slum part sxs 
PCIE-0: Link down
MMC: sdhci@d0000: 0
SF: Detected w25q32dw with page size 256 Bytes, erase size 4 KiB, total 4 MiB
Net: eth0: neta@30000 [PRIME]
Hit any key to stop autoboot: 0 
Marvell>>

The ESPRESSObin's bootloader is U-Boot, which has its home page at https://www.denx.de/wiki/U-Boot.

Now, let's check again that the microSD card has the necessary files using the ext4ls command, as follows:

Marvell>> ext4ls mmc 0:1 boot
<DIR> 4096 .
<DIR> 4096 ..
        18489856 Image
            8359 armada-3720-espressobin.dtb

OK, everything is in place, so there are only a few variables required to boot from the microSD card.

We can display the currently defined variables at any point by using the echo command and optionally reconfigure them by using setenv command. First, check and set proper image and device tree paths and names:

Marvell>> echo $image_name
Image
Marvell>> setenv image_name boot/Image
Marvell>> echo $fdt_name
armada-3720-espressobin.dtb
Marvell>> setenv fdt_name boot/armada-3720-espressobin.dtb

Note that, filenames were correct but the path names were not; that's why I used the setenv command to correctly redefine them.

Next, define the bootcmd variable, which we will use to boot from the microSD card:

Marvell>> setenv bootcmd 'mmc dev 0; \
              ext4load mmc 0:1 $kernel_addr $image_name; \
              ext4load mmc 0:1 $fdt_addr $fdt_name; \
              setenv bootargs $console root=/dev/mmcblk0p1 rw rootwait; \
              booti $kernel_addr - $fdt_addr'

We must be careful to set the preceding root path to point to where we have extracted the Debian filesystem (the first partition in our case).

Save the set variables at any time using the saveenv command.
Finally, we boot up the ESPRESSObin by simply typing the reset command and, if everything works well, we should see the system start and running and, at the end, we should get system login prompt, as follows:

Debian GNU/Linux 9 espressobin ttyMV0

giometti-VirtualBox login:

Now, log in as root with the root password that was previously set up:

Debian GNU/Linux 9 espressobin ttyMV0

espressobin login: root
Password: 
Linux espressobin 4.18.0 #2 SMP PREEMPT Sun Jan 13 13:05:03 CET 2019 aarch64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@espressobin:~#

Setting up the networking

OK, now our ESPRESSObin is ready to execute our code and our drivers! However, before ending this section, let's take a look at the networking configuration since it can be further useful to log in to the board using an SSH connection or just to copy files from/to the board quickly (even if we can remove the microSD and then copy our files from the host PC directly):

Taking a look at available network interfaces on the ESPRESSObin, we see the following:

# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT
 group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group
 default qlen 532
    link/ether 3a:ac:9b:44:90:e9 brd ff:ff:ff:ff:ff:ff
3: wan@eth0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN mode DE
FAULT group default qlen 1000
    link/ether 3a:ac:9b:44:90:e9 brd ff:ff:ff:ff:ff:ff
4: lan0@eth0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN mode D
EFAULT group default qlen 1000
    link/ether 3a:ac:9b:44:90:e9 brd ff:ff:ff:ff:ff:ff
5: lan1@eth0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN mode D
EFAULT group default qlen 1000
    link/ether 3a:ac:9b:44:90:e9 brd ff:ff:ff:ff:ff:ff

The eth0 interface is the one that connects the CPU with the Ethernet switch while the wan, lan0, and lan1 interfaces are the ones where we can physically connect our Ethernet cables (note that the system calls them wan@eth0, lan0@eth0, and lan1@eth1 just to underline the fact they are slaves of eth0). Following is a photograph of the ESPRESSObin, where we can see each Ethernet port with its label:

Despite their names, all ports are equivalent so connect the Ethernet cable into one port (I'm going to use wan) and then enable it after eth0, as follows:

# ip link set eth0 up
mvneta d0030000.ethernet eth0: configuring for fixed/rgmii-id link mode
mvneta d0030000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
# ip link set wan up 
mv88e6085 d0032004.mdio-mii:01 wan: configuring for phy/ link mode
mv88e6085 d0032004.mdio-mii:01 wan: Link is Up - 100Mbps/Full - flow control rx/tx

Note that, in the preceding output, there are also kernel messages that show what you should see if everything is working well.

Now, we can manually set an IP address or we can ask our DHCP server whatever we need to surf the internet with the dhclient command:

# dhclient wan

Here is my network configuration:

# ip addr show wan
3: wan@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP g
roup default qlen 1000
    link/ether 9e:9f:6b:5c:cf:fc brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.100/24 brd 192.168.0.255 scope global wan
       valid_lft forever preferred_lft forever

Now, we're ready to install new software or to try to establish an SSH connection to the ESPRESSObin; to do so, let's verify that we have the following SSH server's configuration in the/etc/ssh/sshd_config file:

# grep 'PermitRootLogin yes' /etc/ssh/sshd_config
PermitRootLogin yes

If we get no output, we cannot log in as root into our ESPRESSObin, so we must change the PermitRootLogin setting to yes and then restart the daemon:

# /etc/init.d/ssh restart

Restarting ssh (via systemctl): ssh.service.

Now, on the host PC, we can try the login via SSH, as follows:

$ ssh [email protected]
[email protected]'s password: 
Linux espressobin 4.18.0 #2 SMP PREEMPT Sun Jan 13 13:05:03 CET 2019 aarch64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Thu Nov 3 17:16:59 2016
-bash: warning: setlocale: LC_ALL: cannot change locale (en_GB.UTF-8)

Doing native compiling on foreign hardware

Before ending this chapter, I'd like to introduce an interesting cross-platform system that's useful when we need several different OSes running on your development PC. This step is very useful when we need a complete OS to compile a device driver or an application but we do not have a target device to compile onto. We can use our host PC to compile code for a foreign hardware across different OS and OS release.

Getting ready

During my career, I worked with tons of different platforms and having one virtual machine for all of them is very complex and really system consuming (especially if we decide to run several of them at the same time!). That's why it can be interesting to have a lightweight system that can execute foreign code on your PC. Of course, this method cannot be used to test a device driver (we need real hardware for that), but we can use it to run a native compiler and/or native userspace code really quickly just in case our embedded platform is not working. Let's see what I'm talking about.

In the Setting up the target machine recipe, regarding the Debian OS installation, we used the chroot command to set up the root's password. This command worked thanks to QEMU; in fact, in the debian-stretch-arm64 directory, we have an ARM64 root filesystem, which can be executed on an x86_64 platform by using QEMU only. It's then clear that, in this manner, we should be able to execute whatever command we'd like and, of course, we will be able to execute the Bash shell as in the next recipe.

How to do it...

Now it's time to see how chroot works:

Execute an ARM64 bash command by using our x86_64 host, as follows:

$ sudo chroot debian-stretch-arm64/ bash
bash: warning: setlocale: LC_ALL: cannot change locale (en_GB.UTF-8)
root@giometti-VirtualBox:/#

Then, we can use each ARM64 command as we did on the ESPRESSObin; for example, to list files into the current directory; we can use the following:

# ls /
bin  dev  home media  opt   root  sbin  sys  usr
boot etc  lib  mnt    proc  run   srv   tmp  var
# cat /etc/hostname 
espressobin

However, there are some traps; for instance, we completely miss the /proc and /sys directories and programs, which rely on them and will fail for sure:

# ls /{proc,sys}
/proc:

/sys:
# ps
Error: /proc must be mounted
  To mount /proc at boot you need an /etc/fstab line like:
      proc /proc proc defaults
  In the meantime, run "mount proc /proc -t proc"

To resolve these problems, we can manually mount these missing directories before executing chroot, but this is quite annoying due to the fact that they are so many, so we can try using the schroot utility, which, in turn, can do all of these steps for us. Let's see how.

For detailed information regarding schroot, you can see its man pages with man schroot.

Installing and configuring schroot

This task is quite trivial in Ubuntu:

First of all, we install the program in the usual way:

$ sudo apt install schroot

Then, we have to configure it in order to correctly enter into our ARM64 system. To do so, let's copy the root filesystem created before into a dedicated directory (where we can also add any other distributions we wish to emulate with schroot):

$ sudo mkdir /srv/chroot/
$ sudo cp -a debian-stretch-arm64/ /srv/chroot/

Then, we must create a proper configuration for our new system by adding a new file into the schroot configuration directory, as follows:

$ sudo bash -c 'cat > /etc/schroot/chroot.d/debian-stretch-arm64 <<__EOF__
[debian-stretch-arm64]
description=Debian Stretch (arm64)
directory=/srv/chroot/debian-stretch-arm64
users=giometti
#groups=sbuild
#root-groups=root
#aliases=unstable,default
type=directory
profile=desktop
personality=linux
preserve-environment=true
__EOF__'

Note that the directory parameter is set to the path holding our ARM64 system and users is set to giometti, which is my username (this is a comma-separated list of users that are allowed access to the chroot environment—see man schroot.conf).

Looking at the preceding settings, we see that the profile parameter is set to desktop; this means that it will be taking into account all files in the /etc/schroot/desktop/ directory. In particular, the fstab file holds all mount points we'd like to be mounted into our system. So, we should verify that it holds at least the following lines:

# <filesystem> <mount point> <type> <options> <dump> <pass>
/proc           /proc         none   rw,bind   0      0
/sys            /sys          none   rw,bind   0      0
/dev            /dev          none   rw,bind   0      0
/dev/pts        /dev/pts      none   rw,bind   0      0
/home           /home         none   rw,bind   0      0
/tmp            /tmp          none   rw,bind   0      0
/opt            /opt          none   rw,bind   0      0
/srv            /srv          none   rw,bind   0      0
tmpfs           /dev/shm      tmpfs  defaults  0      0

Now, we have to restart the schroot service, as follows:

$ sudo systemctl restart schroot

Note that you can also restart using the old-fashioned way:
$ sudo /etc/init.d/schroot restart

Now we can list all available environments by asking them to schroot, as follows:

$ schroot -l
 chroot:debian-stretch-arm64

OK, everything is in place and we can enter into the emulated ARM64 system:

$ schroot -c debian-stretch-arm64
bash: warning: setlocale: LC_ALL: cannot change locale (en_GB.UTF-8)

Since we haven't installed any locale support, the preceding warning is quite obvious and it should be safely ignored.

Now, to verify we're really executing ARM64 code, let's try some commands. For example, we can ask for some system information with the uname command:

$ uname -a
Linux giometti-VirtualBox 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 aarch64 GNU/Linux

As we can see, the system says that its platform is aarch64, which is ARM64. Then, we can try to execute our helloworld program that was cross-compiled before; since, after chroot, the current directory is not changed (and our home directory is still the same), we can simply go back where we did the compilation and then execute the program as usual:

$ cd ~/Projects/ldddc/github/chapter_1/
$ file helloworld
helloworld: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=c0d6e9ab89057e8f9101f51ad517a253e5fc4f10, not stripped
$ ./helloworld
Hello World!

The program still executes as when we were on an ARM64 system. Great!

Configuring the emulated OS

What we just saw about schroot is nothing if we do not configure our new system to do native compilation, and to do so, we can use every Debian tool we use on our host PC:

To install a complete compiling environment, we can issue the following command once inside the schroot environment:

$ sudo apt install gcc make \
           bison flex ncurses-dev libssl-dev

Note that sudo will ask your usual password, that is, the password you currently use to log in to your host PC.

You might not get a password request from sudo with the following error message:
sudo: no tty present and no askpass program specified
You can try executing the preceding sudo command again, adding to it the -S option argument.
It could be possible that the apt command will notify you that some packages cannot be authenticated. Just ignore this warning and continue installation, answering yes by pressing the Y key.

If everything works well, we should now be able to execute every compiling command we used before. For instance, we can try to recompile the helloworld program again but natively (we should remove the current executable in order; make will try to recompile it again):

$ rm helloworld
$ make CFLAGS="-Wall -O2" helloworld
cc -Wall -O2 helloworld.c -o helloworld
$ file helloworld
helloworld: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=1393450a08fb9eea22babfb9296ce848bb806c21, not stripped
$ ./helloworld
Hello World!

Note that networking support is fully functional so we're now working on an emulated ARM64 environment on our hosts PC as we were on the ESPRESSObin.