Fully automated Linux desktop install by preseeding a PXE booted LiveCD



PXE booting a Linux distribution is not difficult, and neither is preseeding the installation process. But combining the two for a fully automated Linux desktop installation over PXE in my home network took quite a lot of googling and trial+error. This post covers three non-obvious facts I learned which hopefully will save you some time.
----

After you've done it a few times, installing an OS from a CD/DVD becomes very boring. Faced with yet another laptop to wipe and install Linux Mint on, I decided the days of selecting my Time Zone and partitioning scheme in an installer GUI were over.
It's time to automate!

The end goal

My end goal is to automate the installation process, so all I end up doing is:
  1. Download the Linux distribution ISO file from internet, and store it on my local file server
  2. Plug a network cable into the machine to be upgraded
  3. PXE boot the machine, and select the ISO to install from a menu
  4. Come back half an hour later to find the distribution successfully installed on the hard disk, exactly as I've specified in the preseed file.
No DVD to burn, no installer questions to answer. What's not to like?

The preseed file provides some mechanisms for unattended system tweaking, and this may be all you need. But tools like Ansible, Salt, Puppet, Chef, etc. provide more flexibility for more advanced post-installation tweaks, so as a secondary goal I want the preseed file to also configure the machine for post-install, effortless running of my Ansible scripts. To this end, the preseed file instructs the installer to:
  • create a particular user/password, capable of sudo
  • install a SSH server, start it on every boot, and make it listen on my preferred port (not 22)
  • NFS mount my file server's shared directory of Ansible scripts
If I can accomplish this, the only manual work after the unattended installation has finished will be to:
  • change the user's password, and
  • kick off my Ansible scripts, which tweak my system further
and I will end up with a desktop machine configured just the way I prefer it, with minimal effort.

Three nuggets of info to get us there

Unfortunately most internet sources leave out three facts which can save you days of work when trying to preseed a PXE booted LiveCD installation.
I'll go through them here.

1. A proxy DHCP server saves you from touching your DHCP server

My home router also acts as the network's DHCP server, and not surprisingly this consumer market device does not support PXE booting clients. (I'm reluctant to delegate the role of DHCP server to e.g. my NAS or a separate Linux machine, because I have to be able to shut down everything but my router, and still have internet access from the household's smart phones).

No worries, the PXE specification already has a solution: the proxy DHCP server (Wikipedia)

There are many ways to add a proxy DHCP server to your network, do whatever works for you. My solution was to dig out an old Raspberry Pi model A (abandoned long time ago when the newer models came out), load the latest version of Raspian onto its SD card, boot up the Pi, and simply add 4 lines to the end of its /etc/dnsmaq.conf:

interface=eth0
dhcp-range=10.2.0.0,proxy
dhcp-boot=pxelinux.0
pxe-service=x86PC,"Raspian DHCP forwards to TFTP server for PXE-files",pxelinux,10.2.0.102


This tells PXE clients in my LAN (range 10.2.0.0 - 10.2.0.255) to request the file pxelinux.0 from the TFTP server at address 10.2.0.102 (i.e. the static address of my file server, running TFTP). Note that I don't keep any pxelinux files, images, or preseed files on the Pi, it's simply a proxy DHCP server, so there is no need to enable TFTP on the Pi.

Thanks to iT-Joe for sharing the Pi proxy DHCP configuration.

2. Unpacking the ISO is an alternative to mounting it

Most guides assume you will mount the ISO on the NFS server's file system, but what if your file server doesn't have this capability, or the mount point does not survive reboots of the NFS server?
Easily solved; simply unpack the ISO, so it becomes a tree of files on your file server. (Disclaimer: Only tested for Linux Mint)

3. IMPORTANT! How to separate your installation parameters from your target system parameters 

Providing the right parameters to the relevant "layer" of the installation process is what makes preseeding a PXE booted desktop LiveCD installation tricky:
  1. PXE parameters - The "PXE layer" needs to know the name and location of the kernel and initrd to boot, the boot method, and how/where to find the rest of the distribution files
  2. Install system parameters - Once the Linux system has booted and the window manager presents the desktop, we need to tell it to start Ubiquity (the installer for LiveCDs). Ubiquity will stop early in the installation process unless we up front provide it a locale and keyboard configuration to use during installation, outside of the preseed file. Naturally, we also need to tell Ubiquity to use a preseed file and provide the URL to it.
  3. Target system parameters - The preseed file defines how the target system should be configured. The file's preseeding directives must be recognized as such by Ubiquity
The key point when lining up these ducks is to understand how the lone double dash (' -- ') is interpreted by Linux when it digests boot parameters. Here is an explanation or two

And here (drum roll!)  is how you combine it all as a stanza in your pxelinux.cfg/default file:

LABEL LinuxMint-18-Cinnamon
    MENU LABEL Wipe disk and install Linux Mint 18
    KERNEL images/linuxmint/18/x86_64/casper/vmlinuz
    APPEND initrd=images/linuxmint/18/x86_64/casper/initrd.lz boot=casper netboot=nfs nfsroot=10.2.0.102:/volume1/pxe/images/linuxmint/18/x86_64 -- locale=en_US keyboard-configuration/modelcode=SKIP automatic-ubiquity url=http://10.2.0.102/mypreseeds/linuxmint18-preseed.txt


Let's dissect the APPEND. First of all, it is a single line, although your browser will insert line breaks for improved readability.
Note the '--':
  • Everything in front of '--' is used by pxelinux to netboot the kernel and ramdisk from their respective paths over NFS, before handing over to casper to pull the rest of the distro files from the specified NFS root path (of my Synology NAS file server)
  • Everything behind '--' is used by the Linux Mint 18 system, after Linux Mint has booted for the very first time. 'automatic-ubiquity' starts Ubiquity, and the subsequent URL points to my preseed file (on my Synology NAS' HTTP server).  From my discussion above you will recognize the parameters specifying the locale and keyboard configuration. Without them Ubiquity will hang forever.

And that's really all there is to parameter separation.

My preseed file

For reference I'm pasting in my Linux Mint 18 preseed file here, it should work for Ubuntu desktops too.  A couple of points worth noticing about this file:
  • For my secondary automation goal, see the success_command at the end of the file. in-target means the shell command runs in the target system's shell (affecting the installed system, usually what you want), not the installer system's shell (which runs in RAM and is lost after the reboot)
  • Make sure you end all but the last line of a multi-line success_command with '\'. Failing to do so will not trigger an error message, the remaining lines are simply ignored (!!!)
  • The 'select no' for xkb-keymap and layoutcode selects a Norwegian keyboard, don't confuse this with a negative answer (No vs. Yes)
  • Don't preseed passwords in clear text. Preseed a hashed password instead, or find some other method.
  • Typically I use echo to append text to files, but my echoes in the success_command were consistently ignored, so I ended up using sed instead. (Please leave a comment if you can explain why. Maybe the '>>' confuses Ubiquity?).
 

# My working preseed file for LinuxMint 18 Cinnamon

d-i     localechooser/supported-locales    en_US.UTF-8
d-i     keyboard-configuration/xkb-keymap  select no
d-i     keyboard-configuration/layoutcode  string no

d-i     debian-installer/splash            boolean false
d-i     console-setup/ask_detect           boolean false
d-i     console-setup/layoutcode           string no
d-i     console-setup/variantcode          string

### Partitioning
# If the system has free space you can choose to only partition that space.
# This is only honoured if partman-auto/method (below) is not set.
#d-i partman-auto/init_automatically_partition select biggest_free

# Alternatively, you may specify a disk to partition. If the system has only
# one disk the installer will default to using that, but otherwise the device
# name must be given in traditional, non-devfs format (so e.g. /dev/sda
# and not e.g. /dev/discs/disc0/disc).
# For example, to use the first SCSI/SATA hard disk:
#d-i partman-auto/disk string /dev/sda
# In addition, you'll need to specify the method to use.
# The presently available methods are:
# - regular: use the usual partition types for your architecture
# - lvm:     use LVM to partition the disk
# - crypto:  use LVM within an encrypted partition
d-i partman-auto/method string regular

# You can choose one of the three predefined partitioning recipes:
# - atomic: all files in one partition
# - home:   separate /home partition
# - multi:  separate /home, /var, and /tmp partitions
d-i partman-auto/choose_recipe select multi

d-i     partman/default_filesystem                     string ext3
d-i     partman/choose_partition                       select finish

# If one of the disks that are going to be automatically partitioned
# contains an old LVM configuration, the user will normally receive a
# warning. This can be preseeded away...
d-i partman-lvm/device_remove_lvm boolean true
# The same applies to pre-existing software RAID array:
d-i partman-md/device_remove_md boolean true
# And the same goes for the confirmation to write the lvm partitions.
d-i partman-lvm/confirm             boolean true
d-i partman-lvm/confirm_nooverwrite boolean true

# Just in case, the positive answer to all other imagineable conformation questions:
d-i     partman-partitioning/confirm_write_new_label   boolean true
d-i     partman/confirm                                boolean true
d-i     partman/confirm_nooverwrite                    boolean true
d-i     partman/confirm_write_new_label                boolean true
d-i     partman-md/confirm                             boolean true

# Time
d-i     time/zone               string  Europe/Oslo
d-i     clock-setup/utc         boolean true
d-i     clock-setup/ntp         boolean true
d-i     clock-setup/ntp-server  string ntp.ubuntu.com

#   LinuxMint will *demand* a user during installation, ignoring the value
#   of the passwd/make-user flag, so we need to provide the user data here.
#   You should not preseed the password in clear text, this is just an example!
d-i     passwd/user-fullname            string ubuntu
d-i     passwd/username                 string ubuntu
d-i     passwd/user-password            password ubuntu
d-i     passwd/user-password-again      password ubuntu
d-i     user-setup/allow-password-weak  boolean true
d-i     user-setup/encrypt-home         boolean false

# Use non-free packages
ubiquity ubiquity/use_nonfree boolean true

# -------- Customize at the end of a successful installation.

# Ubiquity completly ignores the debian installer command 'preseed/late_command',
# instead we need to use 'ubiquity/success_command'.

# We permanently enable the wired network,
# install OpenSSH, and move its port to 12345,
# install Ansible locally (just in case),
# and add a NFS mount point in /etc/fstab to my Ansible scripts
# (which will be automatically mounted after the reboot).
ubiquity ubiquity/success_command string \
  in-target sed -i 's/^managed=false/managed=true/'/etc/NetworkManager/NetworkManager.conf; \
  in-target apt-get -y install openssh-server; \
  in-target sed -i 's/Port 22/Port 12345/' /etc/ssh/sshd_config; \
  in-target apt-get -y install nfs-common; \
  in-target apt-get install software-properties-common;\
  in-target apt-add-repository ppa:ansible/ansible; \
  in-target apt-get update; \
  in-target apt-get -y install ansible; \
  in-target mkdir /mnt/Paas; \
  in-target sed -i -e "\$a10.2.0.102:/volume1/public /mnt/Paas nfs defaults,user 0 0" /etc/fstab;

# Finish off the install by rebooting the freshly installed Linux Mint desktop
d-i  ubiquity/reboot  boolean true



Further reading

The internet is full of guides on how to get PXE booting or preseeding going, but many are outdated, inaccurate and/or misleading. Here some of the better ones I came across:
  • The installer in modern desktop versions in the Debian family (Ubuntu, Linux Mint, ...) is Ubiquity, not Debian Installer, and this directly affects your preseed file in many subtle and direct ways:
    Ubiquity automation, the success_command
  • Some examples, mainly targeting Ubuntu, but probably works for most/all Debian-based distros: 1, 2, 3, 4, 5,
  • A Google translated page from a German Synology DiskStation NAS site.

How to set up Eclipse with the gcc C++ compiler and GoogleTest on Windows7, from scratch

With no other requirement than Windows 7 and an internet connection, this tutorial walks you through all the steps leading up to a free development environment where you can code, build and unit test your C++ programs on Windows 7.

The steps and links were valid as of January 2013, but the internet is not a fixed environmnt ;-) For the most up to date information, consult the tool's own home pages.


1. Download and install the 64-bit version of Java from http://java.com/en/download/manual.jsp
    (If you will be using a 32-bit version of Eclipse, you should be able to get away with 32-bit Java)
  
2. Download  and install 'Eclipse for C/C++ Developers' from http://www.eclipse.org
    (This tutorial is based on the 64-bit Juno version, but 32-bit should also work fine)

3. Download and run the latest MinGW installer from
    http://sourceforge.net/projects/mingw/files/Installer/mingw-get-inst/

  When asked by the installer, check "MSYS Basic System" and the C++ compiler (Optionally uncheck the C compiler).
  IMPORTANT: Yes, you need MSYS, because it contains rm, which is used by mingw32-make when cleaning builds.
    IMPORTANT: You must manually add MSYS and GCC to your Windows PATH, the MinGW installer intentionally does not do this for you !!    (Since I installed MinGW into C:\Apps\MinGw, I added  C:\Apps\MinGw\bin\ and C:\Apps\MinGw\1.0\bin to my PATH)

4. Download GoogleTest from https://code.google.com/p/googletest/downloads/list
   and extract the zip file contents into a local directory.
   Rename the directory to C:\Apps\googletest, so e.g. the file README ends up in that directory.
 
    To build the googletest library I did as the googletest README says:

        A.  Open a DOS window
        B.  cd C:\Apps\googletest\make
        C.  mingw32-make
                 (This compiles the library)
        D.  ar -rv libgtest.a gtest-all.o
                 (This create the library libgtest.a out of the file gtest-all.o)
   
At this point, all the tools are installed.
The remaining steps will all be carried out in Eclipse, so start up Eclipse.

5. In Eclipse, create a brand new project for our C++ code:  
              Click File | New | C++ Project
              Project name: my_fist_tdd_project
              Project type  : Executable, Hello World C++ Project
              Toolchains    : MinGw
    The project Eclipse creates is empty, except for 6 include patchs for MinGw.

6.  Change the project's builder to MinGw's builder:
               Click Project | Properties | C++ Build | Tool Chain Editor
               Select 'GNU Make Builder' in the 'Current builder' pull-down list.
               Click Apply

7.  Change the project's build command:
               Click Project | Properties | C++ Build
               Select '[All configurations]' in the 'Configuration' pull-down list.
               Uncheck 'Use default build command'
               Enter 'mingw32-make' in 'Build Command'.
               Click Apply

8. You should now be able to test your build settings:
               Click Project | Build project
               The message "Info: Nothing to build for " appears in Eclipse's console.
                  
9.  Add the path of googletest's header files to the project's include paths:
               Click Project | Properties | C++ Build
               Make sure the selected build configuration is Debug (It is by default)
               Click Settings | Tool Settings | GCC C++ Compiler | Includes
               For the 'Include paths (-I)' list, click the green plus sign, then the File system.. button.
               Browse to googletest's 'include' directory, and select it. (C:\Apps\googletest\include)
               The absolute path now appears in the list box.
               Click Apply


10.  Add the path of the googletest library to the project's linker paths:
               Click Project | Properties | C++ Build
               Make sure the selected build configuration is Debug (It is by default)
               Click Settings | Tool Settings | MinGW C++ Linker | Libraries
               For the 'Libraries (-l)' list, click the green plus sign.
               Enter 'gtest' in the pop-up window.   (Note: not 'libgtest.a')
               This library name now appears in the list box.
               For the 'Library search path (-L)' list, click the green plus sign, 

                  then the File system.. button.
               Browse to libgtest.a's directory, and select the dir. (C:\Apps\googletest\make)
               The absolute path of the directory now appears in the list box.
               Click Apply
   
11.  Add a source folder:
              Click File | New | Source folder
              Folder name: src    (Or whatever you prefer)

Now, in the final step, let's test our environment for completeness:

12. In the new src source folder, click File | New | Source file
      Name it Counter.h and fill it with:
  

        class Counter {
            private:
                int counter_;
            public:
                Counter() : counter_(0) {}
                int Increment();
        };

    Add a second file Counter.cpp in the src folder, and fill it with:


       #include <stdio.h>
       #include "Counter.h"
       int Counter::Increment() {
           return counter_++;
       }

    Add a third file Counter_tests.cpp to the src folder, and fill it with:

       #include <iostream>
       using namespace std;
       #include "gtest/gtest.h"
       #include "Counter.h"

       TEST(Counter, Increment) {
          Counter c;

          EXPECT_EQ(0, c.Increment());
          EXPECT_EQ(1, c.Increment());
          EXPECT_EQ(2, c.Increment());

          cout << "Congratulations, you have just run a test :-)" << endl;
        }

       int main(int argc, char **argv) {
           ::testing::InitGoogleTest(&argc, argv);
          return RUN_ALL_TESTS();
       }


    Click Project | Run
    and expect to see the output from Counter_tests.cpp in Eclipse's Console window:

      [==========] Running 1 test from 1 test case.
      [----------] Global test environment set-up.
      [----------] 1 test from Counter
      [ RUN      ] Counter.Increment
      Congratulations, you have just run a test :-)
      [       OK ] Counter.Increment (0 ms)
      [----------] 1 test from Counter (0 ms total)

      [----------] Global test environment tear-down
      [==========] 1 test from 1 test case ran. (0 ms total)
      [  PASSED  ] 1 test.


             
The set-up is complete, you can now start coding awsome C++ applications, and unit test them with GoogleTest, in Eclipse.  Happy coding!


 -------------------------------------------------

Hint: You will get the error message

          undefined reference to `WinMain@16'
   
from Eclipse if you forget the main() function in Counter_tests.cpp. Just copy the above Counter_tests.cpp and you should be fine.

Small script integrates DesignSpark and VeeCAD

For various reasons, I often use stripboard and through-hole components for my one-off prototypes. After trying different tools for laying out the stripbard, I've settled on a great, free niché product called VeeCAD.

Unlike most layout tools, VeeCAD is optimized for stripboards, i.e. producing a stripboard layout is pretty fast. Since VeeCAD relies on external schematic editors, so you need to import your completed netlist into VeeCAD before the fun can begin. In fact, every time you make a change to the netlist you must tell VeeCAD about it, by importing the new, improved netlist.

Now, if you're like me, your schematics drawings evolve over time. You tweak them, fix a netlist bug, replace a component, etc. And with VeeCAD, every time the netlist changes, you must click through a series of windows to get VeeCAD onto the same page as your schematic editor. ..... Hmmm......this quickly gets old.
There must be a less boring way to keep VeeCAD on top of my changes?

If your schematic editor is DesignSpark PCB (a free tool from RS Components), and you don't mind installing yet another great, free tool (AutoIt (see Wikipedia), the answer is now Yes!

Below is a small AutoIt script I made, which will click its way through DesignSparks' menus to export your (currently open) DesignSpark schematic to a temporary file, and then click its way through VeeCAD's menus to import the temp file as a netlist. Taks approximately 2 seconds.

All in one go, and without any setup or configuration :-)

The script assumes DesignSpark and VeeCAD are both running on your PC when the script starts.
Simply copy the script below into a text editor, and save it with the file extension "au3". Run it by double-clicking it.

And as always with scripts: Remember to save your work before running the script.


#cs --------------------------------------------------------------------------------------

    Are you tired of the repetetive mouse clicking required to export a DesignSpark netlist
    into VeeCad?
   
    This AutoIt script exports the currently open DesingSpark schematics (in Generic Netlist
    format) to a file on your hard disk. The script then imports this file into the current
    board in VeeCad (in Easy-PC format).

    (The file will be created in the same directory as this script runs from).
   
    The script assumes you have DesignSpark running and displaying a schematic,
    and VeeCad displaying the board you want to import the netlist into.
   
    Tested with Windows XP, AutoIt version 3.3.6.1, SparkDesign PCB 3.0, and VeeCAD 2.26
    Should be compatible with most versions, if not you can easily adapt it. Have fun :-)
    Standard disclaimer: Comes with no guarantee of fitness for any purpose.
       
    Copyright: www.leifove.com        License: GPL version 3.0
   
#ce ---------------------------------------------------------------------------------------

AutoItSetOption("MustDeclareVars", 1)       ; To avoid spelling mistakes in variable names.

; ----------------------------- Define some useful variables ------------------------------

Dim $netlist_filename          = "netlist_from_designspark.net"        ; Filetype must be .net
Dim $full_path_name_to_netlist = @ScriptDir & "\" & $netlist_filename  ; Same dir as this script

Dim $timeout_period             = 3   ;  The longest this script is willing to wait. In seconds.

Dim $veecad_window_title_start      = "VeeCAD :"     ; Try changing this if the two programs are not found,  
Dim $designspark_window_title_start = "DesignSpark"  ; but don't make the titles longer than necessary! 

Dim $veecad_windows, $designspark_windows

; --------------------- Define a handy functions we are going to need ----------------------

Func ExitIfWindowNeverAppears($window_title)
   
    If (0 = WinWait($window_title, "", $timeout_period)) Then
        MsgBox(48, "Script terminated.", _
            "Lost my patience waiting for the " & $window_title & " window to appear.")
        Exit
    EndIf
   
    WinActivate($window_title)    ; Just in case, probably not necessary.
   
EndFunc


; ---------- Verify that DesignSpakr and VeeCAD are running, and in the correct state ------

$designspark_windows = WinList($designspark_window_title_start) ; Count the number of open DesignSpark windows

If $designspark_windows[0][0] <> 1 Then
    MsgBox (48, "Script terminated.", _
            "There must be one (and only one) open DesignSpark window!. I found " & $designspark_windows[0][0]  & ".")
    Exit
EndIf   

If WinExists("Reports") Then
    MsgBox (48, "Script terminated.", "Can't run when DesignSpark's Reports window is open.")
    Exit
EndIf   

If WinExists("Output Netlist") Then
    MsgBox (48, "Script terminated.", "Can't run when DesignSpark's Output Netlist window is open.")
    Exit
EndIf   

$veecad_windows = WinList($veecad_window_title_start)   ; Count the number of open VeeCAD windows

If $veecad_windows[0][0] <> 1 Then
    MsgBox (48, "Script terminated.", "There must be one (and only one) open VeeCAD window! I found " &     $veecad_windows[0][0] & ".")
    Exit
EndIf   

If WinExists("Import Netlist") Then
    MsgBox (48, "Script terminated.", "Can't run when VeeCAD's Import Netlist window is open.")
    Exit
EndIf   


; ----------------- Export a netlist from the current DesignSpark window --------------------

ExitIfWindowNeverAppears( $designspark_window_title_start )
   
Send( "!o" )   ; "Output"
Send( "r" )    ; "Reports ..."

;   Work in the Reports window

ExitIfWindowNeverAppears ( "Reports")
ControlFocus  ( "Reports", "", 1100)

; I could not make ControlCommand work consitently when trying to
; select 'Generic Netlist' in the Reports' listbox:
;       ControlCommand( "Reports", "", "ListBox1" , "SelectString", 'Generic')
; ,so the next two lines are a work-around:
ControlSend ("Reports", "", 1100, "{UP}{UP}{UP}{UP}{UP}{UP}{UP}{UP}{UP}{UP}{UP}{UP}{UP}{UP}{UP}{UP}")
ControlSend ("Reports", "", 1100, "{DOWN}{DOWN}{DOWN}{DOWN}")  ; 4th item from the top.

Send( "!r" )    ; "Run" button

;   Work in the Output Netlist window

ExitIfWindowNeverAppears( "Output Netlist" )
ControlCommand( "Output Netlist", "", 1030 , "Check")     ; Include components
ControlCommand( "Output Netlist", "", 1031 , "Check")     ; Include nets
ControlCommand( "Output Netlist", "", 1033 , "UnCheck")   ; Don't want to view netlist file   
ControlSetText( "Output Netlist", "", 1035 , $full_path_name_to_netlist )  ; Where to save file.

WinWaitActive ( "Output Netlist" )                      ; Since we have been focused on panels
ControlClick  ( "Output Netlist", "", 1)                ; Click "Generate". This also closes window.

; Close the Reports window

WinActivate ( "Reports")
ControlClick( "Reports", "", 2)                         ; Click "Close"   

; ----------------------------- Import netlist into VeeCad -----------------------

ExitIfWindowNeverAppears( $veecad_window_title_start )
Send( "!n" )   ; "Netlist"
Send( "i" )    ; "Import"

ExitIfWindowNeverAppears( "Import Netlist")
ControlCommand( "Import Netlist", "", "TComboBox1" , "SelectString", 'Easy-PC')  ; Netlist Format
ControlSetText( "Import Netlist", "", "TEdit1" , $full_path_name_to_netlist )    ; Netlist Filename

WinActivate( "Import Netlist")                    ; Focus on window, since we have been focused on panels
ControlClick( "Import Netlist", "", "TButton5")   ; Click "Import"   

; ------------------ Good, we're all done with the export and import -------------

MsgBox(0, "Done!", "Netlist export and import finished.")  ; Comment out this line to skip confirmation.

Exit




The above script works by pretending to be a human operator clicking a mouse and selecting menu items by name reference. Should the developers behind DesignSpark or VeeCAD in the future decide to rename the controls in their GUIs, the script may stop working. In that case, have a look inside AutoIt's excellent Help function and I'm sure you'll figure out how to tweak my script to make it run again :-) 

Have fun!

Beautify your legacy C code layout for "free"

When I read code, it often strikes me how an unexpected code layout can slow me down and waste my time. I quickly develop a rash when I come across C code formatted like this:

void command_Ping(Transaction_t* RequestIn,
    Transaction_t* RequestOut)
{

    SomeStruct_t *Input;
    VeryDifferrentStruct_t* Output   ;
    DataIn_t DataInput;
    MyData_t* DataTest;
   int SampleLoop=1, OuterLoop, InnerLoop;
    static Bit16_t  SampleNumber  =0;
    Input=(SomeStruct_t*)&RequestIn->Param;
    Output = (VeryDifferrentStruct_t*) &RequestOut->Param;
   if ((RequestIn->Param2 & HIGH)== HIGH)
        {
#ifdef SIM /* Applicable for simulator */
    CallSim(&RequestIn->Param,&OuterLoop,LOOP&8);
#else   
    memcpy(Output,Input,PACKET_PARAM_SIZE/2);
#endif
    } else {
   if ((RequestIn->Param2 & LOW) ==LOW)
    {
    ChLoop++;
  }
    }
    RequestOut->Length=RequestIn->Length-6;
}


But if you work with a legacy system, you can't improve the above code, because *any* code change in a system without automated tests comes with a severe penalty: extensive, manual regression testing.
So we don't fix it, and the ugly code stays in the system forever, awaiting the next unsuspecting developer that comes along.  Over time, both you and the system become increasingly miserable.

But wait, there is hope!! The reason we don't change ugly code layout by hand is the risk of introducing new bugs, right? What if there was a guaranteed bug-proof way to change source code layout? Then we could safely drop the expensive manual regression test, and legacy systems would see a brighter future :-) Yeah!

As you may have guessed by now, the secret is to let an automated "indenter" tool make the layout changes, through whitespace removal/adding. Assuming that the tool itself is bug-free, the code that comes out of the tool will look unchanged to the C compiler, i.e. containing exactly the same bugs as the original code, and no new ones. But to you and me, the code will look much more in line with our expectations for cleanly written C code:

void command_Ping(Transaction_t *RequestIn,
                  Transaction_t *RequestOut){

    SomeStruct_t           *Input;
    VeryDifferrentStruct_t *Output;
    DataIn_t               DataInput;
    MyData_t               *DataTest;
    int                    SampleLoop   = 1, OuterLoop, InnerLoop;
    static Bit16_t         SampleNumber = 0;

    Input  = (SomeStruct_t *) &RequestIn->Param;
    Output = (VeryDifferrentStruct_t *) &RequestOut->Param;

    if ((RequestIn->Param2 & HIGH) == HIGH) {
#ifdef SIM /* Applicable for simulator */
        CallSim(&RequestIn->Param, &OuterLoop, LOOP & 8);
#else
        memcpy(Output, Input, PACKET_PARAM_SIZE / 2);
#endif
    }
    else {
        if ((RequestIn->Param2 & LOW) == LOW) {
            ChLoop++;
        }
    }

    RequestOut->Length = RequestIn->Length - 6;
}



(OK,  maybe your expectations are different from mine, but the code is easier to read now)

Spoiler: The above code transformation was produced by the open-source tool Uncrustify, using a settings file I generated after some hours of tweaking 350+ Uncrustify settings in UniversalIndentGUI. (Wikipedia)

The beautifier tools I've looked into typically spend a fraction of a second on improving a large C program. So basically, if you point the tool to your large C/C++ system, all .c, .cpp, .hh and .h files will be beautified in less time than it takes you to finish lunch. All that is left for you is to check the improved sources back into your CVS. No regression testing required!

And it gets better: regardless of your platform (Linux, Mac,Windows), you will find high-quality open-source C/C++ beautifiers on the www. To cut your chase short, I strongly recommend you to download and install the "meta tool" UniversalIndentGUI, as it will enable you to try out the 20+  different beautifiers it comes bundled with (e.g. Uncrustify, GreatCode, BCPP, GNU Indent, and others).
And most importantly, UniversalIndentGUI will allow you to see the immediate effect on source code layout as you tweak the numerous settings offered by the different tools. The interactive process UniversalIndentGUI offers for exploring settings will save you hours, as you work on the settings for your preferred layout.

When you are happy with the results in UniversalIndentGUI, simply export the settings file, and point your selected beautifier to it. (Or you can freely grab my Uncrustify settings file here. Tested with Uncrustify 0.53)

No code layout or style will please everyone in a team of developers, but with an automated beautifier, you can significantly improve source code readabilty in legacy systems, (almost) for free.

Happy beautifying!

Continuous Integration trades fear for instant gratification

Automated build and test (a.k.a. Continuous Integration) has rapidly become the way to develop IT systems. There are many reasons for this (a maturing IT industry, availability of C.I. tools, etc), but I think the most powerful driver is the sheer enjoyment a developer can derive from the C.I. process, in particular:
  • Instant feedback on how good you code is (as measured by unit tests), determined in a risk-free environment
  • The ability to fix a bug (found by unit testing or other automated testing methods) while the program flow, class structures, etc are fresh in your mind.
In short, Continuous integration is a perfect match for a world where instant gratification is the default expectation.

Now, contrast this to the working environment for anyone maintaining a complex IT system built 10-15 years ago, i.e. a legacy system without automated testing facilities:
  • Every code change is associated with fear ("Will this change have any unwanted side effects?", "Will I lose my job if a destructive side effect sneaks past the fallible, manual testing stages into Production?")
  • A high percentage of the bugs reported from Production will be of the "Duh! How could we have missed this bug before deployment?" category, quite simply because a completely manual test regime is bound to have incomplete coverage. And that's exactly the category of bugs which make the poor maintainer look incompetent.
If you've tried both worlds, you are not going back. Continuous integration is here to stay.

Why I use Perltidy, or Why coding style matters

There are Few.
areas of programming that-are more likely to, cause religious wars than! coding: standards some developer's


       consider. coding style? to be an area where personal freeDom should reign supreme unrestricted by team project; or company standards but as
these:
few
    lines
Of
text hopefully    demonstrate punctuation-indentation-etc is important WheN conveying ideas and thou'ghts in a written form.
........

OK, if you made it this far down into my post, I promise to stick to the standard rules for punctuation and capitalization of English. By now you probably agree that reading speed and comprehension is affected when the writer ignores commonly accepted practices. Only when the text format matches your expectations can you focus your attention fully on the message, instead of the format. Of course, the same principle applies to reading source code.

One of many, many gems the Perl community has produced is Perltidy, a great source code formatter for Perl code. The details are here:
http://search.cpan.org/perldoc?perltidy

Simply by using this as my .perltidyrc parameter file:

#----------------  PBP options, as listed on page 35 of PBP  ----------------
-l=78   # Max line width is 78 cols
-i=4    # Indent level is 4 cols
-ci=4   # Continuation indent is 4 cols
-vt=2   # Maximal vertical tightness
-cti=0  # No extra indentation for closing brackets
-pt=1   # Medium parenthesis tightness
-sbt=1  # Medium square bracket tightness
-bt=1   # Medium brace tightness (for non-code blocks)
-bbt=1  # Medium block brace tightness (for code blocks)
-nsfs   # No space before semicolons
-nolq   # Don't outdent long quoted strings
        # Break before all operators:
-wbb="% + - * / x != == >= <= =~ < > | & **= += *= &= <<= &&= -= /= |= >>= ||= .= %= ^= x="
# Note that the PBP book has a typo on page 35 when defining the above -wbb
# option, see http://oreilly.com/catalog/perlbp/errata/perlbp.confirmed
# The above string is correct.

#----------  Perltidy default options (i.e. in line with PBP)  --------------
-asc    # Add any missing optional semicolon at the end of a line
-aws    # Add certain whitespace to improve code readability
-tqw    # Default handling of multi-line 'qw' quotes
-ibc    # Indent block comments to same level as the code
-msc=4  # Minimum 4 spaces between code and same-line comments
-fpsc=0 # Don't try to line up all comments to a fixed column
-hsc    # Align hanging side comments
-sbc    # Enable static block comments
-nssc   # No special handling for static side comments
-dnl    # Delete old newlines
-bbc    # Ensure a blank line before every full-line comment
-bbs    # Ensure a blank line before every sub definition (except one-liners)
-bbb    # Ensure a blank line before code blocks (for, while, if, ....)
-lbl=8  # Minimum number of lines between each -bbb inserted blank line

#-------------  Additional options, based on PBP recommendations  ------------
-bar    # K&R style code braces
-nolc   # Long comments indented, even when this make the total line length "too long"
-noll   # Long lines indented, even when this make the total line length "too long"
-nola   # Don't treat labels as special cases when indenting

#----------  Options concerning Perltidy's input and ouput files  -----------
-nst    # Do NOT output to STDOUT (since we want to use -b)
-b      # Backup the input file to .bak and perform all tidying in the original file
-se     # Write errors to STDERR
-ple    # Preserve the EOL character(s). E.g. in case Perltidy runs
        # on Windows, but processes a Perl file for Unix (and vice versa)

#-----------  Some other Perltidy options, intentionally not used   ----------
# The following Perltidy options are NOT consistent with PBP and should NOT be used:
# -lp, -icb, -fws, -nwls, -nwrs, -sfp, -sak/-nsak, -csc/dcsc, -fnl,
# -ce, -bl, -sbl, -bli, -blil, -bbvt, -otr, -sct, -boc, -kis,
# -pbp (because the -wbb list typo in the PBP book carried over to Perltidy (!))
# The following Perltidy options are not used, for other reasons:
# -sot, -mft, -cab, -bol, -bok/-nbok, -bot/-nbot, -iob, -isbc


I am free to sputter butt ugly code like this:

   sub  manipulate_some_data{

my $self   =  shift; my $data = shift;
# Sometimes add to the data we received
if($data  eq 'good data'  ){
$data.= ' is now even better.'
;}return $data;}

via my keyboard, and have it instantly transformed to this:

sub manipulate_some_data {

    my $self = shift;
    my $data = shift;

    # Sometimes add to the data we received
    if ( $data eq 'good data' ) {
        $data .= ' is now even better.';
    }
    return $data;
}

by Perltidy. Without wasting a single of my overworked brain cells.
(I have configured my IDE to run Perltidy every time I save my source file to disk)

(Click here to download the above .perltidyrc file, if your're too lazy for a copy&paste operation)

Perltidy cleans up my code while I develop, which is great, but the true benefit of Perltidy becomes apparent when you or someone else tries to understand your code 4 weeks later: Since all code you and your teammates produce have the same layout, everyone reads code a little faster and get a quicker understanding of what is intended.

This positive effect is particularly evident when fixing a bug or adding some small feature to existing code, because in these situations you typically spend most of your time skimming code, and only a small portion of your time on the few, new lines.

Perl has crossed oceans and climbed mountains to make hard things possible, don't sacrifice this hard-earned property by leaving messy, sloppy, ugly code behind. Clean it up as you go along, by running perltidy from your IDE, and/or as part of your versioning system's commit process.

Free lunches are hard to come by. Perltidy is one of very few I've come across.

C++0x learns some tricks from Perl

Perl 6 has been in the making for some time, but so has also the long awaited upgrade of C++, aka C++0x (Wikipedia). Now the Working Committee for C++0x has released their 1310 pages draft, i.e. the proposal for the new C++ standard (Download the .pdf here).

The list of proposed changes to the core language and its libraries is extensive, but some of the changes are particularly interesting for us Perl developers: C++0x is about to embrace some old-time favorite features from Perl! Assuming the proposed changes are voted into the new standard, C++ developers will in a few years be able to take advantage of these cool features:
  1. The for_each keyword becomes part of the core language, to allow easy iteration over a function, of all elements of a sequence (see page 855). OK, not as cool as Perl's iteration over all elements of a list, but still quite useful.
  2. The Perl workhorse associative arrays, better known as hashes, become part of the C++ standard library (see page 794). The proposal goes further than Perl 5's hashes, by defining 4 different hash types. The 'unordered set' hash type looks an awfully lot like a Perl list.
  3. Regular expressions get their own, standardized library (see pages 1060-1095), where the well-known Perl functions s/// and m// find their equivalents in regex_replace, regex_search and regex_match .
The 1300 pages draft is a brick, but when the votes have been counted and the C++ compilers have been upgraded in 2-3 years time, C++ will be a more powerful language.

Installing Eclipse and the EPIC plugin for Perl

I prefer Eclipse for my Perl development. Just in case you've been living under a rock the last 5 years (or have been stuck in the MS Visual Studio sphere), here is the link to the Eclipse community, where you will find instruction for downloading the Eclipse IDE.  Be warned, the Eclipse community comprises tens of projects, and the web site is not completely intuitive....
For Linux users you are probably better off installing through your distributions package manager. Faster and safer.

Important: Get the Eclipse IDE for Java, since it comes bundled with ant, which you will need for EPIC. (Alternatively, install the Eclipse Platform version, then install ant). On Ubuntu you also need to install the eclipse-pde package (the Eclipse Plug-in Development Environment), it only takes a few seconds in Synaptic.
EPIC's installation instructions claim you only need the Eclipse Platform version, I have found this to be incorrect.

So what is this EPIC thing, you say? Well, for Perl development the EPIC plug-in is what makes the whole difference: A Perl perspective, syntax coloring, variable name completion, error messages next to the troublesome statements in your source, Perl Critic integration, perltidy, single stepping with variable watch, etc, etc.  In short, you don't want to develop Perl in Eclipse without it.

Installing the EPIC plug-in in Eclipse is really simple, similar to other plug-in installs. Just add the right URL under Eclipse's Help | Install software .... menu option, and off you go! To learn more about this great plug-in, head over to the official EPIC web site, where you will installation instructions and other useful info.

The Eclipse/EPIC combination makes for a great IDE for Perl development. I use it regularly on both Windows and Linux machines.

Still in love

Ever since I first got exposed to Perl, the beauty and fluidness of the language continues to impress me. Just like C++, Perl has a steep learning curve, but as your abilities mature you start to appreciate the finer details of the language, and its culture.

http://oslopm.blip.tv/file/1903160/ is an excellent example of the pure joy Perl programmers share; Jos Bouman's talks about the inner secrets of Perl and perl at Nordic Perl Workshop 2006.

The video was published by Oslo Perl Mongers, click their logo to see other videos they've made available.

CodeWarrior meets Eclipse, download Beta now

Having used Eclipse for some years I was relieved to hear that Freescale was merging CodeWarrior with Eclipse. Finally, a Freescale IDE for grown-ups!

To me, Freescale's CodeWarrior IDE for microcontrollers always looked a little outdated. The drab color scheme, the quaint icons, and the tiny window for error messages gave a less than sophisticated impression.
Don’t get me wrong; you could edit and build source code quickly, and to program your microcontroller target was a no-brainer. Single-stepping your target was equally easy, so in essence CodeWarrior IDE did exactly what it promised. But just like other microcontroller IDEs I’ve used, CodeWarrior felt confined.

So how does the Eclipse-based Beta version do? Here are some of my initial observations:
  • Yes!! It looks and feels like Eclipse!
    To someone who’s never tried Eclipse before the concepts of views, perspectives, launch configurations, etc. may be confusing at first. But this flexibility comes at a modest prize: Spend some time in the IDE, and the initial frustration will melt away.
  • Importing  your existing CodeWarrior projects into CodeWarrior for MCU v10.0 is a breeze, just start the Import Wizard from File | Import …. and browse to the CodeWarrior Project file (*.mcp).
  • Where the difference between previous versions of CW and the Eclipse-based version becomes very obvious, is when you attempt to program the microcontroller’s flash for the first time. Where you before could simply click an icon, you now have to create a launch configuration. (Freescale’s site has a nice little video showing you how to get this right the first time, don't worry).
    Once over this hurdle, programming your target’s flash is only a couple of mouse clicks away.
  •  I had the DEMO9S08QD4 demo board from P+E Micro lying around, and after plugging it into the USB port, CodeWarrior updated the demo board’s firmware. Not sure if this was really required, but it doesn’t matter, because the board still works fine under the previous CW version.
  • Yes, CodeWarrior is now available for Linux! But there is a major snag: You need to compile the USB driver for your programmer/development board.......! Linux header files, gcc and the whole shebang! This is just not acceptable, and has to become easier in the official release.
Thumbs up, Freescale.

Booting the blog !!

printf  'Hello World\n';


printf("Hello World");


PRINT "Hello World"


serial_out("Hello World");


LDX  @HELLO_WORLD
JSR   @PRINT_TEXT