Raspbian Stretch

I had cause over the past few weeks to update an RPi to Stretch.  It was an old build so I decided a new install would be best.  I use the lite version as I don't use a screen and don't need a desktop, I just use putty to establish an sssh session.  As usual the download and basic installation process was easy.  The mechanism to start ssh on the installed system worked fine but it was a nightmare trying to setup a static address on the ethernet interface as networking has been changed so that the interface is bound to the mac address and you cant just use eth0 as previously.  I left it for a few days and found a new version was released on 07SEP which removed / corrected this change so that I could setup ip.
I was now able to get the website (lighttpd) working so that RPi became useful.  The next piece was to setup mysql database; install mysql-server now causes mariadb to be used rather than mysql.  It turns out that the developer of mysql is maintaining mariadb whilst Oracle maintain mysql.  Lastly I needed to install PHP which was troublesome.  PHP has been updated to version 7 but after installing it lighttpd told me it was still using version 5.5.  I followed some non-intuitive steps provided by some kind person to fix it.
Now I have a shiny new headless RPi server which will:

• Strip out the adverts and play internet radio stations (streamripper)
• Allow me to play my albums using my own web-based front-end (javascript, jquery, php)
• Let me use any of my bluetooth devices for listening
• Run a bit torrent client for downloads
• Allow file-sharing with Windows PCs (samba)
• look at my artwork database together with commentaries I have written (mysql, php)
• Share videos locally which can be played on TVs or other devices (minidlna)

The old system ran in read-only mode with all data held on usb sticks.  The new installation hasn't got that far yet.  Also I haven't re-added my network of temperature sensing arduinos communicating using LRF24N01+ 800MHz wireless devices.

RPi PC Desktop

In December 2016 I was interested to see the announcement of a version of Raspbian PIXEL intended to run on a PC.
I dont want to run a GUI on RPi generally as the tasks I RPi to carry out are based on command-line processing and I prefer to run RPi headless with little software installed.  However using an old laptop to run Raspbian is a very attractive idea.  I can run GNU/linux GUI using the same underlying Debian flavour of Linux and remain within the RPi world.
Initially the software was only distributed as a live DVD image, so you could boot an image but not install it over the existing system on a laptop.  Development has moved forward and sometime this year an installable image was released. I noticed it last week and have now installed it on Alex's very old notebook (Hi-Grade M72R 2009 or earlier).  The audio jack doesn't work and the battery is completely dead but that shouldn't be too inconvenient.
I am very happy with the result.  We have the chromium browser and a basic set of applications available to us and responses are very reasonable.  I had to think a little to get wifi and bluetooth working but now that they are I should be able to do anything I want with it.

TNMOC

There is so much for me to see at The National Museum of Computing in Bletchley that I can go there again and again.  I treated myself to a couple of hours there on Thursday afternoon.
The story of the decoding of "Fish"/"Tunny" traffic created on German Lorentz cypher machines amazed me as it has done before.    Whereas the more famous Enigma traffic was for highly encrypted for short mobile / battlefield communications Tunny was used by top commanders, including Hitler, in Berlin to send strategic information to army commanders in the different theatres of war.  The Enigma machine was known to codebreakers as simpler variants had been available before the war.  The Lorentz machine automated the coding and decoding process using teleprinters which produced 5 bit baudot codes.  The codes were encrypted using a machine with 12 rotors (enigma used 3) and sent out via radio to other european destinations.  Communications speed was too high for an intercept operator to reliably note down the results so they were recorded on ululating tape and transcribed afterwards.
The big break in understanding traffic came when a message was sent with the same encryption settings twice, but with some small variations.  This allowed Ralph Tiltman at BP after considerable effort to decode it.  Some time later a young Cambridge mathematician Bill Tutte analysed the message in great depth over a 12 month period and managed to deduce from patterns in the coded message how the Lorentz machine was constructed, that it had 12 wheels and how many teeth / positions were on each, a phenomenal achievement based on the contents of a single message.
From this point BP code breakers were able to manually break Tunny traffic manually although it could take weeks to decode messages.
A machine called the Heath Robinson was designed to help the codes breakers analyse messages.  It had a paper tape containing the message which was repeatedly scanned to do a statistical analysis.  However it wasn't fast enough and prone to breaking.
Max Newman persuaded BP management that a better solution was required and Tommy Flowers at the Post Office research in Dollis Hill was approached.  BP didn't feel that his solution using over a thousand valves and optical readers would work but he had such faith that he was able to build a machine, called Colossus and deliver it early in 1944 just in time to decode vital German communications in advance of D day.  Using colossus BP were able to decode Tunny messages until the end of the war.

Windows Sub-system for linux

Windows Sub-system for Linux (WSL) was released about a year ago as a new feature for Windows 10 64-bit.  It is the result of a partnership between Canonical (makers of Ubuntu) and Microsoft. WSL implements a linux userspace which is based on Ubuntu linux binaries. So the file system (/usr, /lib, /home etc) is very familiar and you can use linux commands (cp, grep, apt, gcc).  When you first enable WSL you run apt-get update && apt-get upgrade and install other packages just as you would on any other system.
Win10 provides the kernel functions, Microsoft don't use the linux kernel, instead the Windows kernel is invoked to carry out those activities.  This means the overheads are much lower than those provided by a Virtual Machine which needs a linux kernel and an extra layer of software to translate to Windows.  On the downside, any kernel functions that Microsoft haven't implemented can't be used.
WSL is intended to provide linux command line rather than a graphical interface. You access WSL by starting a bash shell, a very similar experience to opening a cmd command prompt.  You can then use bash and linux functions as normal (using exe's provided by Canonical).  You can access and write to Windows using mount point /mnt/c.  You have your own copy of the linux filesystem in appdata/lxss/... but you should not write from Windows into the linux filesystem.
Installation is just a matter of enabling the WSL feature.
A very informative 20 minute video by Scott Hanselman provides all you need to know to get started.

Cross-Compiling using WSL

Almost by accident I found this weblog by Eswar Prakash which describes how to install a linux sub-system under Windows.  He provides a very clear explanation and comprehensive instructions to create the environment, install and update software.  Once the Linux subsystem has been enabled in Windows it looks very realistic to me.  linux directory structure is in the usual place, I have /home/john to store information and I can use standard linux commands (e.g. df, apt-get, ps -ef).  Windows C: and D: are mounted as /mnt/c and /mnt/d so I can move information between linux and Windows.
The gcc compiler works fine as an assembler for me as well so I can create kernel.img for Baking Pi tutorial with:
arm-linux-gnueabi-gcc -c main.s -o kernel.img -mcpu=arm1176jzf-s -mfloat-abi=soft
As previously I can load kernel.img onto my RPi and run the program with an OS.
I think this is wonderful, I have the power of linux with the ease of a command prompt and there isn't a learning curve to get started.

Baking Pi Tutorial : Getting Started

Alex Chadwick at Cambridge University Computer Laboratory published a wonderful set of bare metal tutorials, some time around 2012 when RPi was first released.  After many years I decided to have a look at them.  I possess an RPi model B which the tutorials refer to which I will use for the tests.  I also have a dodgy SD card from a previous Jessie installation and I will use the FAT partition on that to load programs.

Unfortunately the compilation environment guidance wasn't very clear to me.  I wanted to use Windows for my program development but the downloads page suggested using YAGARTO (yet another GNU ARM toolchain) and MinGW (minimalist GNU for Windows) to achieve this, without providing guidance on how these should be setup for the cross-compiler.  YAGARTO development appeared to be completed or terminated in 2013 and instructions were minimal / unclear.  Although my MinGW installation was successful I couldn't (easily) get YAGARTO and MinGW to work together.

For initial testing I thought I would use a linux environment.  Clearly the obvious one to choose is an RPi!  Compiling on RPi means that we don't need a cross-compiler.  I used an RPI B+ running Stretch as my environment:
 Linux version 4.9.41+ (dc4@dc4-XPS13-9333) (gcc version 4.9.3 (crosstool-NG  crosstool-ng-1.22.0-88-g8460611) ) #1023 Tue Aug 8 15:47:12 BST 2017
I did need to install the library headers before programs would compile:
 apt-get install gcc-arm-none-eabi
I setup a BareMetal directory structure as directed and downloaded the template script for Makefile compilation.
I then typed in the small assembler program OK01 for Lesson 1 and ran make to build a standalone executable kernel.img.
I inserted my SDcard into my Windows computer copied kernel.img across, having renamed the old linux kernel.img.
Finally I inserted the SDcard in RPi model B, applied power and the LED lit as expected.

RPi cross-compiler

If we will be doing bare metal programming we need a cross-compiler.  Usually, when you compile a program you are expecting to execute it on the same machine or same type of system.  If I compile a C program on Windows I expect it will run on my Windows computer or other Windows computers but not on a linux system or a Mac.  A compiler which can generate programs for another system is a cross-compiler.
A compiler has to look at a program and translate it.  This is done in (at least) two stages, the first compiles the program into assembly language and the second assembles it into machine language.  For example a program snippet:
     if (letter='Z')
      {position=26;}
could compile to something like:   which is then assembled to machine code binary:
  load R1,'Z'                                          111100011111100011
  load R2,letter                                      111100101111100011
  compare R1,R2                                   0000110010101010
  branch-not-zero label$                       0001110000000000
  load R1,26                                          111100101111100011
  store position, R1                                01110000001101010
 label$
The assembly code is specific to a system type and versions for Windows and Mac would have different instructions.  A new compiler version has to be written for each system.  Typically the assembler converts each line of code into a binary machine instruction so that a binary file can be loaded into the target system.  A cross-compiler needs to be told what the target system it is so that it can create appropriate binary files.

In addition to a cross-compiler we need some extra utilities to run programs a linker (to join the program with appropriate library routines) a loader (to copy the program on to the target system) and others.

Windows and linux systems are rather different so a Windows cross-compiler toolchain which generates RPi code is not terribly straightforward.  The easiest way to compile bare metal programs was to compile them on a RPi running linux and run them on another.
The  GNU (open source) C compiler gcc provides the necessary facilities to cross-compile so what we need is an environment to run gcc and the appropriate Raspberry pi definitions.
There are various options for compiling under Windows and I will look at some of them.  The main ones cygwin and mingw provide a linux environment allowing gcc to be used but there are others which I will try.

RPi Boot Sequence

What happens when you apply power to an RPi and start it up?  This is a surprisingly interesting and complex subject.
Assuming we have a screen attached we can see a load of messages displayed as the linux Operating System (OS) starts up, at the end of which, when everything has been loaded, a login prompt is displayed.  GNU/linux has a program called the kernel which is loaded into memory and then executed by the ARM CPU.  The messages we see on the screen show what the kernel does at it sets up the hardware and software to make RPi useful.
So how does the kernel get into memory?  To understand this we need a few details about RPi architecture.  A good summary of different RPi models and their configuration is provided on wikipedia.  I shall assume we are using the first RPi, the model B, which includes a BroadCom BCM2835 SoC (System on a chip).  The BCM2835 includes some ROM, 256MB RAM, an ARM11 CPU and a Videocore IV GPU (graphics processor unit). The reset of the RPi board provides circuitry and connectors for power, SDcard, USB, video output, ethernet etc.
The kernel is stored on an SDcard as part of the image which is copied to the SCcard as part of the installation process.  This image contains two parts a FAT32 partition and an EXT partition containing OS files.  The small FAT32 partition contains a minimal number of files to start up the system.  These are bootcode.bin (2nd stage bootloader), start.elf (3rd stage bootloader), kernel.img (the kernel), config.txt and cmdline.txt (parameter files).
When power is supplied to the RPi board the GPU in the BCM2835 is hard-wired to run the 1st stage bootloader program in the onboard ROM which enables it to read an SDcard containing bootcode.bin into 256kB L2 cache memory.  bootcode.bin enables the main memory 256MB and loads start.elf which loads kernel.img into RAM and enables the ARM processor which then executes the kernel.
Here are a couple of references in a forum and blog which I found when investigating this.
When we start bare metal programming we can use this boot sequence to load our own programs without an OS.  All we have to do is compile our program in the right way to run on a RPi and put the code onto an SDcard with the name kernel.img.  When power is applied our program is run by the bootloader.

Bare Metal RPi Intro

Bare Metal programming allows programs to be run on hardware without the need for an Operating System. It is the norm for microprocessor chips such as Atmel and PIC.  The Arduino IDE makes this particularly easy for Atmel chips by providing a mechanism to load programs and many sample working programs which can easily be executed on an Arduino.
By contrast more powerful hardware systems run Windows, MacOSX, GNU/linux or other OS variants.  Raspberry Pi (RPi) comes under this heading, it is often loaded with Raspbian which is a full featured version of Linux tailored for RPi hardware.
It is interesting to consider how to use an RPi without an OS, in particular to better understand how a system starts up, how the components in an RPi communicate with each other, what tools are needed to build basic systems and discover what the different components do.  It would be a huge amount of work to write an OS myself although this was an objective of the nandtotetris tutorials (which I will write about sometime).
Googling gives a good range of information available for RPi.
 Baking Pi - OS development by Alex Chadwick from the Cambridge University Computer Laboratory provides an excellent starting point.  Lesson 1 guides you through writing an assembler program to light the LED on RPi.  Further lessons expand knowledge of assembly language programming and move on to writing to the screen.
Valvers provides a four-step tutorial based on the content of the Baking Pi project which allows you to write C programs and uses a much simpler platform to cross-compile programs.
David Welch appears to have a variety of bare metal programs including  serial port (UART) programming which should be interesting.
Circle appears to provide 30 steps / examples which I will investigate.
OSDev.org  is a serious bare metal development site which appears to provide a small but proper kernel for development.

A number of unusual challenges face us when starting bare metal programming.

1. How to compile a program to run on a different computer?
   Compilers are usually designed to compile programs on the computer you are using so what environment allows you to develop programs to run directly on an RPi.
2. How to load and run an executable?
   The basic approach is to load the program onto an SD card and use the same procedure used to load linux, but it is also possible to load via a serial port (David Welch).
3. How to write programs that don't need an OS?
   It is possible to use the special version of C used for writing the linux kernel and low level assembler libraries can be used.
4. Where can the information on the relevant hardware, interfaces and libraries be found?
   This is crucial information which the tutorials and forum posters can help to provide.

Where to blog

This is a good place to start blogging - it is simple and it is working.
Wordpress is the most widely used blogging software, it is free, supposedly easy to install and has thousands of themes to personalise the look and feel.  It is used by individuals and businesses and a site can be hosted at wordpress.com or by a hosting provider.  Joomla appears to be a popular alternative.
Hostinger provide my hosting, they have an excellent range of facilities and services considering it is free.  It provides mysql and php which wordpress requires.  For the free version you need to install wordpress manually via ftp.  For the premium version, currently only £3 per month, a "one-click" installer is required.
Wordpress Theme Library provides a huge choice of themes, I thought of trying 2017 or writers but I need to blog for a while to determine which I need.
For the moment I will work on blogger and see what comes out.