ESP-IDF

Intro

ESP-IDF (IoT Development Framework) is the official IDE from Espressif Systems for their microprocessors including ESP32.  As such it can be used to build and download images to ESP32 like the Arduino environment.  Whereas Arduino (successfully) endevours to make programming as simple and painless as possible ESP-IDF enables the user to utilise all ESP32 functions.  Even simple IDF programs can be challenging to develop but it is extremely useful in situations where Arduino IDE doesn't work properly for a particular application.  Arduino uses a SETUP + LOOP configuration, whereas IDF uses function interrupts / callbacks to deal with inputs and other events.

Environment

I installed ESP-IDF v6.0.1 for Windows PC.  Installation is very straightforward and it takes a few minutes to download and configure all the code. Clicking the resulting icon gives you a Powershell command line session.  You then use a python script idf.py to confirm which chip is being used (set-target) compile (build), download (flash) and open an ESP32 serial terminal (monitor).  The build process has many (automated) steps and creates a bootloader, executable image and any other required code.

Comprehensive documentation is provided by ESP and there is an excellent set of examples covering most (all?) available functionality.

Examples are vital as there is a very steep learning curve when learning how to control a range of hardware functions.   Most functions (e.g. GPIO, I2C, wifi, web servers) can be implemented by taking one or more example programs and adapting them as required.

RNT Book

There is a bit more to IDF than I can learn by browsing the programming examples and I struggled with IDF attempting to make BreezyBox work on ESP32S3. 

On 25th August, whilst on holiday I received a circular from Rui Santos announcing a new book "Learn ESP-IDF with ESP32".  Random Nerd Tutorials was setup a few years ago by Rui and his partner Sara.  They have published many, many high quality Arduino ESP8266/ESP32 and RPi tutorials from which I have benefited greatly.
They obtain an income from ebook publications and ,although I have been keen to purchase, none has been quite appropriate before.  This one seemed perfect and I jumped in to purchase it on the publication day.

Chapter 1 covers ESP-IDF installation in a Visual Studio (VSC) context and a few basic concepts.  Chapter 2 covers GPIOs; digital outputs, digital inputs, analond inputs and button debouncing using interrupts.  Although the descriptiona are light and superficial you end up with the ability to program many GPIO functions by adapting sample programs.

After Chapter 2 you have the flexibility to cover topics of specific interest: deep sleep, I2C, Wifi, BLE or ESP-NOW.  Using wifi and web servers is my first interest, but I will also cover ESP-NOW which provides networking for ESP32 devices.

USAGE

Once you get used to it, ESP-IDF doesn't seem too difficult.  Examples provided by Espressif look complicated because they are properly written, testing for errors for each hardware operation which makes them rather long.   In comparison Arduino uses standard functions for many different board types so their functions are well known.  For simple applications Arduino is certainly easier and quicker.

I expect to use ESP-IDF in a number of situations:
-  Assembly language programming
-  Whereever a software packages uses dependencies or specifies ESP-IDF install instructions
 -  If I need to use Callback functions.

What is an ESP32?

ESP32 is a 32-bit processor from Espressif Systems with wifi and bluetooth connectivity.  It is a successor to ESP8266 (32-bit cpu and wifi) which became extremely popular with the make community because of its low price and wifi connectivity.  The Arduino IDE incorporated support for ESP8266 and ESP32 which gave hobbyists a simplified approach to build applications for the system.

The first ESP32 was first released in Sepember 2016 so the reange is now ten years old.  There are now two principal designs "S" and "C" based on Xtensa and RISC-V architecture respectively.

ESP32-S3 is based on dual core Xtensa LX7 processors.  Xtensa is marketed by Cadence Systems in San Jose, California.  These processors  can be customised to provide a specific set of functions which the manufacturer requires.  It looks like you can use an IDE to design your processor and it is manufactured based on the results.

ESP32-S3 is a SoC (System on a chip) with 240MHz dul Xtena LX7 CPU, 512KB SRAM (Static RAM), 384KB ROM, 16KB RTC RAM (Very Fast RAM), 8MB PSRAM (Pseudo-static slow RAM), 45 GPIO pins, 3 x UART.  Internal ROM is used internally for wifi drivers, RTOS etc.
The SoC is often combined with extra components in a module board such as ESP-WROOM-32. Mine is an ESP32-S3-N16R8 which has 16MB flash ROM as well as 8MB PSRAM.  The module is a square silver SMD component with an antenna.
The module is mounted on a development board with USB-C ports, 40 pins, boot/reset buttons and an RGB LED.  Mine doesn't have a name but there are some well-known ones: ESP32-DevKitC (Espressif), LolinD32 (Wemos), NodeMCU-32S (Ai-Thinker).

I found some interesting articles which provide more illumination on circuitschools and JLCPCB

I have previously blogged about the ESP32-C3 RISC-V processor. This is important as Espressif have announced they aim to focus on RISC-V in the future which implies that their RISC-V  cores will eventually be as powerful as Xtensa cores.

ESP32 uses the RTOS Real-Time Operating System internally to control hardware.  Programmers can use RTOS functions for time-sensitive operations.  Programming is carried out using ESP-IDF, Arduino or PlatformIO IDEs. Programming can be carried out in C, C++ or micropython.

ESP32C3

Intro

ESP32 processors first became available in September 2016 and I first used them in 2019.  The last ones I purchased in 2023 for my LilyGo T-Display buttons which have been well used since then.

ESP-C3-01M-Kit

My interest was re-kindled when I noticed that the ESP32-C3 range runs on Risc-V.  I  purchased a couple of NodeMCU-Series ESP-C3-01m-Kit but they didn't appear to be totally compatible with the Arduino environment and I didn't progress far with them.  I did subsequently try configuring with ESP-IDF which worked better.

ESP32-C3 Super-Mini

Instead I bought some ESP32 C3 Super mini development boards which seem a lot more compatible with Arduino.

Shellminator

I saw a Hackaday article talking about shell/terminal programs which can be quickly implemented.

After a quick Google I decided to try Shellminator on my ESP32-C3.  Breezybox looks even better but requires a more powerful ESP32-S3 system.

I installed the Shellminator library in Arduino IDE and copied the first example program.

The program works fine and gives me a simple shell.  You can then code applications which make use of the shell.  This is a brilliant start for me and encouraged me to purchase an ESP32-S3 for Breezybox.

Assembly Language

It is good to see that a ESP32 Risc-V processor can be used just like an ARM processor.  At present they are somewhat less powerful, but I am pleased and applaud their progress.  Of course I really ought to try out the assembly language.  It is possible to use inline assembly in the Arduino IDE and it appears possible to include separate assembly modules.  However it looks to be rather unreliable/undocumented/error-prone.

The correct answer is to use the official ESP IDE.  ESP-IDF (Espressif IoT Development Framework) is easy to install.  I use it with Windows.  An icon is provided to invoke the command line environment in a Windows shell.  You use python "idf.py" commands to build, flash and monitor programs on an ESP32.   I will talk about ESP-IDF more in another post.

Luckily for me I found an excellent simple tutorial by Prof Smith at York (US) University.
ESP-IDF uses a package called CMake to create a set of build files for you program, taking into account ESP32 variant, compiler, platform you are using.  To use it you set up a structure of files.  You describe what you are building in the two CMakeLists.txt files.

In this simple example we have a C program defines some variables and prints a result.
Variables are passed in RISC-V registers a0 and a1 into the assembler routine, sub3.s.  The code stores the subtraction result in a0 and returns to C.  In C, a0 is the result "r" which can then be printed out.

Although this is a trivial example it gives us the mechanism to write assembler.  Of course, any usage of the processors GPIO pins or other capabilities requires an understanding of hardware, libraries, structures etc.
It is usual to only write the parts that you want/need to in assembler so the tricky parts can be done in C at first.

Outro

It is good to have an easy entry to RISC-V assembler programming using ESP-IDF.  

Jellyfin Music : Synchronised lyrics

 

I noticed a article by HowToGeek which said that you can automatically display lyrics for for music played on Jellyfin.  Until now I only used Jellyfin for music but I thought this was worth investigating.

Google AI is currently in excellent form.  It tells me exactly how to do the installation.

First I added a new Jellyfin library for Music (I only used TV Shows and Movies previously) containing some sample music.  This allows me to select and play albums in the same way as films.

Jellyfin has a plugin, sensibly called Lyrics, which will try to find suitable lyrics and associate them with a song.  You schedule a job in Jellyfin to regularly search and add lyrics.

I used a couple of Olivia Dean songs for testing.  When I played a track a new icon could be seen for Lyrics.  I was overjoyed to see that lyrics were synchronised so the display moved through them line by line as the song progressed.

I linked the Jellyfin music library to my NASmusic database so that I can play anything with Jellyfin.

Both the PC and Ipad standard Jellyfin Music Apps display the lyrics properly.

I found that for some tracks lyrics are displayed but not synched to the music, for a few no lyrics are available.

Lyrics are taken from lrclib.net, a wonderful open source database.  It turns out that if the track duration is not within a couple of seconds of the one in the database it will not sync but plain text is displayed.

If the name / details of a track are incorrect the database may not be able to find the music - I expect this can be fixed my correcting mp3tags.

BEN2 : improve BASIC programming environment

Intro

In April and May 2025 I had great success with BEN2, firstly implementing WOZMON, the Apple I 6502 monitor program written by Steve Wozniak.  I then followed Ben's instructions to implement Bill Gates MS-BASIC. setup input buffering / flow control and add commands to control the LCD display.

My aim for 2026 was to write some BASIC programs and also to tidy up the hardware, transferring a few extra connections from an attached breadboard to a circuit board.

As a first step I wanted to transfer the BEN2 development environment from Windows 10 to my Windows 11 PC, Athena.  Unfortunately MS have screwed up their WSL environment so that serial ports are no longer shared easily with Windows.  In February I went through the pain and suffering of devising a work-around so that I could continue to develop my system.

Hardware Changes

As part of the work to setup MS-BASIC, Ben introduced the idea of copying and pasting programs to the 6502 display.   Although small snippets of text can be easily pasted, it was necessary to setup input buffering and flow control so that complete programs can be copied and these features require hardware changes.

Firstly, for input buffering, I need to connect 6502 /IRQ pin to ACIA /IRQ instead of VIA /IRQ which I was previously using for my MON4 monitor. The 6502 /IRQ signal also required a pull-up resistor.
Secondly, for flow control, I needed to connect FTDI serial convertor CTS wire to VIA PA1 pin.  I also found it useful to attach an LED to CTS signal so that I had a visual indicator when flow control was in use.

New Board

These changes could not be made on the BEN2 board so for a long while I have had an extra breadboard attached to BEN2 which doesn't look good.

I decided to add a new daughter board CAL to BEN2.  This is a small circuit board with interrupt and CTS connections.  I also attached the FTDI serial board and incorporated speaker connections.

Although the circuit is straightforward it took a significant amount of time and effort to : decide board contents; finalise circuit board layout; soldering and testing the board.  On completion I was able to to physically screw CAL onto BEN and dispense with the breadboard.

<Picture of finished, neater, system required>

Add a simple Emulator

Using WOZMON and MSBASIC I quickly became irritated repeatedly typing commands and programs each time I started/loaded the environment.  The ability to copy+paste programs from PC text files is a great improvement but still a nuisance.

BEN2 is attached to Athena through /dev/ttyUSB0.  Usually we start the picocom terminal emulator on ttyUSB0; all text entered into picocom is sent to BEN2 and any text output on the serial port by BEN2 is displayed in picocom.  It is possible to send text directly from a WSL Ubuntu session to /dev/ttyUSB0 with the command echo e.g.  “echo 1000 > /dev/ttyUSB0”

Picocom allows you to write to ttyUSB0 whilst it is running and already has the device open.

We intend to use python to write commands to ttyUSB0 and read the output produced.
Although we can write to ttyUSB0 whilst picocom is running we cannot read the resulting output.
Quite sensibly only one device can read ttyUSB0 at any one time.

Thus we need to set up a simple emulator in python which allows us to send commands to BEN2 and displays any text sent back.  This can partially replace picocom.

Emulator1 simply loops allowing you to enter commands and displaying the output.
It is very effective, not dissimilar to picocom.

Program logic is principally within the python function keyboard_command.
It displays the yellow command prompt and obtains a line of input.
keyboard_command send the input to BEN2 and prints any response received within the serial device timeout (0.3s).
Bytestrings are used on the serial port.  The “bytes” method is used to encode a string and the “decode” method is used on responses to convert output back to a normal string.

LOAD and SAVE programs

BEN2 doesn't have any persistent storage which means we cant save programs.  They can be typed in to Windows Notepad and copied, debugged, listed and copied back but this is messy.  Now we have a simple emulator I can add LOAD and SAVE commands which will use Windows for program storage.

The SAVE command lists out the current BASIC workspace program, captures the text and writes it to a Windows file.

We also have a DIR command which lists available programs.

The LOAD command simply reads a BASIC program file into python and writes it out to the 6502 display, which has the effect of "typing in" the program.

Restrictions

The  emulator isn't very sophisticated, it uses a timeout to check if there is any 6502 output to display on the screen.  This needs to be quite small (say 0.3s) to allow command output to be displayed.  However it doesn't work for long listings or long-running programs so I will use python emulator to load and save programs, but continue with picocom when using the environment.

Outro

After a lull BEN2 has been given a respray.  The hardware has been consolidated and we have a method for loading and saving BASIC programs.  In fact the LOAD mechanism would can also be used to load machine code programs.

Although I haven't done much programming yet, the system is becoming more usable and I can progress other initiatives.

Centre for Computer History

 Intro

I visited the Centre For Computer History (CCH) in Cambridge for the second time in May 26.   It struck me that, for the most part, their history begins in the late 1970s.  To others it is a history of the first days of personal computers for business / entertainment / education.  For me it is a narrative of developments around the time I started using computers. 

Elliot 903

In December 1975, whilst in the sixth form at school, I was lucky enough to go on a week-long computer programming course hosted at Eton College.  They had their own computer, an Elliot 903 like the one shown in the picture, and a computer teacher, who ran our course.  This was the first computer I actually touched / operated.

I believe we learned programming in the Algol language, but I have no recollection of the details.  Once you had written a program (literally on a piece of paper), you typed it in on a teletype terminal, like the one shown, which put saved the program onto a paper tape.

The computer had no storage so you had to load the compiler on the computer by reading in a special (metal) tape, then you loaded your program paper tape so that the compiler could turn it into machine code.  At this stage the program was loaded in the computer memory and you could execute it.  I think you did this by setting switches on the computer panel.  The program would then run and could print any output on the attached teletype machine.  It was amazing.

Acorn System 1

In 1979 or 1980 we used the Acorn System 1, which is featured at CCH, in our practical classes at the computer laboratory in Cambridge.  This is a real piece of history as it was designed by Sophie Wilson (BBC micro designer, ARM founder) who had studied at the computer laboratory after switching from Mathematics in 1978.  She is three months older than me.  I was a year behind her and switched from Mathematics to the two-year computer science tripos in 1978.

Although the Acorn System 1 was based on what has become my favorite processor, the 6502, I struggled terribly with it as I was incapable of reading the "manual" which was printed in Upper Case in small letters with the lines close together.

Sinclair Spectrum

The Sinclair Spectrum (an early prototype board is shown) was announced in 1982 and I was aiming to make a fortune by writing software for it.  The idea was to create a software package to display 3D wire-frame objects which could be manipulated by moving or rotating them on the screen.  I drafted the programs, but unfortunately the computer didn't arrive for nearly a year by which time I had lost interest.  It did have the capability to save / load BASIC programs to cassette but it was somewhat clunky.  I recall that the game we used most was "The Hobbit" a text adventure game with pictures.  The aim was to to get Bilbo and friends through various tricky situations.

Apple I

This is a copy of the original Apple I computer designed, built and programmed by Steve Wozniak who, together with Steve Jobs, founded Apple. There weren't many originals sold, and a real one now sells for maybe $1million.  The primitive Operating System or "Monitor" enabled you to control the system. You could look at or update memory locations, enter programs (in machine code) and execute them.  Amazingly the monitor program, famously called WOZMON is only 250 bytes long.

The Apple 1 CPU is a 6502, which is the same one I use in my Ben Eater processor, BEN2.  It is wonderful that I am able to use that same code, written  in 1976 on my processor, to do the same tasks.

Apple II

The Apple II first went on sale in 1977 and had a great impact on the business computer world.  Until then specialist word processors (Wang) and some strange finance machines (Burroughts Visual Records Computer) were used in admin departments.
Apple II was designed by Steve Wozniak as a general purpose expandable personal computer.  Visicalc was the first "killer" PC app which invented the idea of spreadsheets.  It was simple but loved by Finance people.
I didn't actually use one until 1983 when I went to check out one at a shop in Kensingon where Theo Fennell (now) a famouse jeweller wanted to inventorise his stock.  He had hundreds of little packets containing diamonds.  I'm afraid I recommended an IBM PC as more practical for his database catalog.

BBC Micro

 

The BBC micro or "beeb" was built by Acorn for a major computer literacy education initiative by the BBC in 1981.  Steve Furber mainly designed the hardware and Sophie Wilson wrote BBC Basic.  It was widely used and provided many peoples introduction to computers.

Annette purchased a machine for Alex in about 1995.  It came with a monitor, a touch screen attachment and some educational games.  The touch screen was a wild idea 30 years ago.  It had a grid of infra-red sensors surrounding the screen surface.
Touching the screen with a finger interrupted a vertical and horizontal sensor, giving you a screen position.  I recall writing a BASIC program for an on screen calculator which Alex could use without a keyboard.

Navidrome

 I was thinking about improving my POC app to play my album collection.  It allows you to play tracks or albums in a browser window and so will work on mobile devices.  However it is very rudimentary, ugly and troublesome to use at the moment.

I happened to google "google self-hosted music streaming".  After a little browsing, particularly to find out peoples prefernces, Navidrome appear to be popular and widely used.  They have a demo-site which is great as you can see what you are getting.

It is available as a docker image so I installed it to try out.  Initially I had a technical problem which meant a five minute task took two hours.  The problem was that the navidrome app couldn't use its data directory because it was owned by "root" rather than "dockuser".  It isn't a common issue, I suspect it may be related to my OMV docker configuration.

On completion of the installation I had a nice browser interface but no music.  All I had to do was to add my NASmusic shared folder to the navidrome compose configuration file.  I restarted navidrome and by magic there are all my albums, beautifully presented.

What is really brilliant is that Navidrome uses a standard API "subsonic".  Any mobile client app that supports subsonic can use Navidrome music.  I chose substreamer from the list of relevant apps as it is free, popular and versions are available for both IOS and Android.  It works very well and makes the system look very professional.  I can always install another client if I want something different.

Of course the purpose of the app is to list on amobile phone or iPad outside the home.  To make this happen I just had to direct port 4533 on the router towards the navidrome container.  This also has the benifit of avoiding the "not secure" browser message, I get with external browser access as I haven't fully mastered https yet.

The app has been successfully tested off-site by Annette in Portugal so we are all set for the summer.

Homelab

What is it? 

I hadn't heard (or noticed) the term homelab until now.  Google quickly gave me a sensible definition.  I feel it describes rather well what I do with my leisure time.  I might describe what I do in more general terms as "tinkering with tech to gain understanding or make it more useful to me".  Reading about tech is slightly interesting but trying out what I read and making it work is much more enjoyable and, if successful, satisfying.  In a few cases tech turns out to be geninely useful, ie more than just a gadget/plaything, and this is a bonus.

There is an element of beating, overcoming, controlling, mastering the target of my interests.  I have a desire to achieve, suceed, make something work or understand how it works.

There is an element of doing things that others cant, although this is largely illusory. I am reliant on tutorials, examples and explanations provided by people who know more and understand their subject a lot better than I will.

Hardware

Many homlab geeks seem to like buying routers, servers, switches and NAS, spending lots of money to impress.  Myself, I like to minimise purchases, reusing old stuff as much as possible.  Here is my list.

TP-Link router and switches, Raspberry PI (B, B+, 2, 3, 4, 5), ESP32, SSD (Samsung, WD), Risc V (Nezha, VisonFire2, Mango, M1S)

Software

I wasnt sure what to include for software, so I found a random reddit list with suitable ideas.

Home Assistant, OpenMedia Vault, Docker, Transmission, Jellyfin, pihole, navidrome, owncloud, purevpn, wireguard.

If I include IoT hardware and look at software I use regularly the list would be longer and not much more interesting.  I will add more ideas if become inspired.

 

Owncloud

 Rationale

I need a central point of cloud storage like Google drive, iCloud, Onedrive (yuk). I use them all to some extent but none of them seem to fit all situations.  If I do have suitable cloud storage I am likely to need more space than the commercial offerings allow for free, and I dont want to be tied to a monthly subscription.

OMV NAS provides a good solution but it isn't integrated into Windows, iOS, Android very well.  It is certainly the right solution for media storage; music, pictures and video collections should remain there.
It doesn't seem appropriate for iOS and Android devices, where the underlying storage is hidden.

I would like to be able to choose whether my files are held on the cloud or synced to a local folder and I would like my solution to work outside the home, using SSL.

Consequently my objective is to have a home application, based on PI43/OMV/Docker which looks like iCloud or Googe-drive and allows me to have as much storage as I want.

Selection

There are a variety of Docker based container apps which could be used.  I found it hard to get a feel for them by reading articles or documentation so I tried out a number of solutions.  Docker is perfect for this as I can "spin up" a container and try out the app quickly.  If I dont need to use it further I can remove the container and my system is unchanged, there is no leftover software to remove (or forget).

Before deciding on Owncloud I tried out the following apps.

Syncthing: You can run the app on different computers and it synchronises the chosen folders between them.  I dont really want to have local copies of files on all sharing devices.

Quickshare: Provides you with a central storage file structure.  You can drag and drop groups of files from/to Windows to/from Quickshare.  I dont really want to be using drag&drop all the time.

Picoshare: A very simple app allowing you to drag and drop files to/from a shared folder.  It doesn't have many features.

Chibisafe: The container has Caddy, a reverse proxy manager incorporated so the folders can potentially be used externally.  There is a Chrome add on allowing you to save files directly from your browser.  Copying files using the browser isn't very nice.

Nextcloud: This is a large commercial offering with a free option.  I found it difficult to find appropriate instructions to make a working solution.  The various versions and software used are confusing.

Owncloud: Has a straightforward installation which I can use for testing.  Like iCloud and Gdrive you can manage folder/files in Windows file explorer or iOS files.

In summary, testing owncloud provided me with an excellent way of saving and managing files easily in a home cloud.  Typically files are saved in the cloud rather than synchronised across devices. This fits my initial requirements well.
My testing using SSL / HTTPS didn't go well so I am not sure how useful it will be outside the home.

Installation

My first approach was to use a template provided by OMV.  It is slightly more complicated than previous installs as it creates three containers for owncloud, mariadb and redis (a fast database cache).   I wasn't able to connect to the server this way so I tried some tests using docker cli commands.  After a while I found that I needed to set the "trusted domain" parameter to allow external collections.  I then proceeded with an install following the official documentation.

The skeleton docker compose file has simple builds for three containers owncloud, mariadb and redis.

The default storage volume is on the server SD card,  which needs changing to use the OMV hard drive.
In the ENV file I needed to add my server host name, PI43 and ip address.
Apart from passwords, there were no further changes I needed to make.
I told OMV to start up this environment and I now have three new containers for ownCloud, mariadb and redis.

In Windows when I enter the owncloud URL http://pi43:8080 I see a basic file structure with folders for documents and photos.

I am able to add/change/delete folders and documents using the web client. But this isn't what I want.

I can install the ownCloud Windows client and connect to the owncloud server.  Now if I look at Windows File Explorer I can see my owncloud files.

This is just what I want.  The files are stored in the cloud but accessible locally just where I need them.

Configuration

Next I needed to setup users.  I may open up my own cloud to others but I will be the first and only user initially.  

So far we have used PI43 system SD storage for files which is a VERY bad idea as it would be childsplay to use all available space and kill the server.  Docker compose has a single volume defined for the owncloud system and user data on the SD card with a mount point /mnt/data. When I tried to allocate this to a NAS shared folder, the system started up with an error.
The system starts as the web user "www-data" which doesn't have permissions to use the NAS.

After some failed workaround attempts I decided just put "admin" storage on the NAS and leave owncloud system files on the SD card.  I found that admins directory structure containing "Documents" and "Photos" is located in .../own_files/_data/files/admin/files

So I added a  volume which maps a NAS shared folder to .../own_files/_data/files/admin/files/MyFiles.  

I also setup a user john whose files are located on the NAS shared folder.

Once this is complete I can configure user john in the owncloud windows client and I can see owncloud "MyFiles" for john in Windows File Explorer.  This is perfect, it works just like Gdrive/iCloud

On the iPad I install the owncloud app and use it to configure the user john.
Now I can access MyFiles in the Files app.

For Android I manage files directly using the Android owncloud app.

HTTP vs HTTPS

So far I have accessed owncloud using HTTP.  This works fine if I am using it at home but doesn't work outside my local network.  In itself it is very useful as nearly all my work which requires storage is done at home.

Obviously cloud storage typically works across the internet and it not safe or sensible to use unencrypted HTTP access from outside the home.
I do have encrypted HTTPS access at home for my web-site but my experiments with setting up https within docker and owncloud have not yet been totally successful so I will leave those considerations for another time.

I am happy that I have something useful here; owncloud will be a great help in organising my storage across devices which is becoming lost and fragmented.

I do have wireguard setup on my mobile devices, it only takes a second to setup a VPN connection to home.  This allows me to use owncloud without hindrance.  Other users / devices would need to be setup to give other people access.

VF2L: Nginx

 Intro

Previously I have found that Docker works well on VF2L even though the server is not very powerful.  VF2L doesn't include Docker-Compose and it was educational for me to use command line tools.  There are limited pre-packaged images available for risc-v, but this isn't an issue as Alpine, Debian or Ubuntu can be used as a basis for installing software packages.

VF2L is convenient to use as a testing ground and I decided to try out nginx here even though I could use Docker on RPI or Windows where more docker capabilities are available (compose, desktop, OMV etc).

Proxy Servers

I don't have a good understanding of Proxy Servers.  Originally they were used so that a number of local clients could share an Internet IP address, more recently this has been replace by NAT (Network Address Translation) but proxy servers are still used by some to cache pages, filter available sites and for authentication.  This type of proxy is known as a forward proxy server. In contrast reverse proxy servers accept requests from the Internet and pass them to appropriate web / application servers for processing.  They protect internal servers from malicious users and provide SSL certificates and load balancing.  The title "proxy server" is used for either forward or reverse proxies, which can be confusing.

Nginx

Nginx is the leading proxy server used currently.  It provides a web server, reverse proxy, load balancing and caching. I noticed that it has an official Docker riscv64 image and decided to try it out on VF2L.
It was as simple as typing a docker run command to download the software and spin up a container.  I can then immediately access the nginx placeholder home page.  It only took a minute!

The nginx container runs on a stripped down debian image (126MB) and I needed to install "nano" so that I could create my own test home page in /usr/share/nginx/html/index.html.

After a little experimentation I configured a more useful container.  This has two volumes "html" for  webpages and "conf.d" for the nginx configuration file.  As these volumes are provided from VF2L I can conveniently edit the files on them outside the container then reload nginx in the container to update the configuration.
I also prefer to alpine linux (52MB) as a base over Debian for Docker servers.
As VF2L is a test server I allocated port 80 to be available for the web server. 

Nginx web server configuration

I started by creating an nginx web server.  This is very simple.  The default.conf file simply says that the server should listen on port 80 and the home page is to be found at /usr/share/nginx/html.
In my browser I enter the VF2L ip address and the container returns the ...nginx/html/index.html webpage from the server.

I added a couple of extra locations.  This provides the ability to setup separate web page mappings.

The URI http://192.168.0.25/second.html maps to ...nginx/html/second.html.
http://192.168.0.25/third/ maps to ...nginx/html/third/index.html.
I found it a little confusing that nginx adds in the location (e.g. /third/ ) to URI but this is useful.

nginx proxy server

Setting up a proxy server is just as easy.  I added a location /pi/ in my server configuration and specified using the proxy_pass argument that requests should be directed to my application server (192.168.0.41).

I also had to add a folder ..../pi/ on my application server containing the web page.

Now the request http://192.168.0.25/proxy/ returns the web page ...html/pi/index.html

nginx is also kind enough to allow you to setup multiple web servers in a single instance. I setup a second VF2L server on port 8080.

A request http://192.168.0.25/proxy/  is directed to "myproxied" and the page /usr/share/nginx/html/proxy/index.html is returned.

External Access

Of course the main purpose of a reverse proxy is to route requests coming from outside the local network.  As a test I setup my router to forward port 8080 to VF2L.  

Now I can enter my external ip address and add "/pi/".  My router sends the request to VF2L which looks at "/pi/" and forwards it to //PI41 which returns the web page html/pi/index.html.

Outro

This experimentation with nginx gives me some basic practical understanding of reverse proxies.  They are particularly useful for docker containers as there may be many http(s) ports in use on different containers and nginx can forward requests to the appropriate destination for processing.

There is a load more functionality which I can look at in nginx (SSL in particular) and I haven't created anything of long-term usefulness.  This sets me up well to use nginx with other docker applications, in the first instance this could be an opencloud or nextcloud server.