Owncloud

 Rationale

I need a central point of cloud storage like Google drive, iCloud, Onedrive (yuk). I use them all to some extent but none of them seem to fit all situations.  If I do have suitable cloud storage I am likely to need more space than the commercial offerings allow for free, and I dont want to be tied to a monthly subscription.

OMV NAS provides a good solution but it isn't integrated into Windows, iOS, Android very well.  It is certainly the right solution for media storage; music, pictures and video collections should remain there.
It doesn't seem appropriate for iOS and Android devices, where the underlying storage is hidden.

I would like to be able to choose whether my files are held on the cloud or synced to a local folder and I would like my solution to work outside the home, using SSL.

Consequently my objective is to have a home application, based on PI43/OMV/Docker which looks like iCloud or Googe-drive and allows me to have as much storage as I want.

Selection

There are a variety of Docker based container apps which could be used.  I found it hard to get a feel for them by reading articles or documentation so I tried out a number of solutions.  Docker is perfect for this as I can "spin up" a container and try out the app quickly.  If I dont need to use it further I can remove the container and my system is unchanged, there is no leftover software to remove (or forget).

Before deciding on Owncloud I tried out the following apps.

Syncthing: You can run the app on different computers and it synchronises the chosen folders between them.  I dont really want to have local copies of files on all sharing devices.

Quickshare: Provides you with a central storage file structure.  You can drag and drop groups of files from/to Windows to/from Quickshare.  I dont really want to be using drag&drop all the time.

Picoshare: A very simple app allowing you to drag and drop files to/from a shared folder.  It doesn't have many features.

Chibisafe: The container has Caddy, a reverse proxy manager incorporated so the folders can potentially be used externally.  There is a Chrome add on allowing you to save files directly from your browser.  Copying files using the browser isn't very nice.

Nextcloud: This is a large commercial offering with a free option.  I found it difficult to find appropriate instructions to make a working solution.  The various versions and software used are confusing.

Owncloud: Has a straightforward installation which I can use for testing.  Like iCloud and Gdrive you can manage folder/files in Windows file explorer or iOS files.

In summary, testing owncloud provided me with an excellent way of saving and managing files easily in a home cloud.  Typically files are saved in the cloud rather than synchronised across devices. This fits my initial requirements well.
My testing using SSL / HTTPS didn't go well so I am not sure how useful it will be outside the home.

Installation

My first approach was to use a template provided by OMV.  It is slightly more complicated than previous installs as it creates three containers for owncloud, mariadb and redis (a fast database cache).   I wasn't able to connect to the server this way so I tried some tests using docker cli commands.  After a while I found that I needed to set the "trusted domain" parameter to allow external collections.  I then proceeded with an install following the official documentation.

The skeleton docker compose file has simple builds for three containers owncloud, mariadb and redis.

The default storage volume is on the server SD card,  which needs changing to use the OMV hard drive.
In the ENV file I needed to add my server host name, PI43 and ip address.
Apart from passwords, there were no further changes I needed to make.
I told OMV to start up this environment and I now have three new containers for ownCloud, mariadb and redis.

In Windows when I enter the owncloud URL http://pi43:8080 I see a basic file structure with folders for documents and photos.

I am able to add/change/delete folders and documents using the web client. But this isn't what I want.

I can install the ownCloud Windows client and connect to the owncloud server.  Now if I look at Windows File Explorer I can see my owncloud files.

This is just what I want.  The files are stored in the cloud but accessible locally just where I need them.

Configuration

Next I needed to setup users.  I may open up my own cloud to others but I will be the first and only user initially.  

So far we have used PI43 system SD storage for files which is a VERY bad idea as it would be childsplay to use all available space and kill the server.  Docker compose has a single volume defined for the owncloud system and user data on the SD card with a mount point /mnt/data. When I tried to allocate this to a NAS shared folder, the system started up with an error.
The system starts as the web user "www-data" which doesn't have permissions to use the NAS.

After some failed workaround attempts I decided just put "admin" storage on the NAS and leave owncloud system files on the SD card.  I found that admins directory structure containing "Documents" and "Photos" is located in .../own_files/_data/files/admin/files

So I added a  volume which maps a NAS shared folder to .../own_files/_data/files/admin/files/MyFiles.  

I also setup a user john whose files are located on the NAS shared folder.

Once this is complete I can configure user john in the owncloud windows client and I can see owncloud "MyFiles" for john in Windows File Explorer.  This is perfect, it works just like Gdrive/iCloud

On the iPad I install the owncloud app and use it to configure the user john.
Now I can access MyFiles in the Files app.

For Android I manage files directly using the Android owncloud app.

HTTP vs HTTPS

So far I have accessed owncloud using HTTP.  This works fine if I am using it at home but doesn't work outside my local network.  In itself it is very useful as nearly all my work which requires storage is done at home.

Obviously cloud storage typically works across the internet and it not safe or sensible to use unencrypted HTTP access from outside the home.
I do have encrypted HTTPS access at home for my web-site but my experiments with setting up https within docker and owncloud have not yet been totally successful so I will leave those considerations for another time.

I am happy that I have something useful here; owncloud will be a great help in organising my storage across devices which is becoming lost and fragmented.

VF2L: Nginx

 Intro

Previously I have found that Docker works well on VF2L even though the server is not very powerful.  VF2L doesn't include Docker-Compose and it was educational for me to use command line tools.  There are limited pre-packaged images available for risc-v, but this isn't an issue as Alpine, Debian or Ubuntu can be used as a basis for installing software packages.

VF2L is convenient to use as a testing ground and I decided to try out nginx here even though I could use Docker on RPI or Windows where more docker capabilities are available (compose, desktop, OMV etc).

Proxy Servers

I don't have a good understanding of Proxy Servers.  Originally they were used so that a number of local clients could share an Internet IP address, more recently this has been replace by NAT (Network Address Translation) but proxy servers are still used by some to cache pages, filter available sites and for authentication.  This type of proxy is known as a forward proxy server. In contrast reverse proxy servers accept requests from the Internet and pass them to appropriate web / application servers for processing.  They protect internal servers from malicious users and provide SSL certificates and load balancing.  The title "proxy server" is used for either forward or reverse proxies, which can be confusing.

Nginx

Nginx is the leading proxy server used currently.  It provides a web server, reverse proxy, load balancing and caching. I noticed that it has an official Docker riscv64 image and decided to try it out on VF2L.
It was as simple as typing a docker run command to download the software and spin up a container.  I can then immediately access the nginx placeholder home page.  It only took a minute!

The nginx container runs on a stripped down debian image (126MB) and I needed to install "nano" so that I could create my own test home page in /usr/share/nginx/html/index.html.

After a little experimentation I configured a more useful container.  This has two volumes "html" for  webpages and "conf.d" for the nginx configuration file.  As these volumes are provided from VF2L I can conveniently edit the files on them outside the container then reload nginx in the container to update the configuration.
I also prefer to alpine linux (52MB) as a base over Debian for Docker servers.
As VF2L is a test server I allocated port 80 to be available for the web server. 

Nginx web server configuration

I started by creating an nginx web server.  This is very simple.  The default.conf file simply says that the server should listen on port 80 and the home page is to be found at /usr/share/nginx/html.
In my browser I enter the VF2L ip address and the container returns the ...nginx/html/index.html webpage from the server.

I added a couple of extra locations.  This provides the ability to setup separate web page mappings.

The URI http://192.168.0.25/second.html maps to ...nginx/html/second.html.
http://192.168.0.25/third/ maps to ...nginx/html/third/index.html.
I found it a little confusing that nginx adds in the location (e.g. /third/ ) to URI but this is useful.

nginx proxy server

Setting up a proxy server is just as easy.  I added a location /pi/ in my server configuration and specified using the proxy_pass argument that requests should be directed to my application server (192.168.0.41).

I also had to add a folder ..../pi/ on my application server containing the web page.

Now the request http://192.168.0.25/proxy/ returns the web page ...html/pi/index.html

nginx is also kind enough to allow you to setup multiple web servers in a single instance. I setup a second VF2L server on port 8080.

A request http://192.168.0.25/proxy/  is directed to "myproxied" and the page /usr/share/nginx/html/proxy/index.html is returned.

External Access

Of course the main purpose of a reverse proxy is to route requests coming from outside the local network.  As a test I setup my router to forward port 8080 to VF2L.  

Now I can enter my external ip address and add "/pi/".  My router sends the request to VF2L which looks at "/pi/" and forwards it to //PI41 which returns the web page html/pi/index.html.

Outro

This experimentation with nginx gives me some basic practical understanding of reverse proxies.  They are particularly useful for docker containers as there may be many http(s) ports in use on different containers and nginx can forward requests to the appropriate destination for processing.

There is a load more functionality which I can look at in nginx (SSL in particular) and I haven't created anything of long-term usefulness.  This sets me up well to use nginx with other docker applications, in the first instance this could be an opencloud or nextcloud server.

Python can replace BAT

 Intro

For a long while I have put up with the limitations of Windows antiquated 50 year old batch file system.  With plenty of trial and error I can set up batch files to do simple tasks but it continually frustrating that they have none of the capabilitiy or power of linux shell scripts (which are even older!).  I need some better way to control BEN2 processes and I foolishly thought to try PowerShell as an alternative.  As you would expect it is horrendous, a whole new useless language to help you dig into the cesspit of Windows.

It then occured to me to use python; after all it is a scripting language, windows friendly, easy to use and has an interface to the underlying Operating System.

Serial Port Control

The arses at Microsoft have seen fit to reduce serial port functionality in Windows 11.  Windows 10 shared ports quite happily with WSL programs.  By default Serial ports are now not available within WSL.  You need to install a program called usbipd from github to attach ports to WSL. 

The instructions are not trivial.  You have to identify the correct device in a list that usbipd gives you then "bind" and "attach"  it to wsl.

This isn't something I am capable of doing within a batch file.

Python

Our python script first reads in output from the command "usbipd list" to see if it can find the COM port, which will be labelled as "USB serial converter".
If we find it we capture BUSID and STATE values for that line.

Once we have these values we can attach the device (BUSID=3-3) in this example to WSL.  The bind command only has to be done once so we dont issue it every time in the script.

Outro

Note that the BUSID can change depending on what is attached to the PC.
Also, although device 3-3 is called COM3 in Windows it is called /dev/ttyUSB0 in WSL.
Windows doesn't help me with working this out.

A very irritating situation :(

M C Escher

Intro

As a child / teenager I was fascinated by the drawings of M C Escher.  I had a book of his graphic art which I often enjoyed looking at.  Recently it occured to me that Escher doesn't seem to be well-represented in the the art world despite being well known.

I purchased a Taschen book "M.C. Escher The Graphic Work" for a few pounds on ebay.  It contains the same works I recall with much fondness from fifty years ago: tesselations; infinity representations; plausible but impossible scenes; transformation and evolution of shapes.

Escher and Penrose

Escher spent many years refining his skills creating intricate woodcuts and lithographs, mainly traditional scenes and landscapes, some of which were plausible but unreal.  At one point, whilst at a Mathematics conference, Sir Roger Penrose visited an Escher exhibition and was intrigued by his work. The interest developed into correspondence and a friendship.  Escher was very taken with Penrose's mathematical study of tesselations (patterns covering a surface) and developed them into shapes, figures and animals interlocked and transforming in space.

Penrose narrated a fascinating BBC Scotland documentery about Eschers work, "The Art of the Impossible", in 2009 and participated in an Oxford lecture on the subject. There is also a Sky Arts documentary Journey to Infinity (trailer) narrated by Stephen Fry. It turns out that Eschers interest in repeated surface patterns was stimulated by a visit to the Alahambra in Granada, Spain.  I have also visited and been enthralled by these patterns both at the Alahambra and in Marrakesh, Morocco.  

The Art World

Escher considered himself a graphic artist working in wood and metal.  He had no mathematical or scientific education but had some deep intrisic understanding and the ability to translate these ideas into a visual form with fantastic technical skill.
He has been completely overlooked or ignored by the art world with practically no works on display or public exhibitions.  His works certainly qualify as good art in my understanding: great technical skill; the ability to communicate significant ideas; thought provoking concepts. There are a number of 20th Century "geometric" art movements during the period he was working as well as surrealism and its derivitives.

I think he was insufficiently "arty" and certainly didn't promote himself as unique visionary whose work you needed to buy.  Perhaps the traditional techniques of woodcut and lithography were not trendy, or the ability to make hundreds of prints from a woodcut lowered their value and interest to collectors.
He was popular with the public during the "hippy" sixties and seventies with work featured on a number of album covers.  I  predict he will become more appreciated in future.

Favorites

I have tried to identify my favorite works but I find it difficult as they cover a number of different subjects and techniques.  For the list below I have chosen those which stick in my mind most clearly.  Images are taken from "M C Escher The Graphic Work", but I found "Kaleidocycles" and "The World of MC Escher" interesting.

Docker on Risc-V

 Intro

My most recent  Risc-V development board is the VisionFive 2 Lite.  It isn't particularly powerful, but is probably similar to a RPI-3.

Interestingly the VF2L image comes with docker included.
I decided it would be useful to see how this compares with my other docker setup PI43+OMV

Images Available

There is an Docker Official Index of 48 riscv64 images as of January 26.  There are various programming languages (Go, Ruby, Perl, python etc), a few OS guests (Debian, Ubuntu, Alpine), some webservers (Apache httpd, tomcat, nginx) and some others mainly technical utilities and a few applications (e.g. nextcloud).

Other examples have been created, usually, they are geared towards multiple architectures and include riscv64.  For example I wanted a container I can SSH into so I googled "docker riscv64 ssh" and found woahbase/alpine-ssh.

This is based on the alpine docker image.  I ran the image to build a container and I was able to ssh in to use it.  This is excellent.

An alpine image is only 7MB size and including SSH only increase it to 14MB.

Docker doesn't create a complete VM, it uses the linux host kernel and syscall interface to provide an environment.

Alpine linux comprises a minimal set of utilities to make it useful, so it combines well with docker and is widely used.

Add applications to an image

Previously on my PI43:OMV server I have used docker compose to create images.  I have pasted in configurations and used compose to build an image and run the container.  I dont have docker compose or "standard" images on VF2L so I need to think and learn a little bit more about docker.

For my images / containers I start off from a minimal Alpine Image. I run this image as a container and I can access the container through a shell, enabling me to install software, create files, bring up Services etc.  

My first experiment was to create an Alpine lighttpd container.  I started by running a "raw" alpine:3.23 image.

I needed to add the openrc package plus a couple of extra commands allowing me to start services (this is a wrinkle in alpine setup).

I could then add the lighttpd package and start the web server on port 80.

In a web browser I can see the lighttpd installation placeholder page.

Create a lighttpd image

Now I know how to configure a container running a lighttpd webserver I need to create a new image including lighttpd which I can start up as a web server.

To do this I create a "Dockerfile" which contains commands to execute whilst building the image.  These can include software installation using the Alpine "apk add" command and bringing up Services.  A working image includes all necessary software (executables, configuration files, working storage etc).  Based on the interactive setup I used the Dockerfile as shown here to build a "alpine-lighttpd" image for my webserver.

 

There is a small downside which needs to be dealt with; the raw image has no persistent storage.  The container uses temporary files for all the storage it needs and this is deleted when the container is removed.  So if I added some html files to the container they would be lost when I removed the container.

The answer is to create one or more volumes to contain our data.  For my lighttpd web server the home page will be stored at /var/www/localhost/htdocs/index.html.  with the command "docker create volume VarWwwLocalhost" I created a volume.  I then mapped the volume to the home page directory in my docker run command which starts the container.

Now I could add index.html and other web pages within the container and they remain available each time the container is loaded.

In fact I can also access the volume from VF2L outside the docker container so I can look at or edit files from VF2L before starting the container.  My first effort is shown below.

A vital feature of docker is that it allows you to redirect ports.  When I run the lighttpd container I specify that I want to map VF2L port 8080 to my container port 80.  Then any URL which specifies port 8080 is directed to the container port 80.

Outro

It is very useful to be able to create my own images or amend standard images for my requirements.  Docker is an excellent tool for making these images and running the associated containers.  VF2L has proved itself capable of running docker and docker containers.  It is wonderful to separate the host OS linux from the image so that I am in total control of the container.  VF2L runs StarFive linux but my images are built on alpine, but they could be built on Debian or Ubuntu.

So far I have built containers for lighttpd (described above) and for ssh access so that I can ssh into a test alpine server.  I have also quickly started up nginx and nextcloud containers to experiment with them.  I am quite amazed at how quick and easy it can be to start a complete application.  If there is an appropriate image it is much simpler than adding linux packages to the operating system and configuring them.

Is Google God?

 

PI43 : OMV : Docker

Intro

A key reason for choosing OMV for my NAS is that it also provides an environment for docker containers.  Although docker is widely used I know little about it and it seems likely that I can benefit from using it on my new server.

I searched for suitable application which have been packaged for docker which I might find useful.  My initial list included Jellyfin (a media player), pihole (an ad blocker) and opencloud/nextcloud (dropbox replacement).

Installation

There are lots of docker install tutorials.  I needed to pick one which installs docker using OMV7.  I found a good youtube video from db tech

The first step is to install omv-extras, which gives you a list of plugins that can be added into OMV.  In addition to docker, I was directed to add OMV scripts and docker-compose "extras".

Next I setup three OMV shared folders for compose, backup and data.  I had a problem setting up the compose share, with OMV repeatedly complaining it couldn't do it.  After some time I discovered that my drive is formatted with NTFS as it was formatted for Windows storage. I moved the docker compose share to the linux OS SD card and the system was happy again.

With the pre-requisites complete I could click on the "install docker" button to create my system.  Finally I setup a user, dockuser, which should be able to to manage my system.

Although we used OMV to install docker and will use the it to create containers we do have access to docker through the command line so OMV doesn't get in our way of using any docker capability.

Overall this was a very straightforward install using OMV, with very little technical thought involved.

Sample Application 

The tutorial I followed used a simple web app docuwiki as an example to get started.  This was very helpful though not useful.  It shows that you start by cut-and-pasting the container definition into docker compose, defining disk storage, allocating ports and setting userids.  You then "spin up" the container; software is downloaded, configured and the app starts.  It is wonderfully straightforward and the app is independent of the local system and any complications or dependencies related to it.  The one constraint is that containers are architecture dependant.  Many / most images seem to have an arm64 version which is what my RPI requires.

 Bittorrent

For a proper test I setup the Transmission bittorrent server.  I can use this to replace an old version I have on an old server.  

Step 1 is to create a container definition and copy the docker compose definition from the linuxserver site.
I created a Samba shared folder for transmission files in OMV.
I then amended the docker compose definition for my own UID, GID, Timezone and my storage volume.  I didn't need to change any of the port numbers.

I then started the transmission container. As usual the software is downloaded, configured and then initialised.  

PI43 port 9091 allows you to see the transmission web interface so that you can add torrents to download files.

As the downloads are saved to a Samba I can access and copy them to any device.

In fact I dont need to use the web interface as I have a windows Transmission client which I use for managing torrents.

On the new transmission system I now have lots of space for downloads.  It turns out that my downloads can be 5-10 times faster than the old system, upto 10MBps.

Pi-hole

Pi-hole is a popular ad-blocker.  I have tried it before but it didn't seem to work well for me.  Setting it up in a docker container enables me to start and stop it at will.  As usual I pasted a default configuration into docker-compose.  In this case I had to make a small change to set the HTTP port to 8081.

The container comes up cleanly and I can signon to the web interface to see what is happening.

Initially pi-hole doesn't do any work.  I tested it by pointing the iPad at PI43 for its DNS and the iPad continued to work ok.  I then changed the router so it pointed to PI43 for its DNS.  Now all devices use pi-hole.

It works well, it doesn't interfere with normal browser operation and shows blank spaces instead of adverts in web pages.  If I want I can take down pihole and devices will use google DNS (8.8.8.8) instead.

Jellyfin

I am most interested in setting up Jellyfin as a media server.  It is widely used and liked.  The configuration is very simple, I just need to set up a shared drive for movies and tvseries.

Once the container had been started I could sign on to the admin interface on browser port 8096.

There is an admin user to setup and confirmation is required for movies and tvseries as the initial folders.

Installing the Jellyfin client on a PC is easy.  For some reason Samsung make it slightly complicated to install the app on the TV but I googled a solution.

Jellyfin provides a very professional interface, as good as Prime or Netflix.  It finds a thumbnail image for shows and is pretty good at discovering descriptions about individual episodes to help you choose what to watch.

I can add shows easily by copying them across to the Jellyfin shared folder.

I will call this my killer app, it makes docker worthwhile.