Building Debian PHP Packages with GitLab CI and Ansible

We have automated the rebuilding of Debian Stretch PHP7.0 packages and the provisioning of an APT repository to enable easy updates, using GitLab CI / CD and Ansible, because we found that our shared hosting web servers were hitting the default PHP maximum number of file handles, following is a write up of how we did this.

The PHP packages we have built are available at deb.webarch.net, where you can also find instructions for installing them.

The Problem

The problem with the maximum number of file descriptors being reached was found when trying to send emails using SMTP from InvoicePlane via PHP sockets:

A PHP Error was encountered

Severity: Warning

Message: stream_select(): You MUST recompile PHP with a
larger value of FD_SETSIZE. It is set to 1024, but you have
descriptors numbered at least as high as 1056.
--enable-fd-setsize=2048 is recommended, but you may want to
set it to equal the maximum number of open files supported by
your system, in order to avoid seeing this error again at a
later date.

Filename: src/SMTP.php

Line Number: 1124

As a result we could no longer send out invoices, the immediate way we addressed this was to migrate the invoice site to another server which was hosting fewer sites, but that was clearly not the best answer.

The Solution

We didn’t want to rebuild the Debian Stretch PHP7.0 packages once and then find that a security update required us to manually repeat the whole process, we wanted to automate everything in order that it would be very quick and easy to build and provision updated PHP packages within a couple of hours of any security updates being made available. We also wanted a solution that could be used, if necessary, when Debian Buster is released later this year.

The solution we have come up, which takes 90 minutes to run from start to finish, is partly based on an example on the GitLab blog, but, as is generally the case with these things, if you don’t write most of the code yourself you find that you don’t really understand what it all does and if you don’t understand what all the code does then not in a position to debug things and fix them when they go wrong, so we started from scratch, but we found guidance from blog posts like this and wiki pages like this and, of course, had lots of help from StackExchange.

GitLab CI / CD

We have a GitLab server available for the use of our members at git.coop (our co-op is open to clients and investors to join) and we also have it set to spin up Debian Stretch Docker containers via GitLab CI / CD so it made sense to make use of this existing infrastructure.

We configured GitLab CI / CD to install Ansible and write a GPG private key and SSH secret key to the Docker container and then hand over to Ansible to continue the more complex parts of the process.

Ansible

We could have just used GitLab CI / CD and perhaps some Bash scripts to automate everything but, since we love Ansible, it made sense to use the .gitlab-ci file to setup the Docker container up so it could then be passed over to Ansible for the next steps.

Once the Docker container has been bootstrapped the Ansible playbook takes over and sets a number of variables such as the number of sockets we want to build PHP to support and then runs three roles, build, test and deploy.

The first, role builds the packages by downloading and installing the Debian PHP source packages and the libraries needed to recompile them, it then installs these packages and then rebuilds them. Before it does that however it creates a directory for the source code and one for the APT repo layout and it creates the front page for the APT repo and a .htaccess file to configure Apache to correctly serve the APT repo. It uses Reprepro to build a APT repository in the container and it GPG signs the packages. This role does the bulk of the complicated work, if it completes without an error it hands over to the test role.

The second role installs the PHP packages, to test that they install without errors and then runs a Bash script to check that the value of --enable-fd-setsize matches the value specified. If these tests pass then it hands over to the last role.

The third role does the deployment, it copies the repository layout, packages and front page to one of our static shared hosting accounts which is available at deb.webarch.net using SSH (even our cheapest static shared hosting accounts have optional SSH access via a chroot, which makes them ideal for CI / CD integration).


Update: 11th February 2019

The initial set of PHP7.0 Debian Stretch packages that were rebuilt, as documented above, using --enable-fd-setsize=2048 wouldn’t support more than 1024 sockets, it turned out that the header files also needed editing and rebuilding the Apache 2 and OpenSSL packages was also needed plus it was discovered that the number of sockets that could be opened in the Docker container that is used to build and test the packages is limited to 1024 somewhere — when the packages are installed on a Xen virtual server they can then open more than 1024 sockets, however it still doesn’t appear to solve the initial problem with PHP Mailer

ILP website relaunched

Today, at an event in Sheffield, Independent Labour Publications relaunched their redeveloped WordPress website.

Tweet announcing the redeveloped site

Screenshot of the redeveloped ILP website

The new WordPress templates have been painstakingly developed by a long standing partner member of Webarchitects Co-operative, Mina Neilson.

The old site was also running WordPress but it hadn’t been updated for many years. Webarchitects sysadmins migrated the site from the 1&1 server it was hosted on, upgraded it, moved WordPress from the /main/ sub-directory it was in and then passed it over to Mina to work on the templates and other updates.

Once the updates were ready to be made live the process of copying the live site had to be repeated since new content had been added, and the URL’s needed to be fixed to work via HTTPS and we also ensured that all the old URLs would be redirected so people wouldn’t get 404’s when following old links.

The migration and upgrading was made a lot easier thanks to the WordPress command line interface, this is something that all our clients can optionally have access to with SSH access to our shared hosting servers.

Other sites that we host that have had templates developed by Mina include our oldest client The European Services Strategy Unit and William Morris House.


Written by Chris

Third Annual CoTech Hack

In contrast to the first and second gatherings of the Co-operative Technologists network, which were held over three days at the splendid Wortley Hall, the 2018 CoTech Hack was held at Space4 in London, the co-working space setup by Outlandish, on 29th and 30th November 2018.

The two workers from Webarchitects who attended, Chris and Nick, mostly worked on the CoTech website, helping migrate it from a WordPress site with the code hosted on GitHub to a static site, generated using Jekyll, hosted on git.coop, with some nice CI for site deployment.

One of the most memorable highlights of the event was the international video call with two co-operators in Argentina from Cambá, they have a network of tech co-ops much like CoTech, FACTTIC. Despite the numerous (hilarious) technical glitches and the fact that the call was conducted after the evening party had started, it was a great start to what will hopefully be a productive international relationship. FACTTIC are hosting a ICA Discourse server at patio.ica.coop with the aim to develop an international co-operative IT Network.

Another great session was the one on Ansible (a server configuration automation tool), several CoTech co-operatives, including Webarchitects, are making extensive use of Ansible and we all agreed to work together on sharing roles and best practices, the notes were written up on a Etherpad during the meeting and have been copied to the CoTech wiki.

We were very pleased to have had four new Webarchitects membership forms completed at the event. 

The two day hack ended with an agreement that we shouldn’t wait another year before the next gathering so hopefully we will all meet up again in the spring of 2019.


Written by Chris

libreho.st international gathering

A friend from a project started over 15 years ago, pointed me at the Libre Hosters gathering in Amsterdam so, I thought, hey! that sounds interesting. I joined the irc chat room, and before long booked tickets to Amsterdam. Travelling by Eurostar would have been fun, but sadly it was twice the price, and would have took an extra day of travelling, which wasn’t available, so a return on KLM to Schipol was organised.

Travel to Amsterdam was uneventful, apart from temporarily having my passport taken from me on entrance to the Netherlands, and travel across the city using the Metro and Tram was easy. So I headed for initial informal gathering on Friday 09/11/2018 at laglab.org, which is part of the De Binnenpret collective, a building collective housing various cultural and political initiatives. During the evening more people arrived, from different collectives and organisations, and the conversation ranged across various topics. It was surprising ( or perhaps not! ) to find that we had so many comrades in common across Europe and beyond. Subjects for conversation ranged from technical details on deploying services, how to cooperate on sharing services, how to define the community, how to scale, when to decentralise…

The chat went on until after midnight, eventually I gave up and went to sleep around 2am.

Saturday 10th, we headed to techinc.nl, the venue for the next couple of days. Techinc is a hackerspace in ACTA, a large building that used to be a  Dental school, but now housing over 160 artists and RADION, a 24 hour nightclub. More people arrived, and the event started with introductions

Ten groups presented brief introductions of themselves.  They represented quite a variety of organisations, some more established, others nascent, some more active, some more dormant.  It was great to hear about the different organisations and there see what services and processes they were using. 

notes from the meeting where collectives described themselves
notes from the meeting where collectives describe themselves

After the presentation by the people physically at the meeting,  a video conference was held with the French Chatons collective, which has the wonderful slogan

“The collective gathers structures that wish to avoid collect and centralisation of personal data within digital silos like GAFAM (Google, Apple, Facebook, Amazon, Microsoft).


Framasoft, one of the initiators of the Chatons project sees itself not as a tech project, because tech is too complicated! It prefers to see itself as a project to “empower the commons” whilst providing a number of of decentralised hosting services for it’s members. Framasoft is relatively big project, employing a number of paid workers.
It’s difficult to build systems that are truly decentralised, especially when the groups and organisations within them have very different capacities, one of the key issues that was mentioned from Chatons and Framasoft was how to prevent organisations with greater capacity and resources becoming more “centred” within networks.

After the introductions with from the organisations a complicated conversation occurred around definitions of a librehosting organisation and what types of content should or could be carried by a librehoster. This was a quite a difficult discussion, as we tried to get an idea of what would be “allowed speech” and how “federation” such as in protocols such as Matrix or ActivityPub affect hosting providers.  After several hours of back and forth around the subject a consensus started that, rather than defining what was allowed, an agreement around shared  norms and values of a librehosters would be better place to define the network at this point.  

There was then discussion about how groups can interoperate. An initial proposal of each member organisation to share a virtual machine with another peer, for external monitoring of each others networks .  This was quite a neat idea, to build a “monitoring ring”, however it didn’t get immediate traction with everyone. However it did start another discussion about information sharing, and a proposal for an api to publish the information about our services got everyone excited, and quickly a variation Space API was hacked together.   You can see it here https://lab.libreho.st/librehosters/directory/  

Fuelled by a feeling of success and collaboration, the libreho.st was purchased, and quickly the infrastructure required for a domain (DNS, email, and web hosting) was put together, distributed across the participating organisations. 

Day 2 started back again at techinc.nl and had two areas of work

  1. A discussion on governance
  2. practical work on implementing services on top of the domain.  A minimum of a gitlab service, a discourse forum, and a keycloak federated login service. 

The governance discussion was important, but in the end did not result in the production of a formal process.  However there was a clear understanding that face to face meetings would have a priority over discourse, and discourse over matrix/irc chat when it came to any conflict resolution.

So to sum up… 3 days of conversations and hacking, contacts and friends made across Europe, and fine start to grow some collaboration on. 

Written by Kate

Increase in support rates

Webarchitects Co-operative will charge £90 per hour for general technical support work and £120 per hour for development work from 1st December 2018.

The management committee voted to make these changes on 15th November for the key reason that we need to increase income to afford better wages. We currently pay ourselves £10 an hour (before tax and other deductions) and we consider this to be too low (it is however above the current Living Wage for outside London, £9 an hour, but it is below the London rate, £10.55 an hour and it is in line with the TUC policy of £10 an hour minimum wage). We would like the rate to be £15 an hour, but this is not currently affordable.

Examples of technical support (£90/hour) work include:

  • Sysadmin, general GNU/Linux systems administration tasks
  • Installing Free / libre open source content management systems such as WordPress, Drupal etc.
  • Editing HTML / CSS content
  • Manipulating web images, resizing, changing colours, adding transparency
  • Migrating a WordPress site from another hosting service to a hosting account on our servers
  • Configuring our DNS servers to host client domains
  • Applying security updates to servers
  • Checking server logs for application errors
  • Archiving dynamic website as static HTML
  • Helping clients to configure their email clients

Examples of development work (£120/hour) include:

  • Devops, developing, writing and committing code to git repositories for server configuration
  • Writing Ansible roles and playbooks to configure, maintain and update servers
  • Configuring GitLab CI for testing and deployment of code
  • Writing Bash scripts to automate tasks on servers

We generally host our code on git.coop, which is our members GitLab server. Examples hosted there of development for clients include work for Co-operative Press Ltd and CRIN. We try to make as much of our Ansible code open and available to be shared as possible, you can find many Ansible roles on git.coop.

Users with shared hosting and managed virtual servers are not likely to notice this change (for example, WordPress hosting packages and Nextcloud servers). Essential security updates will still be applied as part of these packages. We are increasingly automating the work we do through the use of Ansible, a tool for scripting the provisioning and maintenance of servers, enabling us to to be more efficient — we are able to do more work in less time.

From 1st December 2018 we will be no longer offering fixed reduced rates for support contracts, we are however willing to negotiate reduced, fixed rates for members on a case-by-case basis.

Civil conversation

The Government’s new Civil Society Strategy consultation includes co-operatives – because they’re part of ‘civil society’ in the Westminster mindset. With half-hearted approval, Co-ops UK notes that “where co-ops are going in terms of platforms, social action and worker control is too radical for DCMS [Department of Culture, Media & Sport] just now.

“The Strategy is far more preoccupied with initiatives to help charities to digitise their operations … not what we‘d pitched for. There is some encouraging but also vague stuff around supporting more investment in ‘social tech’ ventures (…) how business can be more responsible and socially good. It doesn’t mention links between ownership and purpose.”

Let’s hope the next administration is more aware of the depth of the co-operative vision, purpose, and reforms needed to promote co-operative ownership.


Written by Jonathan

Her Majesty’s Government snooping update

Challenges continue against the UK government’s mass interception programme, originally revealed by Edward Snowden. The European Court of Human Rights has found it contrary to rights to privacy, and freedom of expression. The new Investigatory Powers Act 2016 (IPA) allows the Secretary of State alone to issue ‘warrants’ for deep data-mining (even by other nations’ governments) of any website and anyone’s online activity, journalists included. Judicial Commissioners may then review the process, possibly later, and a single Investigatory Powers Commissioner has ‘oversight’. These ‘safeguards’, the EU Court says, fall far short of what’s necessary.

Unfortunately, the UK intelligence-sharing set-up with the USA and other allied states was approved by the Court. Why? Because the government apparently disclosed during proceedings a two-page note on intelligence-sharing, off the record, with no heading, no author, and no indication of whether it was a policy or a summary!

Webarchitects are following this with interest. A fairly easy-to-understand explanation is online at Privacy International.


Written by Jonathan

Welcome to the Webarchitects blog

Welcome to the Webarchitects Co-operative blog, the intention is to post articles here, on a regular basis, that are related to the work of the co-operative, news from members of the co-operative and articles that we think will be of interest to members of the co-operative.

This site is set up so that anyone with an account on our members forum can sign on here and comments are only open to members, see our Join Us page for information on how to join our co-operative.

It will probably take us a few weeks to get everything sorted out here, (plugins, themes, categories etc) and we are also using this site as a place to have a play with the new WordPress Gutenberg editor which is due to be the default WordPress editor with WordPress 5.0.