Today we are very happy to be releasing version 0.9 of our shared web hosting framework, this is the first version to be based on Ansible and also the first version for which we have made all the provisioning code public, it represents the embodiment of over 20 years of experience of providing a web site hosting service.
Key improvements over our last generation of shared web hosting servers include:
Switching from Apache mod_php to php-fpm, this has resulted in a far lower resource footprint for Apache when it is serving static resources and as a result significant speed improvements and also the ability to serve content using HTTP/2.
Using YAML for account configuration, this is far more flexible and allows more fine-grained configurations options and allows us to do things we couldn’t do before like automating the adding of additional WordPress sites to existing accounts.
All new accounts are being setup using this code base and we have also started to roll it out for client servers however we don’t plan on upgrading all our existing shared servers until the Christmas / New Year period because the upgrade will involve some hours of downtime for each server and in addition we have a few key features we want to have in place before then, version 1.0 should include:
Client accessible backups, each account on a shared hosting server will have a directory with a months worth of snapshots of files and database backups.
A major rewrite of the provisioning process, at the moment user accounts are configured in parallel, all at the same time, this makes adding users and editing account configurations rather slow, we intend to reorganise the code so that single user accounts can be added and updated without touching other accounts.
Support for the automatic install of additional content management systems, if you have any requests please get in touch, one we are keen to support is Flarum.
From 1 May we have decided to increase our hosting prices by ~ 15% (dependent upon the particular product provided) in order that we can remain sustainable as a business. We last increased our prices on 1 April 2016 and we will endeavor not to increase our prices for a further two years, other than when we register for VAT (20% in the UK) towards the end of this year.
Webarchitects are pleased to be hosting the next gathering of a network of tech co-ops across the UK and beyond. CoTech was formed to grow the interaction and work between co-operators, to nurture an increase in the common wealth (and fun) between them.
The thirty-plus member co-ops are focussed on a wide variety of internet/tech/arts activities, so between them there is a large pool of talents. Working in the spirit of co-operation rather than competition has generated a larger income to them than before CoTech was launched in 2016.
The Spring Gathering venue is DINA, 32 Cambridge Street, Sheffield. This building (originally the city’s first ‘Free School’) was renovated by volunteers in 2016 as a not-for-profit social enterprise. It has disabled access, various rooms, bar and a vegetarian/vegan cafe.
Local housing co-operatives have kindly offered help with accommodation to people travelling to Sheffield. Argentinian co-operators will be among the attendees.
CoTech Spring Gathering runs from Thursday 25th to Friday 26th April 2019, including a Thursday night social evening with DJ Serious Sounds.Full details here and booking is online at Tickets For Good.
We have automated the rebuilding of Debian Stretch PHP7.0 packages and the provisioning of an APT repository to enable easy updates, using GitLab CI / CD and Ansible, because we found that our shared hosting web servers were hitting the default PHP maximum number of file handles, following is a write up of how we did this.
The PHP packages we have built are available at deb.webarch.net, where you can also find instructions for installing them.
The Problem
The problem with the maximum number of file descriptors being reached was found when trying to send emails using SMTP from InvoicePlane via PHP sockets:
A PHP Error was encountered
Severity: Warning
Message: stream_select(): You MUST recompile PHP with a larger value of FD_SETSIZE. It is set to 1024, but you have descriptors numbered at least as high as 1056. --enable-fd-setsize=2048 is recommended, but you may want to set it to equal the maximum number of open files supported by your system, in order to avoid seeing this error again at a later date.
Filename: src/SMTP.php
Line Number: 1124
As a result we could no longer send out invoices, the immediate way we addressed this was to migrate the invoice site to another server which was hosting fewer sites, but that was clearly not the best answer.
The Solution
We didn’t want to rebuild the Debian Stretch PHP7.0 packages once and then find that a security update required us to manually repeat the whole process, we wanted to automate everything in order that it would be very quick and easy to build and provision updated PHP packages within a couple of hours of any security updates being made available. We also wanted a solution that could be used, if necessary, when Debian Buster is released later this year.
The solution we have come up, which takes 90 minutes to run from start to finish, is partly based on an example on the GitLab blog, but, as is generally the case with these things, if you don’t write most of the code yourself you find that you don’t really understand what it all does and if you don’t understand what all the code does then not in a position to debug things and fix them when they go wrong, so we started from scratch, but we found guidance from blog posts like this and wiki pages like this and, of course, had lots of help from StackExchange.
GitLab CI / CD
We have a GitLab server available for the use of our members at git.coop (our co-op is open to clients and investors to join) and we also have it set to spin up Debian Stretch Docker containers via GitLab CI / CD so it made sense to make use of this existing infrastructure.
We could have just used GitLab CI / CD and perhaps some Bash scripts to automate everything but, since we love Ansible, it made sense to use the .gitlab-ci file to setup the Docker container up so it could then be passed over to Ansible for the next steps.
Once the Docker container has been bootstrapped the Ansible playbook takes over and sets a number of variables such as the number of sockets we want to build PHP to support and then runs three roles, build, test and deploy.
The second role installs the PHP packages, to test that they install without errors and then runs a Bash script to check that the value of --enable-fd-setsize matches the value specified. If these tests pass then it hands over to the last role.
The initial set of PHP7.0 Debian Stretch packages that were rebuilt, as documented above, using --enable-fd-setsize=2048 wouldn’t support more than 1024 sockets, it turned out that the header files also needed editing and rebuilding the Apache 2 and OpenSSL packages was also needed plus it was discovered that the number of sockets that could be opened in the Docker container that is used to build and test the packages is limited to 1024 somewhere — when the packages are installed on a Xen virtual server they can then open more than 1024 sockets, however it still doesn’t appear to solve the initial problem with PHP Mailer…
Remember to visit the ILP’s new-look website, launched today at our Future of Progressive Politics event @CircleSheffield. Thanks due to web hosts @webarchcoop and designer Mina Neilson for their invaluable help. https://t.co/euBeHksrOm
The new WordPress templates have been painstakingly developed by a long standing partner member of Webarchitects Co-operative, Mina Neilson.
The old site was also running WordPress but it hadn’t been updated for many years. Webarchitects sysadmins migrated the site from the 1&1 server it was hosted on, upgraded it, moved WordPress from the /main/ sub-directory it was in and then passed it over to Mina to work on the templates and other updates.
Once the updates were ready to be made live the process of copying the live site had to be repeated since new content had been added, and the URL’s needed to be fixed to work via HTTPS and we also ensured that all the old URLs would be redirected so people wouldn’t get 404’s when following old links.
One of the most memorable highlights of the event was the international video call with two co-operators in Argentina from Cambá, they have a network of tech co-ops much like CoTech, FACTTIC. Despite the numerous (hilarious) technical glitches and the fact that the call was conducted after the evening party had started, it was a great start to what will hopefully be a productive international relationship. FACTTIC are hosting a ICA Discourse server at patio.ica.coop with the aim to develop an international co-operative IT Network.
Another great session was the one on Ansible (a server configuration automation tool), several CoTech co-operatives, including Webarchitects, are making extensive use of Ansible and we all agreed to work together on sharing roles and best practices, the notes were written up on a Etherpad during the meeting and have been copied to the CoTech wiki.
The two day hack ended with an agreement that we shouldn’t wait another year before the next gathering so hopefully we will all meet up again in the spring of 2019.
A friend from a project started over 15 years ago, pointed me at the Libre Hosters gathering in Amsterdam so, I thought, hey! that sounds interesting. I joined the irc chat room, and before long booked tickets to Amsterdam. Travelling by Eurostar would have been fun, but sadly it was twice the price, and would have took an extra day of travelling, which wasn’t available, so a return on KLM to Schipol was organised.
Travel to Amsterdam was uneventful, apart from temporarily having my passport taken from me on entrance to the Netherlands, and travel across the city using the Metro and Tram was easy. So I headed for initial informal gathering on Friday 09/11/2018 at laglab.org, which is part of the De Binnenpret collective, a building collective housing various cultural and political initiatives. During the evening more people arrived, from different collectives and organisations, and the conversation ranged across various topics. It was surprising ( or perhaps not! ) to find that we had so many comrades in common across Europe and beyond. Subjects for conversation ranged from technical details on deploying services, how to cooperate on sharing services, how to define the community, how to scale, when to decentralise…
The chat went on until after midnight, eventually I gave up and went to sleep around 2am.
Saturday 10th, we headed to techinc.nl, the venue for the next couple of days. Techinc is a hackerspace in ACTA, a large building that used to be a Dental school, but now housing over 160 artists and RADION, a 24 hour nightclub. More people arrived, and the event started with introductions
Ten groups presented brief introductions of themselves. They represented quite a variety of organisations, some more established, others nascent, some more active, some more dormant. It was great to hear about the different organisations and there see what services and processes they were using.
After the presentation by the people physically at the meeting, a video conference was held with the French Chatons collective, which has the wonderful slogan
“The collective gathers structures that wish to avoid collect and centralisation of personal data within digital silos like GAFAM (Google, Apple, Facebook, Amazon, Microsoft).“
Framasoft, one of the initiators of the Chatons project sees itself not as a tech project, because tech is too complicated! It prefers to see itself as a project to “empower the commons” whilst providing a number of of decentralised hosting services for it’s members. Framasoft is relatively big project, employing a number of paid workers. It’s difficult to build systems that are truly decentralised, especially when the groups and organisations within them have very different capacities, one of the key issues that was mentioned from Chatons and Framasoft was how to prevent organisations with greater capacity and resources becoming more “centred” within networks.
After the introductions with from the organisations a complicated conversation occurred around definitions of a librehosting organisation and what types of content should or could be carried by a librehoster. This was a quite a difficult discussion, as we tried to get an idea of what would be “allowed speech” and how “federation” such as in protocols such as Matrix or ActivityPub affect hosting providers. After several hours of back and forth around the subject a consensus started that, rather than defining what was allowed, an agreement around shared norms and values of a librehosters would be better place to define the network at this point.
There was then discussion about how groups can interoperate. An initial proposal of each member organisation to share a virtual machine with another peer, for external monitoring of each others networks . This was quite a neat idea, to build a “monitoring ring”, however it didn’t get immediate traction with everyone. However it did start another discussion about information sharing, and a proposal for an api to publish the information about our services got everyone excited, and quickly a variation Space API was hacked together. You can see it here https://lab.libreho.st/librehosters/directory/
Fuelled by a feeling of success and collaboration, the libreho.st was purchased, and quickly the infrastructure required for a domain (DNS, email, and web hosting) was put together, distributed across the participating organisations.
Day 2 started back again at techinc.nl and had two areas of work
A discussion on governance
practical work on implementing services on top of the domain. A minimum of a gitlab service, a discourse forum, and a keycloak federated login service.
The governance discussion was important, but in the end did not result in the production of a formal process. However there was a clear understanding that face to face meetings would have a priority over discourse, and discourse over matrix/irc chat when it came to any conflict resolution.
So to sum up… 3 days of conversations and hacking, contacts and friends made across Europe, and fine start to grow some collaboration on.
Webarchitects Co-operative will charge £90 per hour for general technical support work and £120 per hour for development work from 1st December 2018.
The management committee voted to make these changes on 15th November
for the key reason that we need to increase income to afford better
wages. We currently pay ourselves £10 an hour (before tax and other
deductions) and we consider this to be too low (it is however above the
current Living Wage for outside London, £9 an hour, but it is below the London rate, £10.55 an hour and it is in line with the TUC policy of £10 an hour minimum wage). We would like the rate to be £15 an hour, but this is not currently affordable.
Examples of technical support (£90/hour) work include:
Sysadmin, general GNU/Linux systems administration tasks
Installing Free / libre open source content management systems such as WordPress, Drupal etc.
Editing HTML / CSS content
Manipulating web images, resizing, changing colours, adding transparency
Migrating a WordPress site from another hosting service to a hosting account on our servers
Configuring our DNS servers to host client domains
Applying security updates to servers
Checking server logs for application errors
Archiving dynamic website as static HTML
Helping clients to configure their email clients
Examples of development work (£120/hour) include:
Devops, developing, writing and committing code to git repositories for server configuration
Writing Ansible roles and playbooks to configure, maintain and update servers
Configuring GitLab CI for testing and deployment of code
Writing Bash scripts to automate tasks on servers
We generally host our code on git.coop, which is our members GitLab server. Examples hosted there of development for clients include work for Co-operative Press Ltd and CRIN. We try to make as much of our Ansible code open and available to be shared as possible, you can find many Ansible roles on git.coop.
Users with shared hosting and managed virtual servers are not
likely to notice this change (for example, WordPress hosting packages
and Nextcloud servers). Essential security updates will still be applied
as part of these packages. We are increasingly automating the work we
do through the use of Ansible, a tool for scripting the provisioning and
maintenance of servers, enabling us to to be more efficient — we are
able to do more work in less time.
From 1st December 2018 we will be no longer offering fixed
reduced rates for support contracts, we are however willing to negotiate
reduced, fixed rates for members on a case-by-case basis.
The Government’s new Civil Society Strategy consultation includes co-operatives – because they’re part of ‘civil society’ in the Westminster mindset. With half-hearted approval, Co-ops UK notes that “where co-ops are going in terms of platforms, social action and worker control is too radical for DCMS [Department of Culture, Media & Sport] just now.
“The Strategy is far more preoccupied with initiatives to help charities to digitise their operations … not what we‘d pitched for. There is some encouraging but also vague stuff around supporting more investment in ‘social tech’ ventures (…) how business can be more responsible and socially good. It doesn’t mention links between ownership and purpose.”
Let’s hope the next administration is more aware of the depth of the co-operative vision, purpose, and reforms needed to promote co-operative ownership.
Challenges continue against the UK government’s mass interception programme, originally revealed by Edward Snowden. The European Court of Human Rights has found it contrary to rights to privacy, and freedom of expression. The new Investigatory Powers Act 2016 (IPA) allows the Secretary of State alone to issue ‘warrants’ for deep data-mining (even by other nations’ governments) of any website and anyone’s online activity, journalists included. Judicial Commissioners may then review the process, possibly later, and a single Investigatory Powers Commissioner has ‘oversight’. These ‘safeguards’, the EU Court says, fall far short of what’s necessary.
Unfortunately, the UK intelligence-sharing set-up with the USA and other allied states was approved by the Court. Why? Because the government apparently disclosed during proceedings a two-page note on intelligence-sharing, off the record, with no heading, no author, and no indication of whether it was a policy or a summary!
Webarchitects are following this with interest. A fairly easy-to-understand explanation is online at Privacy International.
