Dicoado/Technical documentation

This is the public technical documentation page for Dicoado.

Please read this page carefully.

Visibility of this document

This page contains public information, safe to everyone.

In compliance with Wikimedia expectations and guidelines, we only use open source software, so, everyone in the world already know the structure of our system. Most information here can already be guess by malicious users, so, deleting this page, or omitting some sections, will not increase security in the system.

This is known as the Kerckhoffs's principle. In short, omitting technical details would just give a feeling of false security.

Thanks for any improvement in this page! \o/

Team

These are the authorized persons and their role and their #SSH accounts:

• User:DSwissK

  Main volunteer wiki administrator and main project contact.

  vivian: personal Unix account, sudoers

  www-dicoado: access for Filezilla, not sudoers

• Nicolas Nallet

  wikivalley-nicolas - working Unix account, sudoer

• Sébastien Beyou

  wikivalley-sebastien - working Unix account, sudoer

• User:ValerioBoz-WMCH - staff system administrator (main on-request sysadmin contact)

  valerio-bozzolan - volunteer and working Unix account, sudoer

• User:Valerio Bozzolan - volunteer system administrator (when not in working hours)

  valerio-bozzolan //

• rififi - volunteer system administrator, historical great tech contributor

  rififi - personal unix account, sudoers

Special thanks to DSwissK and rififi for all these great community efforts \o/

Server info

Name	wmch-dicoado1
Operating System	Debian GNU/Linux bullseye
LAN IP	192.168.128.35
WAN IP	89.207.237.145
RAM	12GB (probably over-scaled)
vCore(s)	4
Partition 1	/ 16.7 GiB
Partition 2	/var 5.89 GiB
Partition 3	/home 64 GiB
Special mounts	bind: /home/www-dicoado/web-resources mounted on: /var/www/dicoado

When in doubt, check this page, visible to members of Wikimedia CH:

https://members.wikimedia.ch/wiki/Infrastructure/Servers

DNS

At the moment the DNS are is managed from an external provider (Infomaniak), by:

• User:DSwissK (domain owner)
• User:ValerioBoz-WMCH (for extra tech support)

To add additional DNS managers, contact the domain owner.

---

The above DNS information must be kept in sync with this internal WMCH documentation (that can be seen and updated by any WMCH member + Valerio):

https://members.wikimedia.ch/wiki/Infrastructure/Domains#Infomaniak_Dicoado

Maintained by: User:ValerioBoz-WMCH

Editable by: any WMCH member.

Visible to: any WMCH member.

VPN

SSH is blocked from external traffic.

In order to access the server via SSH, you need access in the "WMCH VPN" first.

Contact User:Ilario or User:ValerioBoz-WMCH or - if you are a member of WMCH - see this page:

https://members.wikimedia.ch/wiki/Infrastructure/VPN

SSH

First, be sure to have #VPN credentials.

Then, if you have no credentials, contact the #Team.

To access the server, use your username, and the server's IP in the #VPN's LAN:

ssh your-user@192.168.128.35

IMPORTANT: We strongly encourage all users to only adopt SSH-key-based authentication. Ideally all users should have no password and only use SSH keys. This allows a level of military security, without any possibility to brute-force. Help: wikitech:help:SSH

IMPORTANT: If you have failed a login, do not retry more than 3 times, or you will be banned by fail2ban.

Uploading files with Filezilla

There is only one authorized user allowed to use Filezilla using SFTP.

Here all the public parameters about that:

Protocol	SFTP
Server	192.168.128.35
Username	www-dicoado
Password	Indeed a wiki is not a good place to share a super-secret password. Never share passwords with anyone or with any unauthorized person or a kitten will die right now.

If this user is not allowed to connect, see #Team.

If the user is not allowed to write something, probably the #Hardening script needs a fix.

Log files

See last webserver error log in real-time:

sudo tail -f /var/log/apache2/error.log

See last visits in real-time:

sudo tail -f /var/log/apache2/dicoado.log

IMPORTANT: Never share log info to anyone but team members, since they may contain personal data.

Wiki (fr)

The data of the website fr.dicoado.org (images, cache…) is stored on:

/home/www-dicoado/storage/fr

The code of MediaWiki is in:

/opt/mediawiki/<VERSION>

The configuration is in:

/etc/mediawiki

There is a README.md there with procedures to create a new MediaWiki version and create a new wiki.

Webserver configurations:

/etc/apache2/sites-available/org-dicoado-fr-ssl.conf
/etc/apache2/my-include/dicoado/dicoado-mediawiki.conf

Homepage (static files)

The website dicoado.org has this document root:

/var/www/dicoado/www/httpdocs

Webserver configurations:

/etc/apache2/my-include/dicoado/dicoado-mediawiki.conf

Production vs Demo

Subject	Production	Demo
Home Path	https://dicoado.org/ /var/www/dicoado/www	https://demo.dicoado.org/ /var/www/dicoado.demo/www
Wiki Path DB wiki DB Cargo	https://fr.dicoado.org/ /var/www/dicoado/fr/httpdocs nxxs_dicoado_fr nxxs_dicoado_cargo_fr	https://demo.fr.dicoado.org/ /var/www/dicoado.demo/fr/httpdocs nxxs_dicoado_fr_demo nxxs_dicoado_cargo_fr_demo

Procedure to pull production into demo

1. copy production databases to demo
2. copy /var/www/dicoado/fr/ to /var/www/dicoado.demo/fr/

TODO:

Conclude this script:

/root/dicoado-demo-pull-production.sh

Procedure to publish demo in production

IMPORTANT:

NEVER COPY DEMO DATABASES IN PRODUCTION.

DEMO DATABASES ARE VERY PROBABLY ALWAYS OUTDATED.

Always copy just demo files in production.

Use this procedure if your files in demo are more updated than production.

This involves to run maintenance operations in the production database.

1. do a backup of the database in production
2. stop production services
3. set production in read-only and/or set production offline and/or maintenance page
4. copy /var/www/dicoado.demo to /var/www/dicoado

   Suggested: rsync --archive --delete

5. run maintenance scripts in production

   sudo -u www-data -s /bin/bash

   cd /var/www/dicoado/fr/httpdocs/w/

   php maintenance/update.php

6. on fail:

   rollback databases and files from backup

7. on successs

   start production services again

   unset production from read-only and/or make it online and/or maintenance page

Maintenance

Maintenance scripts should be executed as the user www-data to avoid permissions issues, e.g.

sudo -u www-data php maintenance/showJobs.php

(Request the rights "sudo to www-data" if your Unix user don’t have it.)

At the moment there is not any particular maintenance mode.

Relevant pages:

• mw:Manual:$wgReadOnly

MariaDB

The databases can be accessed and modified in a secure way from command line using standard ways.

Example to see database tables:

sudo mysql nxxs_dicoado_fr
SHOW TABLES;
exit

To backup databases:

sudo mysqldump nxxs_dicoado_fr       > nxxs_dicoado_fr.sql
sudo mysqldump nxxs_dicoado_cargo_fr > nxxs_dicoado_cargo_fr.sql

IMPORTANT: Never expose an SQL dump in a web directory and always restrict file permissions.

To restore a database:

sudo mysql YOUR_DATABASE_NAME < file.sql

IMPORTANT: This will nuke the already-existing database.

Permissions hardening

Ideally, to reduce attack surfaces, all files that are supposed to be read-only by the webserver, should be at least assigned to the user www-dicoado (so are writable during SFTP uploads). Example:

sudo chown www-dicoado: file.txt

Ideally, only files that should be writable by the webserver should be assigned to the webserver user www-data. Example:

sudo chown www-data: file.png

To simplify hardening, there is this script that restores the ideal situation:

sudo /root/dicoado-harden.sh

Be sure to harden the correct thing. Inside the script there is a configuration line. Change that accordingly to your purpose (hardening demo, or hardening production).

Cron jobs

The webserver user has some cron-jobs. To edit them:

sudo -u www-data crontab -e

Example (may be outdated):

# Hourly, at random minute (xx:17)
# Keep just last execution.
 17 *  *   *   *     /var/www/dicoado/fr/cron/hourly.sh

# Daily, at a random hour (23:22)
22 23  *   *   *     /var/www/dicoado/fr/cron/daily.sh

# Monthly, at a random day (2nd) and random time (23:58)
58 23  2   *   *     /var/www/dicoado/fr/cron/monthly.sh

Systemd service

A systemd service is running all the time, so that if you touch a template, it reflects changes immediately and not after an hour.

Example interaction:

systemctl status dicoado-wiki-fr

The related executable file is here:

/opt/wmch-scripts/mediawiki-job.php

Skin Source Code

DicoAdo is based on the skin Foreground but with local modifications.

See the source code of the DicoAdo's skin on Wikimedia GitLab:

https://gitlab.wikimedia.org/repos/wikimedia-ch/dicoado/dicoado-skin

SSL Certificates

DicoAdo relies on Let's Encrypt for certificates, using the Debian package certbot. We use the "webroot" renewal method.

Every certificate can be renewed with one of these (check your case):

certbot certonly --webroot --webroot-path /var/www/html -d YOURDOMAIN
certbot certonly --webroot --webroot-path /var/www/html -d YOURDOMAIN -d www.YOURDOMAIN

The webroot renewal method will create a temporary file here:

/var/www/html/.well-known/acme-challenge/<tempfile>

Your certificate will be released only if this URL matches:

http://YOURDOMAIN/.well-known/acme-challenge/<tempfile>

Note that the webserver needs a reload after every renew.

Mailboxes

The documentation about DicoAdo mailboxes is available here:

https://members.wikimedia.ch/wiki/Infrastructure/Emails#Mailboxes_in_dicoado.org

Maintained by: User:ValerioBoz-WMCH

Editable by: any WMCH member.

Visible to: any WMCH member.

Backups

IMPORTANT: At the moment WMCH have not automatic snapshots. They need to be requested by the volunteer project leader. WMCH has to verify budget accordingly. Contact User:Ilario for budget verification.

IMPORTANT: At the moment WMCH has not offsite backups. They need to be managed by the volunteer project leader. WMCH probably cannot provide this kind of service at the moment.

The volunteer project leader is supposed to be the only one to have off-site backups.

Wikimedia CH can be contacted to activate a single snapshot and eventually restore it (in case, budget to be verified).

Wikimedia CH can be contacted to evaluate automatic daily snapshots (in case, budget to be verified).

See also

• phabricator:tag/dicoado.org - Task board for Dicoado
• phabricator:tag/wmch-infrastructure - Task board for WMCH