Over the course of today we've been made aware of folks port-scanning the
general internet to discover private Matrix servers, looking for publicly
visible room directories, and then trying to join rooms listed in them.
If you are running a Matrix server that is intended to be private, you must correctly
configure your server to not expose its public room list to the general public -
and also ensure that any sensitive rooms are invite-only (especially if the
server is federated with the public Matrix network).
In Synapse, this means ensuring that the following options are set correctly in
your homeserver.yaml:
# If set to 'false', requires authentication to access the server's public rooms
# directory through the client API. Defaults to 'true'.
#
#allow_public_rooms_without_auth: false
# If set to 'false', forbids any other homeserver to fetch the server's public
# rooms directory via federation. Defaults to 'true'.
#
#allow_public_rooms_over_federation: false
For private servers, you will almost certainly want to explicitly set these to
false, meaning that the server's "public" room directory is hidden from the
general internet and wider Matrix network.
You can test whether your room directory is visible to arbitrary Matrix clients
on the general internet by viewing a URL like
https://sandbox.modular.im/_matrix/client/r0/publicRooms (but for your server).
If it gives a "Missing access token" error, you are okay.
You can test whether your room directory is visible to arbitrary Matrix servers
on the general internet by loading Riot (or similar) on another server, and
entering the target server's domain name into the room directory's server
selection box. If you can't see any rooms, then are okay.
Relatedly, please ensure that any sensitive rooms are set to be "invite only"
and room history is not world visible - particularly if your server is
federated, or if it has public registration enabled. This stops random
members of the public peeking into them (let alone joining them).
Relying on security-by-obscurity is a very bad idea: all it takes is for someone
to scan the whole internet for Matrix servers, and then trying to join (say)
#finance on each discovered domain (either by signing up on that
server or by trying to join over federation) to cause problems.
Finally, if you don't want the general public reading your room directory,
please also remember to turn off public registration on your homeserver.
Otherwise even with the changes above, if randoms can sign up on your server
to view & join rooms then all bets are off.
We'll be rethinking the security model of room directories in future (e.g.
whether to default them to being only visible to registered users on the local
server, or whether to replace per-server directories with per-community
directories with finer grained access control, etc) - but until this is sorted,
please heed this advice.
If you have concerns about randoms having managed to discover or join rooms
which should have been private, please contact security@matrix.org.
If you've been waiting all this time to start implementing some of the privacy improvements the team has been making over the last few months, now's the best time to do it. Clients interacting with identity servers or 3rd party identifiers (3PIDs) have some changes to make, and identity servers themselves have a whole new authed API so they can expose terms of service requirements to users.
0.10.1 has been released on the app store. It includes minor improvements and bug fixes like the call issue. Full release descriptions can be found on respective repos: Riot, matrix-ios-kit and matrix-ios-sdk.
This release includes better logging to track app kills in background but it seems that iOS13.2.2 released by Apple yesterday fixes the issue. We are looking for more feedbacks on that topic.
On develop, the app can now use the integrations manager advertised by the homeserver.
We have finished implementing long click on a Room item, to configure notification settings of the room and to be able to leave the room. We can now ignore user (after a report of content only for the moment). The list of ignored users is displayed in the setting. Users can be un-ignored. We are also working on improving performance and improving code structure. As usual, we have also fixed some bugs. A release will be done at the beginning of next week, then we will try to work on the login flow and account creation flow.
The Riot gang landed 1.5.1 which contained the emoji picker (thanks Tulir!).
Additionally they have been working furiously to make some progress against e2ee device cross signing and have just merged the ability to authenticate via DMs. It's behind a labs flag and will only work if both parties are enabled, but this is big step towards our cross signing dreams. Watch this space for more cross signing features over the coming weeks.
Seshat gained the ability to encrypt the event database and index. A PR has landed and is awaiting review. Now your Matrix client can securely store and index your E2EE messages.
I took time to port Quaternion from Qt Quick Controls 1 (deprecated upstream) to Qt Quick Controls 2, the lighter UI widgets kit that Spectral also uses. Most of regressions are fixed and the result is likely to land in the master branch sometime next week. The overall looks will remain the same, just a minor refresh of visuals. Aside from improving performance the porting should help to solve widget scaling issues on multi-monitor configurations.
Aside from that we’ve continued to work on sharding out the database which we’ll put live once we have migrated matrix.org onto new hardware (woo!) and finally we’ve been fixing some bugs affecting event auth rules.
Coming up on the horizon are ephemeral messages (the ability to send messages with a specific ttl), more io perf work and a bit further down the line we’ll dust off our attempts to shard out room processing from the master process, meaning Synapses running in worker mode will have much more CPU headroom.
another Synapse container image, but this one is new: If you had problems with LDAP in the official Synapse image, try this image: https://gitlab.com/famedly/container/synapse-ldap/container_registry. It's based on the official images, but updates the LDAP auth provider to the latest commit of the master branch. The official image comes with the latest version released to pypi.org, which is a bit older. Aside of that change, it's exactly the same, so you can use it as a drop in replacement.
Hey folks, I've released matrix-appservice-irc0.13.1 which fixes a critical bug in 0.13.0 where messages from matrix would crash the bridge. Users brave enough to be running develop do not need to do anything. https://github.com/matrix-org/matrix-appservice-irc/releases/tag/0.13.1. This would only have affected you if you tried to install or update the 0.13 bridge in the last two weeks.
Dept of Ping 🏓
Rank
Hostname
Median MS
1
maunium.net
492.5
2
aryasenna.net
633
3
matrix.vgorcum.com
656
4
room409.xyz
686
5
kolosowscy.pl
717.5
6
uraziel.de
740
7
nltrix.net
757
8
dodsorf.as
777.5
9
neko.dev
885
10
casavant.org
918.5
That's all I know 🏁
See you next week, and be sure to stop by #twim:matrix.org with your updates!
For the last several months the team has been working on tightening up privacy in Matrix, and with the 1.4 release of Synapse and Riot quite a lot has been done in the area. One of the remaining pieces was to release all the specification changes to help other client/server implementations achieve the same goals, and now we've done that.
The Client-Server r0.6.0 and Identity Service r0.3.0 spec releases both cover the privacy improvements added through a number of MSCs in the last few months. Of particular note is that identity servers are now expected to support terms of service endpoints, which requires authentication that clients might need to worry about - check the spec changelogs for details.
The full changelog for the Client-Server r0.6.0 release is:
Breaking Changes
Add id_access_token as a required request parameter to a few
endpoints which require an id_server parameter as part of
MSC2140.
(#2255)
New Endpoints
Add POST /account/3pid/unbind for removing a 3PID from an identity
server.
(#2282)
This morning (06:11 UTC) it became apparent through mails to support@matrix.org that a security researcher was working through the TLS Certificate Transparency logs for *.matrix.org,*.riot.im and *.modular.im to identify and try to access non-public services run by New Vector (the company formed by the original Matrix team, which hosts *.matrix.org on behalf of the Matrix.org Foundation, and develops Riot and runs the https://modular.im hosting service).
Certificate Transparency (CT) is a feature of the TLS ecosystem which lets you see which public certificates have been created and signed by given authorities - intended to help identify and mitigate against malicious certificates. This means that the DNS name of any host with a dedicated public TLS certificate (i.e. not using a wildcard certificate) is visible to the general public.
In practice, this revealed a handful of internal-facing services using dedicated public TLS certificates which were accessible to the general internet - some of which should have been locked to be accessible only from our internal network.
Specifically:
kibana.ap-southeast-1.k8s.i.modular.im - a Kibana deployment for a new experimental Modular cluster which is being set up in SE Asia. The Kibana is in the middle of being deployed, and was exposed without authentication during deployment due to a firewall & config error. However, it is not a production system and carries no production traffic or user data (it was just being used for experimentation for hypothetical geography-specific Modular deployments). We firewalled this off at 07:53 UTC, and are doing analysis to confirm there was no further compromise, and will then rebuild the cluster (having fixed the firewall config error from repeating).
AWX deployments used by our internal Modular platform, which were behind authentication but should not be exposed to the public net.
Various semi-internal dev and testing services which should be IP-locked to our internal network (but are all locked behind authentication too).
Additionally, certain historical Modular homeservers & Riots (from before we switched to using wildcard certs, or where we’ve created a custom LetsEncrypt certificate for the server) are named in the CT logs - thus leaking the server’s name (which is typically public anyway in that server’s matrix IDs if the server is federated).
We’re working through the services whose names were exposed checking for any other issues, but other than the non-production SE Asia Kibana instance we are not aware of problems resulting from this activity.
Meanwhile, we’ll be ensuring that semi-internal services are only exposed on our internal network in future, and that Modular server names are not exposed by CT logs where possible.
TL;DR: You can list all the public non-wildcard TLS certs for a given domain by looking somewhere like https://crt.sh/?q=%25.matrix.org. This lets you find internal-sounding services to try to attack. In practice no production services were compromised, and most of our internal services are correctly firewalled from the public internet. However, we’re reviewing the IP locking for ones in the grey zone (and preventing the bug which caused an experimental Kibana to be exposed without auth).
We’d like to thank Linda Lapinlampi for notifying us about this. We’d also like to remind everyone that we operate a Security Disclosure Policy (SDP) and Hall of Fame at https://matrix.org/security-disclosure-policy/ which is designed to protect innocent users from being hurt by security issues - everyone: please consider disclosing issues responsibly to us as per the SDP.
A couple of weeks ago I shouted here about a project I've been working on named Install Party, which provides tools for provisioning and managing servers for Matrix homeserver install workshops/parties.
Since then, I've been working on improving it, and today it's finally reached v1.0! This version includes configurable DNS and infrastructure providers, the ability to create multiple server in one run, user-defined post-install scripts, as well as codebase cleanups and a better documentation.
Parallelized file transfer: The bridge now has an option to use multiple telegram connections and a streaming connection to the Matrix media repo when copying files. This should make it much faster and use less ram for big files.
Matrix doesn't have native captions, so !tg caption <text> now exists to send the next image or file to telegram with <text> as the caption.
Animated sticker bridging and helm charts were merged into master.
Bridges-in-nodejs-fans, today we have released 0.4.1 of the matrix-appservice-node library. For those not aware (presumably most), this library is a barebones piece of kit that helps you to listen over the AS api for transactions, in a more barebones manner than matrix-appservice-bridge. The changes in this release are a total transformation of the library into Typescript, and updating dependency packages which had gotten out of date.
It's not as cool as it sounds. Basically you put all the titles and links in a config file and whenever someone says e.g "I really like S05E09", it'll give you the name of that episode and a link to it. You can also just mention an episode title and it'll give you the link.
That's right folks Synapse 1.5.0 is here and ready to make your life just a little bit better.
First things first, this release includes a security fix (#6262, below). Administrators are encouraged to upgrade as soon as possible.
Aside from that, the main thing you'll notice in 1.5.0 is a massive performance improvement to the room directory, which means that servers with large directories to scan will return much more quickly. This is especially true for matrix.org but all servers will benefit.
Another key win is finally fixing some bugs in the sqlite -> postgres migrator script. Sqlite mode is there strictly for testing purposes and should never be used in a production setting let alone a federating homeserver. So if you are currently using Sqlite now is the time to migrate. What's more the script is now in CI so it can't easily break in the future (with apologies to anyone bitten by the old script...).
A final final point, we have some preparatory work for e2ee cross signing, the overall feature is not ready for release just yet but we are getting really close. Watch this space.
CAS login now provides a default display name for users if a displayname_attribute is set in the configuration file. (#6114)
Reject all pending invites for a user during deactivation. (#6125)
Add config option to suppress client side resource limit alerting. (#6173)
Bugfixes
Return an HTTP 404 instead of 400 when requesting a filter by ID that is unknown to the server. Thanks to @krombel for contributing this! (#2380)
Fix a bug where users could be invited twice to the same group. (#3436)
Fix /createRoom failing with badly-formatted MXIDs in the invitee list. Thanks to @wener291! (#4088)
Make the synapse_port_db script create the right indexes on a new PostgreSQL database. (#6102, #6178, #6243)
Fix bug when uploading a large file: Synapse responds with M_UNKNOWN while it should be M_TOO_LARGE according to spec. Contributed by Anshul Angaria. (#6109)
Fix user push rules being deleted from a room when it is upgraded. (#6144)
Don't 500 when trying to exchange a revoked 3PID invite. (#6147)
Fix transferring notifications and tags when joining an upgraded room that is new to your server. (#6155)
Fix bug where guest account registration can wedge after restart. (#6161)
Fix monthly active user reaping when reserved users are specified. (#6168)
Fix /federation/v1/state endpoint not supporting newer room versions. (#6170)
Fix bug where we were updating censored events as bytes rather than text, occasionally causing invalid JSON being inserted breaking APIs that attempted to fetch such events. (#6186)
Fix occasional missed updates in the room and user directories. (#6187)
Fix tracing of non-JSON APIs, /media, /key etc. (#6195)
Fix bug where presence would not get timed out correctly if a synchrotron worker is used and restarted. (#6212)
synapse_port_db: Add 2 additional BOOLEAN_COLUMNS to be able to convert from database schema v56. (#6216)
Fix a bug where the Synapse demo script blacklisted ::1 (ipv6 localhost) from receiving federation traffic. (#6229)
Updates to the Docker image
Fix logging getting lost for the docker image. (#6197)
Internal Changes
Update user_filters table to have a unique index, and non-null columns. Thanks to @pik for contributing this. (#1172, #6175, #6184)
Allow devices to be marked as hidden, for use by features such as cross-signing.
This adds a new field with a default value to the devices field in the database,
and so the database upgrade may take a long time depending on how many devices
are in the database. (#5759)
Move lookup-related functions from RoomMemberHandler to IdentityHandler. (#5978)
Improve performance of the public room list directory. (#6019, #6152, #6153, #6154)
Edit header dicts docstrings in SimpleHttpClient to note that str or bytes can be passed as header keys. (#6077)
Add snapcraft packaging information. Contributed by @devec0. (#6084, #6191)
Kill off half-implemented password-reset via sms. (#6101)
Remove get_user_by_req opentracing span and add some tags. (#6108)
It won't have escaped many peoples' notice that the matrix.org homeserver has been struggling over the last few weeks. The main problem is poor disk I/O performance on our database server: for various reasons, it seems that the server can no longer keep up with the demands we're putting on it.
The good news is that we have a plan to sort it out, and we're working on setting up alternative hosting which will be able to handle our traffic for the foreseeable future! Please bear with us over the next couple of weeks while we get new servers set up.
Just released the first version of the Matrix Notepad, a small webapp designed to allow people to collaborate on text files over Matrix. There are a bunch of bugs that I'm still working on fixing, but if you'd like to check it out it's here! I will be adding incremental fixes for various issues, but feel free to add to the issues. :) If you'd like to chat, I'll be checking #matrix-collaboration:kb1rd.net.
This week much of the team's focus has been on performance, both in terms of dealing with matrix.org's immediate woes and some more strategic work on helping Synapse to scale. On the latter front, Erik has been making great progress in supporting multiple Postgres databases.
We've also put out a release candidate for Synapse 1.5.0 with lots of bugfixes and changes under the hood. As ever, help with testing the RC is much appreciated!
matrix-appservice-bridge has had a new release (v0.11.0) which allows developers to disable usage of the stores entirely. This should solve problems facing some bridges which no longer use them.
Two of the Riot pull requests that I twimmed last week, the reaction emoji picker and edit html parser improvements, have been merged into develop. The remaining one (reply rendering improvements) still needs some code and design work.
The Ruby SDK is out with another new version, 1.5.0 this time. Adding an event and a check to let applications track and handle errors that occur in the background listener thread of the client abstraction.
This version also exposes a setter for the open_timeout value on the lower-level API client (and the client abstraction through it), for users that are plagued by slow-to-open TCP connections to their relevant HSes.
As always, feel free to drop by #ruby-matrix-sdk:kittenface.studio if you have questions, comments, corrections, or just want to discuss the project - or use cases - in general.
To make managing maubot plugins possible with Helm, I added a simple standalone plugin executor to maubot. It's basically just a simple script that sets up all the things plugins want (matrix client, config, database, webserver) and passes them to the plugin.
The standalone executor can be used to run any maubot plugin without any changes to the plugin itself, but building good docker images needs to be done separately for each plugin. The supportportal plugin has a standalone docker image and a helm chart using that image.
Dept of Ping 🏓
Rank
Hostname
Median MS
1
matrix.tetraodon.nl
298
2
linuxgl.ch
319.5
3
flip.earth
402
4
matrix.allmende.io
546
5
midov.pl
601
6
tout.im
622
7
matrix.kimani.dev
648.5
8
matrix.vgorcum.com
660
9
uraziel.de
675
10
kif.rocks
711
That's all I know 🏁
See you next week, and be sure to stop by #twim:matrix.org with your updates!
Last week Matrix had a presence at UbuCon Europe and PyCon Ireland. We gave workshops on using Matrix to create bots, and also a session on installing Synapse (see also: Brendan's entry below!)
Tomorrow I'm off to sunny Manchester with Michael from the Ops team. We'll visit OggCamp, where we'll show off Matrix with a fun demo I previewed on Matrix Live a few weeks ago.
Last week we set MSC1219 (key backups), MSC2241 (verification over DMs), and MSC2313 (ban lists) for the spec core team to focus on. Those 3 are rolling on into this week as we didn't get a lot of work done last week :)
In other Spec news, Matthew uncovered a stone tablet describing what would in future be known as "MSC2324":
For a workshop anoa and I did at Ubucon Europe last Sunday on how to install Synapse, I worked on a side project that creates a server, attaches a domain name to it and installs Riot and Caddy on it. Attendees can then SSH into it, follow the instructions to install a Matrix homeserver, and use Riot to register an account on it, log in, and join a federated room with all of the other attendees' homeservers in it.
We tried it out last week-end for the actual workshop and it worked quite well 🙂
The project is called Install Party, and lives at https://github.com/babolivier/install-party, and if you want to chat about it I've just created #install-party:abolivier.bzh 😉
This week we've been thinking about the future and brainstorming on ideas to improve perf for small instances and sparing some cycles for MSC1228. Next week we'll return to improving IO usage on matrix.org.
Synapse 1.4.1 was released, which fixes a small regression in 1.4.0.
On Riot-iOS land, Riot-iOS 0.10.0 has been released on the App Store. We have started a stabilisation sprint. In parallel, we are still polishing the privacy work
RiotX: we are merging waiting PRs and have postponed the release to ensure a maximum of stability and polishing. The release will be done at the beginning of next week and will contains: read marker, camera picker and improved file picker, share to RiotX capability and many bugfixes.
Heya, matrix-3ds-sdk is a new matrix sdk - for the Nintendo 3DS! It is still deep in development, so expect bugs, some things not working properly etc.
Current features include:
logging in
sending text messages to rooms
redacting events from rooms
resolving aliases to room ids
basic framework to get a /sync loop working
If you are into 3DS homebrew development or are interested in helping out making a full client based on this, please contact soru (@sorunome:sorunome.de)! 3ds homebrew dev is pretty new to her, so there are plenty of open questions / debugging help would be great!
I've made a docker image of the develop branch of synapse, and automated it to build daily. It's on my docker hub as mvgorcum/docker-matrix:develop, based on the avhost dockerfile.
Note that there are no checks, the image is simply built from the develop branch of the synapse git repo every night.
I'm building the Synapse 1.4.1 multi-arch docker image. It will be pushed to Docker Hub in a couple of hours.
Dept of Tulir 🇫🇮
A special section this week from the guy with one editor open for everything, tulir:
I haven't done anything on my own projects this week, but I did contribute to a bunch of other projects:
Updated SmsMatrix to the latest matrix-android-sdk to fix outgoing sms duplication bug (https://github.com/tijder/SmsMatrix/pull/60)
Fixed Riot web sending reply fallbacks in edited message content (https://github.com/matrix-org/matrix-react-sdk/pull/3551)
Fixed some things in the Riot web edit html -> markdown parsing (https://github.com/matrix-org/matrix-react-sdk/pull/3552)
Made Riot web reply rendering much nicer and more compact (https://github.com/matrix-org/matrix-react-sdk/pull/3553)
Added full emoji picker for reactions to Riot web (https://github.com/matrix-org/matrix-react-sdk/pull/3554)
The first two are already merged (and SmsMatrix even got a new release on f-droid), the html parsing is waiting for code review and the emoji picker and reply rendering are waiting for design review.
Also, I made a read-only status.matrix.org rss feed room since some people wanted one: #matrix.org-status:maunium.net. I don't remember if I TWIMed these before, but #xkcd:maunium.net and #commitstrip:maunium.net are similar read-only rooms, new xkcds and commitstrips are posted there whenever they come out.
Dept of Ping 🏓
Rank
Hostname
Median MS
1
matrix.tetraodon.nl
340
2
flip.earth
353.5
3
matrix.bn4t.me
397.5
4
midov.pl
435
5
aragon.sh
443
6
finallycoffee.eu
493
7
dodsorf.as
564
8
matrix.vgorcum.com
575.5
9
maunium.net
595
10
room409.xyz
598
That's all I know 🏁
See you next week, and be sure to stop by #twim:matrix.org with your updates!
We've released Synapse 1.4.1 as a small but important bugfix to 1.4.0.
This fixes a regression which crept in with our newly implemented "erase redacted data after N days" feature where some APIs
would fail when hitting erased redactions - anyone on Synapse 1.4.0 will want to upgrade asap.
Fix bug where redacted events were sometimes incorrectly censored in the database, breaking APIs that attempted to fetch such events. (#6185, 5b0e9948)
Matrix.org hit some IO problems earlier this week, while largely a problem with our hosting provider, we’re spending a bit of time to make Synapse more resilient if the same thing were to happen again. This will mean the ability to shard the DB (by table) and spread the load so we are not so dependent on high performance from a given db box. Outside of that we’ve been working on the final polishing of the room directory and getting the sqlite -> Postgres port script into better shape.
Not much for Dendrite this week as anoa is off at Ubucon 2019. But we had a few valuable bugs reported by the community, and a pressing reminder to get Dendrite's Monolith mode in as part of its CI.
I've not got much for this week, but the IRC bridge has been undergoing some serious refactors and changes for a larger release. Should be quite a big one when it lands :)
A PR for riot-web has emerged that adds support for search in E2E encrypted rooms. The PR is utilizing Seshat to perform event indexing and search on riot-desktop. While the PR is missing any sort of UI, it is in a usable state.
RiotX: We have fixed quite a lot of issues during the stabilization sprint. We are now working on Sprint 4: read marker, report content, mark all room read, etc. François is changing the media/file picker and we will also be able to share elements from other apps to RiotX. We will schedule a release soon (tm) (should have happen this week, but has been delayed due to stabilization)
Polyjuice Client, a Matrix client library for Elixir, has a new release. There is now a short tutorial that will teach you how to make a simple echo bot with it.
Then:
Igor, a bot framework for Elixir, has had its first release.
Dept of Ping 🏓
Rank
Hostname
Median MS
1
matrix.tetraodon.nl
323
2
aime.lesmatric.es
361
3
nerdsin.space
370
4
ru-matrix.org
371.5
5
c-base.org
377
6
fachschaften.org
430
7
linuxgl.ch
435
8
secureim.de
479
9
kif.rocks
484
10
aragon.sh
511.5
That's all I know 🏁
See you next week, and be sure to stop by #twim:matrix.org with your updates!
