It is January 13th, 2021 as I finish writing these initial words. Major updates may get a date stamp next to them.
Bottom line up front
- Someone is sending the directory authorities (and fallback dirs) lots of traffic.
- This causes the dirauths to no longer be able to reliably communicate.
- This means consensuses are no longer reliably produced every hour.
- No new consensus three hours in a row means new connections to v3 onion services stop working because of a bug. Existing connections survive, and no other part of Tor breaks at the three hour mark.
- There is an alpha release for experts who know what they are doing. It is making its way into all supported stable Tor versions.
Please keep these facts in mind:
It is unknown if the traffic hitting the dirauths is maliciously motivated. People keep calling it an attack. I don't think we have the evidence to back that up at this time.
There is no evidence that the traffic overload is actively trying to hurt v3 onions. A similar situation existed last year and onions didn't go down then. Claims that it is "the" government or rival drug markets are not backed up with any evidence that I've seen.
If you have evidence of who is behind this traffic, please let someone know. Tor Project or me (blog comment, an email (listed on About Me), or IRC message)
While I currently work on Tor-related stuff for my job, nothing contained in this post has anything to do with my work. Everything contained in this post is public unclassified knowledge. Opinions expressed, if any, are my own.
Traffic starts hitting dirauths (again)
Roger points out on January 6th that "the overload is back".
It's not OMGWTFBBQ levels of traffic. It's not from one IP nor is it from IPs all over the Internet. One dirauth says it seems to be a poorly written custom Tor client requesting directory information too often.
Three missed consensuses
On January 9th, 10th, and 11th, there are repeated instances of 3 or more consensuses in a row that are not generated.
This is the trigger for v3 onions no longer working. Consensuses are generated every hour and are valid for 3 hours. Most parts of Tor (v2 onions, general circuit building, etc.) do not require a live (currently valid) consensus, but can get by just fine with a recently valid consensus (expired less than 24 hours ago).
January 12th saw a few missed consensuses, but never 3 in a row. No consensus has been missed so far on the 13th.
The bug and its fix
The v3 onion service code was written to require a live consensus, and it didn't need to be (devs are verifying this). The fix for this bug changes the requirement to just a recently valid consensus. It's getting tested as I write these words on January 11th. The fix, or something very similar to it, will be merged and backported in the coming days, at which point it's up to the packagers for your OS or your tor-derived software (e.g. Tor Browser) to notice the update and distribute it to you. I would expect Tor Browser to be updated very quickly. Debian will probably take a day or two. Other distros, I have no idea.
If you're watching tor's logs, the current date includes "January" and "2021", and you see the following message, then you have most likely hit the bug.
Tried for 120 seconds to get a connection to [scrubbed]:6697. Giving up. (waiting for rendezvous desc)
The 6697 is not important. The "waiting for rendezvous desc" is important.
Status of the fix making it into Tor
Primary sources for this section: bug #40237.
- Jan 12th: the fix is merged into 0.4.5.3-rc [blog post]
Upcoming events:
- backports to other supported versions of Tor
- packaging
Fallback dirs getting hit too
On the 11th we notice the fallback dirs are also failing. This is major evidence in my opinion that this is not a purposeful attack on the dirauths. I, and at least two dirauths, think it is most likely a bad custom Tor client implementation that requests directory information too often.
We see the same failing of fallback dirs on the 12th.
This graph shows how the entire network is fielding more directory requests these days. This graph shows more context for where load usually is. The 1.5 Gbit/s the dirauths saw for ~half of 2020 is talked about in this ticket.
An actual attack could disguise itself like this, but if this were an attack, I would expect it to be more consistently effective at preventing consensus generation.
It might be over now
(Last updated 15 Jan 2021)
The 14th saw no missing consensuses. If you had trouble reaching v3 onion services on the 14th, your issue is unrelated to the topic of this blog post.
The 15th so far has seen no missing consensuses.
FAQs
What Tor relays/clients need to be updated?
No relays need to be updated. Tor clients hosting v3 onion services and Tor clients wishing to visit v3 onion services will need to be updated when the fix is released.
How do I update my Tor?
(Last updated 13 Jan 2021)
You don't yet, unless you're willing to compile and use a version of Tor that isn't considered stable yet. If you're willing, then see https://blog.torproject.org/node/1969.
Should we temporarily downgrade to v2 while waiting for a fix?
If absolutely necessary, sure. Please keep in mind v2's issues (briefly described below in the glossary) and be aware that "temporary" probably means ~1 week (crossing my fingers). I personally will just suffer and wait for the fix to be released.
Are v3 onion services fundamentally broken?
No.
Is this really major?
Eh ... yes because of the impact, but no because the fix is easy and will be out quickly.
Glossary and preemptive rebuttals
Directory authorities / dirauths
There are 9 relays operated by highly trusted individuals that decide the state of the network. They decide what relays are a part of the network and what certain network parameters should be set to.
In a way they are a "single" point of failure and make the Tor network "centralized." Decentralizing their role to 100s, 1000s, or every relay would:
require massive fundamental changes to how Tor works. By itself this probably is not a convincing reason to not do it.
open Tor up to new attacks it currently isn't vulnerable to. This should be a bit more convincing. The keyword to Google for most of them is "Sybil".
Having a "single" high quality root of trust is a valuable property that "decentralize all the things!" people do not generally appreciate enough, in my opinion.
You: This v3 onion fix doesn't actually address the root problem: the dirauths weren't able to communicate and create consensuses.
Yes! You are absolutely right that something about how the dirauths work should change such that they can continue communicating with each other even in the presence of malicious (purposefully malicious or not) traffic! This ticket is one idea and a good place to start your research if I say nothing more and you want to research what is being done yourself. They might also update this ticket.
V3 onion service
The new type of onion service. Names are 56 characters long, like tv54samlti22655ohq3oaswm64cwf7ulp6wzkjcvdla2hagqcu7uokid.onion. V2 onions are 16 characters long, like 3g2upl4pq6kufc4m.onion.
V2 onion services use old crypto and old protocols that are obsolete and dangerous now or will be soon. The code for v2 is messy and the protocol is not extensible. V2 onions are vulnerable to harvesting by malicious HSDirs, and these malicious HSDirs exist today (and are removed as soon as they are detected). Support for v2 onion services will be removed from Tor soon. V3 onions are the future despite the current events.
Fallback directory mirror / fallback dir
Tor clients don't usually directly fetch consensus information from a dirauth anymore. There's too many people using Tor and only 9 dirauths. Instead, a large number of relays that are high quality have opted in to also be hardcoded into Tor source code so clients can usually fetch consensus data from them on first run. After first run and successful connection to Tor, clients get this stuff from their guard instead.
Consensus
The dirauths vote on what relays are currently in the network and what certain network parameters should be set to. The "average" of their votes become the consensus. They make a consensus every hour and each is valid for three hours. Clients typically fetch a new consensus every two hours.
You can see information from the current consensus here. Recent consensuses are archived here and older ones archived here.
The repo is on Github here, and in case that turns out to be a lie in the future, the code as of the initial writing of this post is here.
About brainfuck
Brainfuck is an esoteric programming language with only eight commands (i.e. it's meant to be fun/challenging, not useful). Memory is represented as cells on an infinitely long tape and accessed with a single pointer. Typically each cell is a single byte, and by convention "infinitely long" means 30,000 cells. The eight commands are:
>move cell pointer to the right,<move cell pointer to the left,+increment value in current cell,-decrement value in current cell,[enter loop if current cell value is not 0, else jump to end,]end loop,,accept one byte of input and store in current cell,.output byte in current cell.
All other characters in brainfuck source code are comments.
As an example, the following program writes "Hello World!" followed by a newline to stdout.
++++++++[>++++[>++>+++>+++>+<<<<-]>+>+>->>+[<]<-]>>.>---.+++++++..+++.>>.<-.<.++
+.------.--------.>>+.>++.
Thoughts
Writing an interpreter and checking that other people's brainfuck programs run in it was a lot easier than actually learning brainfuck and writing it. That said, I do want to learn it well enough to write some simple things.
There are a couple simple optimizations I'd like to try implementing that are documented in the README. The most obvious is to coalesce repeated instructions. I intend on writing a script to measure the improvement these optimizations have.
Brainfuck resources on the web
This post first appeared on my old blog in April 2020. It is preserved, but maybe not updated, here.
I was training for a half marathon in May 2020. Training three days a week. Increasingly longer runs each time. Ya know. Preparing. Then COVID-19 hit the US. I'm now on week 3 of working from home and the race is
Coincidentally I've been eating so poorly (for so long) that increasing miles weren't decreasing pounds.
Thus far we're still allowed to go outdoors for exercise as long as we maintain social distance. I lack a big firm goal of a half marathon, but don't want to regress back into the comfortable laziness I've been enjoying for far too long. So I've decided to modify my exercise regimen to (at least initially) involve more frequent but shorter runs.
Also I'm going to count calories, which in the past has been the key to actually successfully losing weight. If I don't count, I don't make much progress and regress quickly. Oh and being accountable helps.
So here we go. Time to be accountable to the Internet and improve my life while on "lock down."
This is the weight I've lost. I want to lose ~2 pounds per week, which in the past has been attainable. I want to lose ~30 pounds, so I'm looking at about 13 weeks (91 days) here. This will put me at a BMI of ~22, squarely within the "normal" range of 18.5-24.9.
Here are the runs I've gone on.
What follows are three posts announcing new major releases of BM (Blog Maker), the software I wrote and "maintained" that hosted my old blog, in January 2017 and March 2020. These are preserved here, but not updated. Links will be broken.
BM v3.0.0 is Released
Today I've released a new major version of BM, consisting of about 140 commits! See the changelog for a summary of all the changes, and please report issues at the issue tracker. Here's the important and exciting highlights.
Important
make cannot be called by the user anymore as BM needs to setup the
environment for it. It probably should have never been called by hand, and
hiding the Makefile in v3.0.0 further discourages manual make calls.
Post URLs have changed, but probably won't do so again for a while--if ever--as this is quite rude of me to do. Now post URLs are limited to the first three words of the post title plus the ID. Before it was all words of the title.
Exciting
Permalinks have been added. This was prompted in part by the previous change. If the option for it is enabled (which it is by default), a little permalink will be added in every post's header. Permalinks consist soley of a post's ID, so they will never change so long as you don't manually change a post's ID!
BM can optionally make the source post files available for download by your
readers. If the option is set, not only will /posts/foobar-12345678.html be
generated as usual, but /posts/foobar-12345678.bm will as well, the latter
being an exact copy of the file you edit.
A 404 page has been added. Special webserver configuration is required to get the most out of it. See the wiki.
Thoughts
This release took a very long time. It introduced many backend changes, most concerning a major design change: move as much dependency logic into the Makefile as possible. Before the majority of BM's logic was in three large scripts. Now
- post data is extracted into seperate files (in
meta/) - as little work is done as possible when changes are made (this will need continued improvement)
- later build steps only depend on the post data files that they require
It was an interesting challenge figuring out how to call my bash functions and use bash variables inside the Makefile. The gist of how it works is
- sourcing
globals.shbefore callingmake - calling bash's
set -aat the top of theglobals.shfile - calling
set +aat the end for good measure
This exports all function and variable definitions into the environment that make runs in.
Future
Going forward, I already have a few features in mind for minor releases, even after tackling the huge pile that built up while working on v3.0.0.
The first easy thing I'll likely do is adding a license option, so users can easily license the contents of their blogs.
But another big idea that a Redditor suggested to me is themeing. Right now BM has one look. It's "easy" to change that look, especially if what you want to change is coloring or spacing. But wouldn't it be nice to be able to easily switch between themes with a single command? You could download additional themes, share them, etc. This is a large change that will take some time, and undoubtedly a new major version.
Another big idea that I'm chewing on in my free time is moving all posts and customizable files into a single directory. This would pave the way towards easily allowing you to version control your posts and your configuration options (and maybe even theme). In fact, it's conceivable that BM could even make commits for you. This is another large change that would also take a new major version.
In general, you can see the things I'm thinking about and working on at the issue tracker.
Do you use BM?
Finally, I'd like to ask anyone out there who uses BM and doesn't mind their blog, wiki, whatever being public to please let me know! I'd love to hear about your experience using BM.
BM v4.0.0 is Released
Yesterday I released yet another new major version of BM! The changelog has a summary of changes. As before, please report any issues at the issue tracker.
Important
There are two big changes that should be noted.
Your configuration file needs to move. It used to be in include/bm.conf, but
that directory has been emptied out. Your configuration file now belongs in your
posts directory, posts/bm.conf. BM comes with a script in tools/ to help you
transition from v3 to v4, but really it's as simple as moving your configuration
file. After you've moved it, you may delete the include directory. It should
be empty.
The other major change is themes! Themes allow you to quickly change the look of your website. They can easily be shared as all the important bits and pieces are in one directory per theme. Here's the "terminal" theme that I created and will officially support in addition to the default theme.
For information how how to set your theme, see here. For information about creating your own theme, see here. It's very easy, especially if you start out copy/pasting an already good one.
Other new features
Page signing was added. Now, given a gpg fingerprint, BM will automatically cryptographically sign all output files (even the CSS!) and leave a note in the footer saying so in officially supported themes.
(Ignore the version number, this was added in v4.0.0. I should probably decide something about "in development" versioning...)
If page signing is enabled, then /pubkey.gpg will also be automatically
generated with the public key used for signing.
Licensing your content has been made easier. A new config option,
<span class="createlink"><a href="/ikiwiki.cgi?do=create&from=posts%2Fbm-major-releases&page=LICENSE_TEXT" rel="nofollow">?</a>LICENSE TEXT</span>, was added. The contents of it will be placed verbatim in the
footer of officially supported themes. I have set my <span class="createlink"><a href="/ikiwiki.cgi?do=create&from=posts%2Fbm-major-releases&page=LICENSE_TEXT" rel="nofollow">?</a>LICENSE TEXT</span> to the
following string in order to get the Creative Commons image link you see on my
blog.
<a href='https://creativecommons.org/licenses/by-sa/4.0/'><img
src='https://i.creativecommons.org/l/by-sa/4.0/80x15.png'/></a>
The above produces
Such complicated license text is obviously not necessary.
Future
Some of the next things I want to work on include
- Adding an easier way to modify theme metadata
- Move the selected theme symlink to the post directory in order to...
- Put all the user-specific files (config, post files, theme) in one directory so it can be completely version controlled and swapped in and out.
- An option to exclude a post from the homepage
- An asset directory, which has its contents copied to the build (for images and things that aren't post files but you want to host and make available for download)
To watch my progress or to suggest things, see the issue tracker.
If you're using BM, I would love to hear about it! Please let me know somehow.
BM v5.0.0 is Released
Hey look. This dead project is getting a new major version. Don't count on this
continuing to happen! 
Important
The default/bundled markdown parser is changed from Markdown.pl to
cmark-gfm. While making the change, I sometimes noticed the content of my
pages being rendered differently. Once the change was finally fully made,
however, the content renders the same. I have no idea why it would be
different, nor do I know what I was doing to make it break/unbreak.
Thus, to be cautious, I'm calling this a breaking change. Thus a new major version for BM is required.
The full spec for Github Flavored Markdown is
here. BM bundles cmark-gfm v0.29.0, so
assuming the spec still says it applies to that version at the top, BM should
support everything you read there. I haven't tested anything other than
| tables | tables |
|---|---|
| tables | tables |
I do not expect to update the bundled cmark-gfm with any regularity. I don't
even expect to update BM!
Other new features
Since v4.0.0
A static directory. Put stuff in static/ and it will be copied to
build/static/. Put your resume at static/docs/resume.pdf and link to it
with [my resume](/static/docs/resume.pdf).
RSS feed generation. I think I implemented it poorly. I don't know. I don't use RSS feeds.
This post first appeared on my old blog in December 2019. It is preserved, but maybe not updated, here.
Furthermore, it's unlikely I've figured out how (or bothered to set up a way) to sign all pages.
My hosting provider went out of business.
I didn't get my onion service's keys off the box in time. Stupid. Kept putting it off like an idiot.
I took this opportunity to stop offering a v2 onion service. Now you have to use that that hot v3 goodness. Oh nooooo.
It's at http://tv54samlti22655ohq3oaswm64cwf7ulp6wzkjcvdla2hagqcu7uokid.onion now.
Like you've always been able to (but probably no one has ever done), you can
verify this post was written by me by downloading this page, downloading the
signature of this page by appending .asc to the URL, and using (e.g.) GnuPG.
Oh and hopefully you already have my key or have a good reason to trust that
the key in the footer of my website is mine. I am
B7E105FC4E6D9377F89CBA4C83BCA95294FBBB0A. But the preceeding sentence is
meaningless if you didn't already know that. But now I'm repeating myself. Ugh
trust. Identities.
$ wget -q https://matt.traudt.xyz/posts/yes-my-websites-w6t3nxCA.html
$ wget -q https://matt.traudt.xyz/posts/yes-my-websites-w6t3nxCA.html.asc
$ gpg --verify yes-my-websites-w6t3nxCA.html.asc
gpg: assuming signed data in 'yes-my-websites-w6t3nxCA.html'
gpg: Signature made Thu 19 Dec 2019 07:51:10 PM EST
gpg: using RSA key B7E105FC4E6D9377F89CBA4C83BCA95294FBBB0A
gpg: Good signature from "Matt Traudt <sirmatt@ksu.edu>" [unknown]
gpg: aka "Matt Traudt <matthew.traudt@nrl.navy.mil>" [unknown]
gpg: aka "Matt Traudt <pastly@torproject.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: B7E1 05FC 4E6D 9377 F89C BA4C 83BC A952 94FB BB0A
If you're familiar with PGP you know what can be different from the above without concern. If you're not familiar with PGP, you shouldn't be trusting things because they are "PGP verified."