Tuesday, April 24, 2018

2018-04-24: Why we need multiple web archives: the case of blog.reidreport.com

This story started in December, 2017 with Joy-Ann Reid (of MSNBC) apologizing for "insensitive LGBT blog posts" that she wrote on her blog many years ago when she was a morning radio talk show host in Florida.   This apology was, at least in some quarters, (begrudgingly) accepted.   Today's update was news that Reid and her lawyers had in December claimed that either her blog, and/or the Internet Archive's record of the blog had been hacked (Mediaite, The Intercept).  Later today, the Internet Archive issued a blog post to deny the claim that it was hacked, stating:
This past December, Reid’s lawyers contacted us, asking to have archives of the blog (blog.reidreport.com) taken down, stating that “fraudulent” posts were “inserted into legitimate content” in our archives of the blog. Her attorneys stated that they didn’t know if the alleged insertion happened on the original site or with our archives (Reid’s claim regarding the point of manipulation is still unclear to us).
At some point after our correspondence, a robots.txt exclusion request specific to the Wayback Machine was placed on the live blog. That request was automatically recognized and processed by the Wayback Machine and the blog archives were excluded, unbeknownst to us (the process is fully automated). The robots.txt exclusion from the web archive remains automatically in effect due to the presence of the request on the live blog.   
Checking the Internet Archive for robots.txt, we can see that on 2018-02-16 blog.reidreport.com had a standard robots.txt page that blocked the admin section of WordPress, but by 2018-02-21 they had a version that blocked all robots, and as of today (2018-04-24) they had a version that specifically blocked only the Internet Archive's crawler ("ia_archiver").  As of about 5pm EDT, the robots.txt file had been removed (probably because of the Internet Archive's blog post calling out the presence of the robots.txt; cf. a similar situation in 2013 with the Conservative Party in the UK), but it may take a while for the Internet Archive to register its absence.

2018-04-25 update: Thanks to Peter Sterne for pointing out that www.blog.reidreport.com/robots.txt still exists, even though blog.reidreport.com/robots.txt does not.  They technically can be two different URLs though the convention is for them to canonicalize to the same URL (which is what the Wayback Machine does).  HTTP session info provided below, but the summary is that robots.txt is still in effect and the need for other web archives is still paramount. 

Until the Internet Archive begins serving blog.reidreport.com again, this is a good time to remind everyone that there are web archives other than the Internet Archive.  The screen shot above shows the Memento Time Travel service, which searches about 26 public web archives.  In this case, it found mementos (i.e., captures of web pages) in five different web archives: Archive-It (a subsidiary of the Internet Archive), Bibliotheca Alexandrina (the Egyptian Web Archive), the National Library of Ireland, the archive.is on-demand archiving service, and the Library of Congress.  For a machine readable service, below I list the TimeMap (list of mementos) generated by our MemGator service; the details aren't important but it is the source of the URLs that will appear next.  

Beginning with the original tweets by @Jamie_Maz (2017-11-30 thread, 2018-04-18 thread), I scanned through the screen shots (no URLs were given) and looked for screen shots that had definitive datetimes (most images did not have them).  The datetimes are (with ones for which we have evidence in bold, and the ones that we inferred by matching text are maked with "(inferred)"):

2006-01-20 (inferred)
2006-06-13 (inferred)
(update: because of canonicalization errors, some of the URLs are not being excluded; see below)

Most of those dates are pretty early in web archiving times, when the Internet Archive was the only archive commonly available, and many (all?) of the mementos in other web archives were surely originally crawled by the Internet Archive, even if on a contract basis (e.g., for the Library of Congress).  Nonetheless, with multiple copies geographically and administratively dispersed throughout the globe, an adversary would have had to hack multiple web archives and alter their contents (cf. lockss.org), or have hacked the original site (blog.reidreport.com) approximately 12 years ago for adulterated pages to have been hosted at all the different web archives.  While both scenarios are technically possible, they are extraordinarily unlikely.  

While we don't know the totality of the hacking claims, we can offer three archived web pages, hosted at the Library of Congress web archive (webarchive.loc.gov), that corroborate at least some of the claims @Jamie_Maz.


Evidence for this tweet can be found at (approximately midway): http://webarchive.loc.gov/all/20060125004941/http://blog.reidreport.com/ 

2018-04-25 update: the above image is for the left-hand image in the tweet; the right-hand image in the tweet can be found about 1/3 of the way down at: https://web.archive.org/web/20070222030051/https://blog.reidreport.com/labels/Tim%20Hardaway.html


Evidence for this tweet can be found at (approximately 2/3 down): http://webarchive.loc.gov/all/20060608144033/http://blog.reidreport.com/


I'm not sure this evidence maps directly to one of tweets, but it fits the general theme of anti-Charlie Crist: http://webarchive.loc.gov/all/20060615134635/http://blog.reidreport.com/

This memento also exists at archive.is; it is a copy of the Internet Archive's copy but it is not blocked by robots.txt because it is in another archive: http://archive.is/20060615134635/http://blog.reidreport.com/


Evidence for this tweet can be found at (approximately midway): http://webarchive.loc.gov/all/20061010125903/http://blog.reidreport.com/


Evidence for this tweet can be found at (approximately 1/3 down): http://webarchive.loc.gov/all/20081018020856/http://blog.reidreport.com/ 

In summary, of the many examples that @Jamie_Maz provides, I can find five copies in the Library of Congress's web archive.  These crawls were probably performed on behalf of the Library of Congress by the Internet Archive (for election-based coverage); even though there are many different (and independent) web archives now, in 2006 the Internet Archive was pretty much the only game in town.  Even though these mementos are not independent observations, there is no plausible scenario for these copies to have been hacked in multiple web archives or at the original blog 10+ years ago.  There may be additional evidence in the other web archives, but I haven't exhaustively searched them.

We don't know the full details of what Reid's lawyers alleged, so perhaps there are details that we don't know.  But the analysis from the Internet Archive crawl engineers, plus evidence in separate web archives suggest that the claim has no merit.

The case of blog.reidreport.com is another example of why we need multiple web archives.  


Thanks to Prof. Michele Weigle and John Berlin for bringing this issue to my attention and uncovering some of the examples.   

Memento TimeMap for blog.reidreport.com:

2018-04-25 update: As noted above, Peter Sterne brought to my attention that the non-standard URL of www.blog.reidreport.com/robots.txt still exists (and is blocking "ia_archiver") even though the more standard blog.reidreport.com/robots.txt is 404. 

Another 2018-04-25 update: The NYT has covered the story ("MSNBC Host Joy Reid Blames Hackers for Anti-Gay Blog Posts, but Questions Mount"), and there was an interview with Reid's computer security expert ("Should We Believe Joy Reid’s Blog Was Hacked? This Security Consultant Says We Should"), Jonathon Nichols.  

 I embed a statement from Nichols (released by Erik Wemple), and a tweet from Nichols clarifying that they were not suggesting that Wayback Machine's mementos were hacked, but rather the hacked blog was crawled by the Internet Archive.  

This is where it's important to note that there maybe a discrepancy between the posts that Nichols is concerned with and those that @Jamie_Maz surfaced.  There is (semi-)independent evidence of @Jamie_Maz's pages, with the ultimate implication that for those pages to have been the result of a hack, blog.reidreport.com would have had to been hacked as many as 12 years ago -- and for nobody to have noticed at the time.  

Reid (& Nichols) could always unblock the Internet Archive and share the evidence of the hack. 

Yet another 2018-04-25 update: Apparently there are some holes in the http vs. https canonicalization wrt robots.txt blockage, allowing some of posts to surface.  Here's an example (via @YanceyMc):

Also, @wvualphasoldier deleted his tweets then protected his account, so that's the reason the above embed no longer formats correctly. (2018-04-26 update: @wvualphasoldier has now unprotected his account, but his earlier tweets about Joy Reid are still deleted.)

Yet, Yet Another 2018-04-25 update:

Thanks to Prof. Weigle and Mat Kelly for providing examples of some of the URLs that are slipping through the robots.txt exclusion.

Here's one: https://web.archive.org/web/20060805055643/https://blog.reidreport.com

and another: https://web.archive.org/web/20050728132003/https://blog.reidreport.com:443/

Which has the following information that I thought I saw in the original @Jamie_Maz tweets but now I can't find it, so perhaps I'm misremembering.  It certainly fits the overall theme.  Edit: it's in this post:

2018-04-26 update: I saw this last night but I'm adding it now.  In Erik Wemple's article "Is MSNBC’s Joy Reid the victim of malicious ‘screenshot manipulation’?", he links to a PDFs to Google and the Internet Archive from Reid's lawyers.  The letters do not have the attachments, but from the text of the letters I was able to infer some of the references they discuss, such as "...making sexual innuendos about Senator Orrin Hatch..." and "Things people say when they're on the Fox News Channel".  Those two quotes can be found at both:


Picking this apart further, the post "Things people say when they're on the Fox News Channel" is one of about 12 posts Reid made on January 10, 2006.  The above mementos at LC and IA were archived on January 11, 2006 (specifically, 2006-01-11T22:17:38Z).  Assuming the timestamp of "11:28" is EST (Reid was in Florida at the time), the difference between EST and GMT is 5 hours.  The interval between the posting time and the archiving time is pretty small, less than 30 hours:
2006-01-11T22:17:38Z archive time
2006-01-10T16:28:00Z posting time (I've assumed "00" for seconds)
29 hours 49 minutes 38 seconds, or 1 day, 5 hours, 49 minutes, 38 seconds.

Call it 30 hours.  It's possible that the blog's timezone was GMT, so the interval would become 35 hours.  It's also possible the timezone was PST (Blogger was a Google service in 2006, so there's a good chance the machines were in CA), then the window shrinks to 27 hours.  But EST and 30 hours is a pretty good guess.

Blogger does allow you to change creation dates on blog posts, so it is possible to go into a blog "today" and author a post that was "created" in 2006.  But the Internet Archive saw what it saw on 2006-01-11T22:17:38Z, so if the blog was hacked and posts were backdated to appear as 2006-01-10, then a hacker would have to have logged into the site and posted content without Reid or her readers noticing.  The archived page shows six posts on January 11, 2006, 12 posts on January 10, and 11 posts on January 09.  Even if we accept the premise that some of those posts are fraudulent, it is clear that Reid was a prolific blogger and regularly interacted with the blog (history note: blogging was very popular before Twitter & Facebook all but replaced it some years later).  I don't know what kind of readership Reid's blog had in 2006, but at the very least she was interacting with it many times per day, and would have had occasion to notice posts that she did not author.

Reid's lawyers also mention the post "Best Love Life EVER: Celebrity Wife-Swap Edition", which is also available at the above linked archived pages.  It purports to be published on January 11, 2006 at 9:23am, the first post (of six) of the day. Repeating the same analysis as above:

2006-01-11T22:17:38Z archive time
2006-01-11T14:23:00Z posting time (I've assumed "00" for seconds)
7 hours 54 minutes 38 seconds

This establishes a boundary of 8 hours between when the allegedly fraudulent post was purportedly made and when the page was archived.  The last post of the day was 4:51pm ("The hypothetical situation room").  Repeating the above analysis again:

2006-01-11T22:17:38Z archive time
2006-01-11T21:51:00Z last posting time (I've assumed "00" for seconds)
0 hours 26 minutes 38 seconds

In other words, the window for the last time that Reid interacted with the blog and when the blog was archived is less than 30 minutes.  It's entirely possible that Reid pressed "publish" and did not look back at the blog, and just moved on to the next task.  It's also possible that an adversary logged into the blog, posted the content before the 2006-01-11T22:17:38Z archival time, and changed the creation date to be earlier in the morning, before the presumably legitimate content was published.  But also keep in mind that such an adversary would not know in advance the time that the Internet Archive would visit the page (IA's "save page now" did not exist in 2006).  

In summary, while we can't rule out an external adversary logging in and inserting fraudulent content right before archiving time, it would take an extraordinary set of circumstances for the content to appear before the page was archived and not be noticed by Reid (or brought to her attention by her readers).

(also, apologies if I've made datetime arithmetic or TZ conversion errors; corrections welcome)

2018-04-27 update: I've deduced why the number of comments for the top-level pages (at least the few that I've looked at) do not match some people's expectations.  In short, the html page and the corresponding javascript page that holds the index for which posts have what number of comments are out of sync.  In one example I looked at, the html page is crawled on 2006-01-11 and the js index is crawled on 2006-02-07.  When the js runs and can't find its post id in the js index, it assumes zero comments and prints the string "comments?".

I'm including a twitter thread here in lieu of a proper write-up. 


  1. The whole blog is still accessible on Wayback Machine right now. Just change the protocol in the link from 'http' to 'https'


    1. The canonicalization hole that allow the "http" --> "https" swap to work has been closed by the IA. The web.archive.org links above will no longer work, so it's back to the webarchive.loc.gov links.

  2. While IA had the canonicalization hole open I should have pushed copies of IA's pages to archive.is. But I've done this with six of the webarchive.loc.gov pages mentioned above just in case:







  3. IA links from my colleague, Dr. Michele Weigle while the IA canonicalization hole was still open last night. Providing them here in case IA opens up again or perhaps they can assist discovery in other archives:

    monday, march 07, 2005
    What's the gayest thing on TV...?,

    saturday, july 23, 2005
    The adventures of Happy Jack,

    http://web.archive.org/web/20051112185718/http://technorati.com:80/tag/gay+celebrities links to the following 3

    * friday, september 16, 2005
    How Renee got her groove back,

    * friday, september 16, 2005
    Hollywood confidential...Whazzupwitu?,

    * thursday, september 22, 2005
    Okay, maybe it was kids...,

    monday, september 26, 2005
    Sorry, but this stuff is just gross...,

    monday, december 12, 2005
    Nobody likes Kathy,

    "Who are all these people going to see Brokeback Mountain? ..."
    tuesday, january 17, 2006
    Pretty cowboys in love (or, last gasp of gay chic?),

    wednesday, february 21, 2007
    Miss Charlie, Miss Charlie,

    Thursday, January 15, 2009
    Charlie!, http://web.archive.org/web/20110309132417/https://blog.reidreport.com/2009/01/charlie.html

  4. Does anyone else get the idea that Joy Reid and/or her lawyers played their cards against actual web archive experts and then found out their cards had BIG letters spelling out Ultracrepidarianism?