Public archive for Delving Bitcoin

Meta
1

It would be great to have a public archive of the posts here, so that

  1. if this site goes away, we still have all the content, and
  2. it is searchable by indexers like https://bitcoinsearch.xyz.

As far as I can tell, there are three approaches:

Scraping

Use some kind of crawler (selenium, wget, etc.) to manually create a backup by scraping the site.

  • I think this is not great given how JavaScript heavy Discourse is (i.e. it loads posts incrementally based on cursor position). It’s also brittle - scraping code is often complicated.
  • …but, it can be done by anyone, with no permissions necessary.

API

Use the Discourse API to pull content. This could be done either periodically as a full crawl or incrementally on a continuous basis.

  • I think this is a fine way to go assuming the API offers everything we need and someone can provision API credentials.
  • We may be able to get everything we need out of the list posts endpoint.

DB dump

Get a SQL dump of the database and run some kind of sanitization/export script on it every so often.

  • The upside: easy to be complete and the schema is probably very stable.
  • The downside: this can only be performed by administrators.

I’d say we should either go with API or DB, and then post the results on a git repo somewhere public (preferably a few places!).

Thoughts from the admins?