Old Forum and Wiki now live! 😀

General CPCWiki related
, ,
1

Hello everyone!

I am glad to announce that the wiki and the old forum are now back online!

CPCWiki has finally caught up with the latest Mediawiki versions, a new sleeker appearance and modern functionality.

The old forum, in the meantime, is back online, albeit in read-only mode. You’ll see that all permissions have been revoked to keep it frozen in time.

Believe me, there was a ton of work involved - by myself, but more importantly by @Highlander who did a fantastic job setting up the infrastructure.

That said, there are bound to be issues. I know, for instance, that some extensions on the wiki (most were disabled anyway) do not work perfectly.

So, I’d be grateful if you could test things and report here with any findings.

With this, I consider the first and biggest phase of migration complete. We’ll now focus on fixing issues and fine-tuning things.

Thanks for your patience!

PS worth mentioning that, the moment the wiki went back up, the usual bots descended again in droves. They managed to make the site(s) unresponsive, even on this new, vastly superior server. We have implemented a series of measures (rate limiting, on-site bot detection and denial, WAF rules blocking entire countries etc.) However, I think that Cloudflare challenges are now much better and you should only see one if there’s something suspicious with your connection.

image
image1920×1103 339 KB

image
image1920×1163 320 KB

2

Looks awesome! Good work!

3

Yeah ! Congratulations to both of you.

Thanks a lot for your great work.

4

Fantastic!!
Very valuable information has been recovered. Great job!!

5

Well done, I’ve missed CPC Wiki over the summer. Great to see everything’s not lost.

6

Amazing!!! Thanks @Gryzor and @Highlander for bringing things back to life!

7

Nice stuff :slight_smile:

8

Incredible work to all involved. So glad to have the wiki back!

9

Welcome back man!

10

Sweet.

11

Thanks for your hard work! The new forum looks nice. I was so used to the previous one, it might take a while for me to feel at home, but I’ll get there :smile: .

12

Welcome back mate!

Yeah, it looks a bit different, but once you start digging in it you’ll see it’s not that different! :slight_smile:

13

“Frozen in time”, you’re spot on. The idea is to preserve the conversations, the value, the history.

You were 100% right to rebuild a working forum, if only to be able to render the pages for preservation.

Now, those pages aren’t meant to change, are they? Well, no by definition.

Maintaining a whole working software that dynamically renders page that are meant to … never change, feels pointless.

And with the current state of the Internet, you’re kind of forced to maintain it, upgrade it, etc, for fear of it being hacked.

Have you considered turning it into a static website? You could use a website fetcher (e.g. wget or similar) to fetch a complete standalone HTML snapshot of the whole set of forum pages. Then unplug the SMF installation and serve that set of pages at the same URLs. Instantly, serving pages consumes much less CPU, attack surface is reduced and maintenance is simplified, while guaranteeing that the rendered HTML, CSS etc remain constant through time.

One drawback: every non-public aspect (private messages etc) would be lost. You can announce the change several times and do it after e.g. a one year delay to let people record whatever they want.

What do you think @Gryzor?

14

That is actually my biggest concern that due to both of those considerations, the forum will be abandoned at some point in time and then all the information is lost.

I think we need to do something about it, that it is in this forum (maybe in an archive section). Happy to help with this.

15

Not a bad concern, to be honest, but in practice the solution kind of breaks down. I’ve been thinking about it since last night and there are a few stumbling blocks that are inherently impossible (?) to overcome:

1. wget issues

SMF generates dynamic URL permutations for every single topic:

  • Sorting filters (?sort=subject;desc, ?sort=starter)

  • Pagination and jump links (?topic=123.prev_next=next, ?topic=123.prev_next=prev)

  • Utility views (?action=printpage, ?action=profile, ?action=help)

  • Session identifiers and board index variations

If you point wget -m at SMF without dozens of regex rules, the spider WILL fall into infinite crawl loops. Since the forum has some 250k posts, it can easily easily generate 15 MILLION (from what I see) scrapable URLs - good luck with those!.

2. The URL & Web Server Problem

I ran a couple of tests; standard web servers do not serve static files from query strings like SMF uses.

To explain: when a user visits index.php?topic=1240.0, the web server sees index.php as the file path and ?topic=1240.0 as query arguments. If PHP is gone, server simply tries to serve the static index.php file over and over. To map static scraped files to original SMF query-string URLs, you have to write convoluted rewrite rules for Caddy to translate query arguments into flat filenames on disk. If that mapping breaks, you break every inbound link from Google and the CPCWiki itself, which is a SEO nightmare.

3. Attachments/Downloads

SMF serves attachments dynamically via index.php?action=dlattach;topic=.... This will likely fail fantastically witha crawler.

4. Search

if we keep it as static pages, it’ll kill the search engine (such as it is). We could build a separate client-side search index (like Pagefind) into the scraped HTML templates but that defeats the purpose, partially.

Solution?

The best solution I could come up with is to parse the db with a script to generate a list of unique topic URLs and then use wget to fetch the URLs in the generated list - but that doesn’t solve the rest of the issues.

Hardening

If security against vandalism etc is the concern, we can easily shut the whole thing down by just revoking INSERT, UPDATE, DELETE, DROP, ALTER ON on the db. Make the webroot RO, return 503s for specific actions like:

@blocked {
    query action=login*
    query action=register*
    query action=post*
    query action=pm*
}
respond @blocked "Forum is read-only" 403

…and Bob’s your uncle.

But hey, even if an attacker somehow gains access to do some stuff on the system as it currently stands, we still have backups in a dozen places now :slight_smile:

16

That IS a concern in the long term. I haven’t given it much thought, but there will come the time when the CPC will naturally cease to exist. No idea how to proceed from there.

17

That is a great start :grin:

18

Awesome that the old forum is back.

I really missed it and am so glad to be part of it again.

Thanks a lot for all the hard work you put in to make this happen. :grinning_face: :flexed_biceps:

19

Welcome back man! :slight_smile: