Welcome to the new server

[Ai]

Congratulations, you're looking at this on the brand new server... At last!


It has been a bit of a rocky journey! We're using a non-standard installation of Ubuntu using a ZFS filesystem. This is a feature that they're looking to roll out in the next LTS 20.04 in a months time, but not to the level we've taken it! The feature will only allow the use of whole drives, not individual sections of drives like we need. We had no end of issues during the move, firstly the server refused to boot as it didn't want to read the /boot/EFI partition. We eventually fixed that using an EFI Shell, simply accessing the fileset and renaming the directory twice did the trick although we still have no idea why!


Then we got everything running and installed ready for the move and during our tests we found the MySQL on ZFS is a NIGHTMARE. The databases were performing around 6 times slower on a server with no traffic than they were on the old, fully loaded server. We tried every trick in the book to tune this but ultimately failed. In the end we had to wipe the new server and start from scratch. This time we reserved a 1TB partition from each drive and used that to create a software Raid10 EXT4 array to place the MySQL data files on. This worked much better



I've learned a hell of a lot about custom partitioning, and a lot more than I ever wanted about EFI Shells. Ultimately we have ended up with a server that:

• More power processors than before
• Same RAM as before (speed and capacity) - 32GB
• 14TB of RaidZ1 ZFS disk array for server and site data (before we had3TB in Raid1 and a single 3TB backup drive)
• Every site (and a lot of key linux areas) reside of their own ZFS volume which gives us data snapshots (point in time recovery). 1/4 drives can fail with no impact on data and minor impact on speed.
• For the sites and key linux areas, the snapshots run hourly and are stored for 48 hours. Daily are stored for 2 weeks and weekly are stored for a month. If we delete something or it breaks we can recover up it up to a month later.
• MySQL sits on a 2TB Raid10 array meaning we can tolerate 1/4 drives (potentially 2/4 drives if it is the right two!) failing with little impact.
• Backups are made daily consisting of all site data and the databases. These are both stored on the server for immediate use and copied remotely to a backup server in another datacentre for redundancy. Akuma also copies these backups to his home server. This gives us 5 copies of the data - 1. Original Data, 2. ZFS Snapshots, 3. Backups on server, 4. Backups on another server, 5. Backups on Akuma's NAS. The maximum amount of data we stand to lose in the event of a dual drive failure is up to 48 hours. If I get clever enough, I'll also work out how to remotely back up the snapshots to my own NAS's ZFS array with incremental updates on the fly. This will have to wait for another day though!
• The server software is vastly updated, although we're still limited to using an older version of PHP due to incompatibilities with our scripts and newer versions.

A lot of you may be wondering what happened to the planned upgrade to Xenforo. Unfortunately (I'm using this word a lot today!) things did not go smoothly. There were many issues that I was unable to overcome. These include:

• No direct upgrade path for the downloads script - we have 2844 files for download (fiction and videos), importing them manually would be a nightmarish job.
• Multiple issues upgrading from such old scripts that needed work arounds.
• Inability to recreate a lot of the current functionality, specifically the front page.
• Escalating costs for plugins.
• Zenforo 2.X was also a very new release at the time and as such contained many "features".

This is still part of the plan, but it has had to take a backseat to real life issues and time constraints. To get the project rolling again will cost around $300 in licence extension (yes, you have to pay annually for plugin and forum licenses if you want to stay up to date with current software and upgrade) fees. It will also rely on either Akuma being able to dedicate a significant amount of time to 1. Learning the software, and 2. Custom coding the missing functionality; or more money for a freelance coder to do the job. We'll have to wait and see how things pan out as right now donations are only just covering the single server fee, I paid the new server setup fee and the extra month for the old server out of my own pocket.


Remaining issues from the move:

• DiaperedAnime lost around 2 hours of data. Unfortunately I was unable to restore the changes made while we tested it on the new server back to the old server due to non-backwards compatible software differences. No other sites lost data.
• AnimeOTK has some broken download files. This was caused by use of non-utf8 characters in the filenames. Transfer and copy commands really don't like non-utf8 characters (in this case they were latin1 characters) so they had to be automatically converted during transfer. This has resulted in around 15 broken downloads - please report them to Akuma if you find them as there is no simple way to locate them without trying every download manually!

EDIT: New Issues Since Discovered

• All sites had a bug whereby the apparent IP address of a user was the same as the server IP address. This borked the who's online list and prevented people logging in since it also registered this single IP against invalid password attempts. This bug has been fixed as of 15.40 GMT on 26th February.
• All videos and fiction in the downloads area is returning an error "file not found". We're unsure of the cause at this time as the files are there and it may be related to the character encoding issue experienced during transfer.

[Ai]

TL;DR Version:
Ai and Akuma spent weeks doing techy, geeky shit and swearing at the server.
Ai pulled an all nighter and finally got it working.
Shit's good and data has never been more secure from server failure.

[nelson1977]

The download area with the spanking clips doesn't work!

[mastervegeta]

I tried to enter AOTK on my tablet this afternoon,put my correct username and password,and was greeted with a screen saying I put the wrong username or password and used my limit of trials and had to wait 15 minutes to try and log in again. O_O

[guesswho]

Nice job. Where and how can I donate to help with the costs?

[Gus1996]

Originally Posted by mastervegeta

I tried to enter AOTK on my tablet this afternoon,put my correct username and password,and was greeted with a screen saying I put the wrong username or password and used my limit of trials and had to wait 15 minutes to try and log in again. O_O

Nice to know I'm not alone, except in my case:

I'm the new gustantoide, after the recent Server Update or whatever that was, I lost access to my gustantoide account ... in fact, when I went to the Recover Password page, it told me that my e-mail had never been registered ... so I created a new account (with my seemingly never registered e-mail) and ... I guess I'll have to stay with this new account from now on.

That won't change much for me since I only got downgraded from Member to Probationary User, but please don't expect to receive any answers to any PMs you might have sent to my old account.



PS: also, I just noticed that the Recently Updated RPs that used to be on the lower part of the Home page just vanished (was it a planned change ?)

[anotherclone]

I also have this problem when I try to log in. If I try to change the password using e-mail, then I get the same message. If I try to create an account with new mail and an old login, I get the message "That username is already in use or does not meet the administrator's standards".
I created another account, but after logging out I ran into the same problem.
I'm afraid it will be with this account.

[Synthet]

Originally Posted by anotherclone

I also have this problem when I try to log in. If I try to change the password using e-mail, then I get the same message. If I try to create an account with new mail and an old login, I get the message "That username is already in use or does not meet the administrator's standards".
I created another account, but after logging out I ran into the same problem.
I'm afraid it will be with this account.

UPD:There is no more problem! Thank you very much and sorry for the panic!

[17JosephLC]

Would this also affect notifications for thread posts? I don’t know if it’s a problem on my or your sides, but I’m not getting any emails when my subscribed threads are posted on, even though I have them set for instant notification.

[OTKUGO66]

Congratulations on the new server and good luck in your continuing technical struggles!