SquidgeWorld Archive

Chapter Text

This is, by far, not complete. But over the next few days, I intend on getting as much documented as I can so that if anyone else wants to approach the OTW's code for implementation, they can.

Step 1: Decide on a distribution and get it installed.

I've been partial to Centos from early on, having been part of the RedHat Linux folks, so Centos was my distribution of choice. I know that the archive wants folks to operate with Vagrant or Docker versions, and I initially had success with the MacOS version. Thing is, I only had an old spare Mac Mini 2012, and that didn't have the horsepower to get me where I wanted to go. So armed with an Intel NUC server fully decked out, I got Centos 8 installed on the NUC and began the process.

I will preface this documentation by saying that I mostly followed the OTW's "Linux Setup" page, though because of everything going on, not every single set of documentation is complete.  I had to pull some things from Linux Setup, OS X, Vagrant, and Docker.  All can be found linked to the original Linux Setup document.

As for Centos, I extrapolated based on OTW's MySQL usage that I would need about 4Gb storage for my MySQL database.  I created a 2Tb SSD partition for /var and decided that since that's where my MySQL databases would go by default, that I would put the OTW code on the same.  So I created /var/otwarchive in anticipation of the code load later on.  It should be noted that I had an issue later on with icons in the archive not loading, and one obscure document I found said that it was most likely due to /tmp and /var being two different partitions, and symlinking couldn't be done across partitions.  I've since lost that link, but don't need it since I've hacked my way to working icons.

Centos itself was installed with server and development utilities.  A short list of what was necessary is:

• ruby 2.6.5
• rubygem
• keyutils-libs
• krb5-devel
• libcomm_err-devel
• libffi-devel
• libkadm5
• openssl
• openssl-devel
• zlib-devel
• mariadb (regular, server, and devel packages)
• cmake
• libxml2
• xz-devel
• libxslt-devel
• libgcrypt-devel
• libgpg-error-devel
• redis (regular and devel packages)
• bzip2
• expat-devel
• fontconfig-devel
• freetype-devel
• libpng-devel
• libuuid-devel
• alsa-lib
• giflib
• java-1.8.0-openjdk (regular and devel) NOTE: There is a later version of this. DO NOT INSTALL IT. Stick with 1.8.0
• ttmkfdir
• xorg-x11-fonts-Type1
• elasticsearch
• memcached
• libmemcached, libmemcached-lib, and libmemcached-devel
• pwgen
• Calibre - installed directly from their website
• PHP and php-devel (not needed, but some utilities I may use do, so...)
• ImageMagick and ImageMagick-libs
• Python2 and Python3 including development libraries

One thing to remember is that not everything in Centos has an equivalent package in Ubuntu, which is what OTW uses for their archive.  This may be a stopping/stumbling block for some, but I persisted.

With these installed, now to the configuration!

Chapter Text

When you install several of these packages, you have to configure them, including:

• Set up the service to enable it
• Set the service to start on reboot
• Run a quick configuration
• Start the service and test it

MySQL:

After running through the above, you run the /usr/bin/mysql_secure_installation to set up your initial MySQL server. Note your username and password

Redis:

After running through the above, you must edit /etc/redis.conf though there's not much. Bind it to 127.0.0.1 so it only gets calls from your server. Pay attention to what port you set it to; 6379 is the default. Two things to note.  First, redis is what controls certain things in your system, such as the population of the filter_counts table. This was my biggest stumbling block, because I couldn't get filter_counts to work until I got redis functioning.  Second, while the OTW is a massive database, if you are working on a smaller system, the default configuration is really all you need.

Memcached:

All you have to do with Memcached is get it installed and enable it. Works out of the box with the right OTW configuration. More on that later.

elasticsearch:

This one is a bit of a bigger ball of fish.  Once installed, you have to edit /etc/elasticsearch/elasticsearch.yml to be correct.  I configured:

• cluster.name: this is set to my cluster information, even if it's just one server, so squidgeworld-cluster
• node.name: named for my node, so squidgeworld-node
• network.host: I left this at 127.0.0.1 - only from the localhost
• http.port: Leave this at 9200.  But note the port number because you need it later.

Chapter Text

Now's when we get into the guts of things!

As your end user (NOT root), run the following command to install the OTW code into your chosen directory:

• git clone https://github.com/otwcode/otwarchive.git otwarchive

This installed it into the otwarchive on my /var/ directory.  So now my code is installed in /var/otwarchive/  And now that the code is installed, you can run the "bundle install" and get the software installed and configured

Once that's done, you'll need these commands/files:

• Copy the file config.yml to local.yml - you will need to customize this file!
• Copy database.example to database.yml - you will need to customize this file!
• Copy newrelic.example to newrelic.yml - you will need to customize this file!
• Copy redis-cucumber.conf.example to redis-cucumber.conf - no need to touch this after you created it
• Copy redis.example to redis.yml - you will need to customize this file!
• Make a backup of resque_schedule.yml so you have the original
• Copy s3.example to s3.yml - NOTE: I am not using s3 (Amazon disk space) but OTW is. I made further changes elsewhere to get rid of this, but I left the s3.yml all the same

Breaking these down one by one:

local.yml 

This is the biggie.  You're going to want to change a lot in here, and it's mostly self explanatory, like email addresses. Of course, when you change the email addresses to your own domain, you need to make sure you have those defined in your mail transfer agent (MTA).  I use sendmail, which is fun in itself.  You also should install opendkim, opendmarc, and configure your domain records to include your opendkim signature.  That prevents (hopefully) your mail from looking like spam. And the biggest issue I've had is some mailservers rejecting SqWA email as possible spam.

Anyway, change your SESSION_SECRET to be a hex number that's custom to you. And DO NOT use the same hex number in DEV and PROD both! 

The value for ES_URL should include the elasticsearch port that you wrote down earlier. My value for this is: 'http://127.0.0.1:9200'

The value for MEMCACHED_URL is just completely wrong. Remove that line and replace it with:  MEMCACHED_SERVERS: ['127.0.0.1:11211:1']
I do not know why the config.yml documentation is incorrect, other than it is. When you try and run production mode with the MEMCACHED_URL instead of MEMCACHED_SERVERS you'll get a weird "500" error that makes no sense. This is where James helped me out the most. Bless him, I owe him a beer.

There are lots of values in here to modify, including AKISMET_KEY and AKISMET_NAME. I signed up for an Akismet account, even though I haven't A) figured out how it works (or if it's already working) and B) don't really have to worry about spam, at least right now.

database.yml

If you've done any work with databases, and good god why would you attempt this if you haven't, then this should be explanatory

newrelic.yml

I know the OTW uses newrelic for monitoring, but I don't. But because I didn't want to comment out all that code, I signed up with a free newrelic account.  You'll find your license key once you sign up, and stick it in this file.  Other than that, there's a few small tweaks, such as "app_name" that you have to touch to make it your own. 

redis.yml

Remember when you customized your redis configuration earlier and wrote down the port? You need it here.  For me, on a small database, this is what my redis.yml looks like:  NOTE: Since redis was configured to run on port 6379, then this will work for either development or production environments.  If you wanted to run a test environment, you would need to change your system's redis installatioon to run on port 6398:

redis_autocomplete:
test: 127.0.0.1:6398
development: 127.0.0.1:6379
production: 127.0.0.1:6379

redis_general:
test: 127.0.0.1:6398
development: 127.0.0.1:6379
production: 127.0.0.1:6379

redis_hits:
test: 127.0.0.1:6398
development: 127.0.0.1:6379
production: 127.0.0.1:6379

redis_kudos:
test: 127.0.0.1:6398
development: 127.0.0.1:6379
production: 127.0.0.1:6379

redis_resque:
test: 127.0.0.1:6398
development: 127.0.0.1:6379
production: 127.0.0.1:6379

redis_rollout:
test: 127.0.0.1:6398
development: 127.0.0.1:6379
production: 127.0.0.1:6379

resque_schedule.yml

This file is used by the system to do certain jobs. You can modify it, if you can understand it. I customized a couple of things, but mostly I left it alone.

s3.yml

Because I commented this out mostly, I don't think it's needed.  But in case, I left it. My contents (just 2 lines) are:

bucket: disabledbucket
s3_region: us-east-1

NOTE: So in order for me to run in production WITHOUT using Amazon S3 storage, I had to change the word "production" to "unproduction" in the following files:

/var/otwarchive/app/models/collection.rb
/var/otwarchive/app/models/pseud.rb
/var/otwarchive/app/models/skin.rb

Now that you have all this, you're really close to setting up the archive!

Chapter Text

So now you're ready to setup your environments!  You're getting close!

For me, I changed directory to the /var/otwarchive home to do all these functions.  Once in there, I ran:

• bundle exec rake db:create:all - This creates all 3 of your databases, development, test, and production

Because I'm focusing only on production environment, because that was the hardest thing to set up, just know that at this point, I set the environmental variable "export RAILS_ENV=production".  This is so everything I do from here on out is for the production environment.

• bundle exec rake db:schema:load - This loads the schema into the database
• bundle exec rake db:migrate - This migrates you to the latest schema

Now it should be noted that the db:migrate was supposed to import roles into the system. However, this was not the case, and later on I ended up doing an insert to get the roles set up properly.  Here's my roles table:

Yes, you need all these for one reason or another.  Especially if you are going to allow regular users to wrangle tags, which I suggest if you're doing this alone. Why? Because it's a huge undertaking after doing the imports from an eFiction database that had 7,000 users, 35,000 stories, and tens of thousands of tags.

Now you need to load the required tags (the ratings, categories, etc).  First there's a script you can run to get the OTW's required stuff inserted:

• bundle exec rails runner script/ensure_required_tags.rb

As for other required tags, I was never able to get media (Movies, TV Shows, etc) tags to load through the interface; it always gave me an error.  So instead, I just ran the following in a MySQL window:

insert into tags (name, canonical, created_at, updated_at, type, sortable_name)
values('Anime & Manga', 1, NOW(), NOW(), 'Media', 'Anime and Manga');

insert into tags (name, canonical, created_at, updated_at, type, sortable_name)
values('Books & Literature', 1, NOW(), NOW(), 'Media', 'Books and Literature');

insert into tags (name, canonical, created_at, updated_at, type, sortable_name)
values('Cartoons & Comics & Graphic Novels', 1, NOW(), NOW(), 'Media', 'Cartoons and Comics and Graphic Novels');

insert into tags (name, canonical, created_at, updated_at, type, sortable_name)
values('Celebrities & Real People', 1, NOW(), NOW(), 'Media', 'Celebrities and Real People');

insert into tags (name, canonical, created_at, updated_at, type, sortable_name)
values('Movies', 1, NOW(), NOW(), 'Media', 'Movies');

insert into tags (name, canonical, created_at, updated_at, type, sortable_name)
values('Music & Bands', 1, NOW(), NOW(), 'Media', 'Music and Bands');

insert into tags (name, canonical, created_at, updated_at, type, sortable_name)
values('Other Media', 1, NOW(), NOW(), 'Media', 'Other Media');

insert into tags (name, canonical, created_at, updated_at, type, sortable_name)
values('Theater', 1, NOW(), NOW(), 'Media', 'Theater');

insert into tags (name, canonical, created_at, updated_at, type, sortable_name)
values('TV Shows', 1, NOW(), NOW(), 'Media', 'TV Shows');

insert into tags (name, canonical, created_at, updated_at, type, sortable_name)
values('Video Games', 1, NOW(), NOW(), 'Media', 'Video Games');

With these loaded, you're now ready to create users!

First, we're going to use the Ruby shell interface to create two required users. The first is your user, and the second is the required orphan account.  You'll need to the orphan account otherwise some basic functions of the archive will not work/will error out.  To do this:

• bundle exec rails c - This will bring you to a rails console.  Once in, you need to create users with this template.  Just updated it to be your values:
• User.new(login: "myaccountname", email: "myemailaddress@mydomain.com", password: "MyChosenPassword", password_confirmation: "MyChosenPassword", age_over_13: "1", terms_of_service: "1").activate - Of course change the values to what you want to use
• User.new(login: "orphan_account", email: "throwawayemail@mydomain.com", password: "MyChosenPassword", password_confirmation: "MyChosenPassword", age_over_13: "1", terms_of_service: "1").activate - Theoretically there should be no email going to this account. I set mine to an email address that is redirected to /dev/null. The password should be non-guessable because this account should NEVER be used.

Now you need to create an admin user.  My first user was called "whhtemp" so I chose to use the same name as my admin. Don't worry - the system will preface your username with "admin-" so there's not an issue. But still.  In order to create an admin user, still in /var/otwarchive directory:

• bundle exec rails runner script/create_admin.rb
• myaccountname, myemailaddress@mydomain.com, board, communications, docs, open_doors, policy_and_abuse, superadmin, support, tag_wrangling, translation - It's going to popup with a prompt.  If you're running the archive by yourself like I am, you're going to copy and paste this in to the prompt. If it fails, then you probably didn't install the pwgen package. If it works, then woohoo! 

You're ALMOST READY!  One more thing to do (still in the /var/otwarchive directory):

• bundle exec rake skins:load_site_skins - This loads the skins that are delivered with the archive.  You can use them at the beginning, but thing is, to distinguish yourself from OTW's AO3, and now Squidge.org's SqWA, you need to plan on building your OWN skin. And don't forget, you have half a dozen new icons you have to replace, and about 30 bajillion hardcoded instances of AO3 to change in client-facing code. Save that for later.

With the skin loaded, go to a window and type in:

• RAILS_ENV=production bundle exec rails server -p 3333 -b (server.ip.address.here) -e production

When the server starts up, go to your web browser and go to the address you specified.  For me, it was 50.43.78.22 and then tack on the port number.  So it was: 50.43.78.22:3333

You should be able to log in as an admin.  So in my URL, I actually changed the URL to: 50.43.78.22:3333/admin/login and this brought me to a login prompt, albeit a strange looking one, and that's because the skins aren't fully loaded.  As long as you're logged in, click on the "Skins" menu item and then choose "Approved Skins". Here you should see the included skins. Tick the "Cache" button and then type in the default skin title, which is "Archive 2.0".  Save that at the default, and say yes at the prompt to give it to all users.  Once you do that, the site loads and it looks like you are at the AO3.  

Welcome to your archive!  But you're far from done.

Chapter Text

Before you get any further, might as well get a couple of things straight. First of all, you'll see the archive looks different as an admin. From the gradient-red up, it looks like the archive. But there's a new menu starting with "Manage Users" and ending with "Manage API Tokens". You might as well go through and play with these to see what's in each.

First, I would set up your regular user to be a tag wrangler.  Do this under "Manage User", search for them, and then set them with all of the powers you want to give them.  NOTE: If "Tag Wrangler", "Translator", etc doesn' show up, then you're missing the roles insertion from a previous chapter.

Next, I would set up what's in Settings, and then tackle skins to make sure your skins are set.   You'll have to know CSS for this.  Good luck with that.

Under Locales, make sure that you have at least English (US).  Otherwise, things break, yo.  I set mine up to be the primary, used for email, and used for interface.

When you have everything set, then you can entertain creating and/or importing users.  If you are coming from an eFiction archive, notes:

• eFiction allows you to have multiple pennames and use the same email address for both.  Nope - not with AO3 code.  Prep your users to let them know if they have multiple accounts with the same email address, only the first one will be created.  Now you can get creative with your scripting and create a separate pseudonym, but nada - I didn't do that. 
• Limitation on AO3 software is one email address, per above. Now I had 7,000 users, and some of those people have been gone for a decade, so I'd set their email to "bad-user-email@squidge.org" which is a /dev/null account.  But on AO3 software, I had to update this to be "bad-user-email+theirpenname@squidge.org" to not run afoul of the index limitation.
• eFiction allows you to have a space in your penname, so "My Penname Is Awesome" is valid in eFiction - but NOT as an AO3 code login, though it's fine for an AO3 pseudonym.  So I created users that had spaces in their names with underlines as the login column ("My_Penname_Is_Awesome") while their pseudonym stayed the same ("My Penname Is Awesome").  Works fine.
• I had a shell script do all this, pulling users one by one from my eFiction database and then dumping them into the users table and then the pseudonym table, but also I included the eFiction UID column on the users table so that I could trace them back later.

Next, I mapped eFiction parent categories to AO3 base categories, which was mostly straightforward.  I had to make a few judgment calls here and there.  Once I did that, I ran a script to import the category based on the eFiction parent category.  Yes, I have a script for this.  It was just changing a few values in the script and it inserted into the tags and common_taggings table.  My script did one parent category at a time, imported the category (fanfiction_categories), and then imported all of the characters (fanfiction_characters where charname not like '%/%') and relationships (fanfiction_characters where charname like '%/%').  NOTE: Before I did this, I stripped several charname fields and categories from having any single quote/apostrope or commas, because this blew up imports.  Sure, "Dara O'Brion" imports as "Dara OBrion" and "S01E21 He Shall, From Time To Time" to "S01E21 He Shall From Time To Time", but hell, it's better than having to import by hand!

This imported everything, even tags not used.  Since I was dealing with an archive that started in 2007, we had a ton of stuff in there and I wanted to keep history.

Next came importing stories. But the eFiction used this bullshit "FIND IN SET" stuff that meant characters were in one long field, same with tags, warnings, etc.  Basically I found a utility that stripped these out into their own values in their own table.  So if SID 12345 had charid values of (32, 555, 435, 658), I ended up with a table with four rows, each with 12345 as the SID, and then followed by the separate value - so 32 on the first, 555 on the second, etc.  I did this for characters, tags, and warnings.  eFiction had a single rating, so I just had to map those from eFiction to AO3 code.  Oh, and remember what I said about two different pennames, one email address?  I did matching based on the email address, so that penname "My Penname" with 4 stories on eFiction and "Another Penname" with 3 stories on eFiction, but both of those accounts used the same email address, well "My Penname" came through on SqWA with 7 stories total, and the "Another Penname" was thrown out, because reasons.  People are free to create a second pseudonym on SqWA and attribute those stories accordingly.  I'm not the boss of them.

If anyone is really interested, I have scripts to import stories (the base story and a creatorship record, each chapter and a creatorship record, any mapped warning -OR- "No Archive Warnings Apply", one mapped rating, all tags, all characters, and all relationships, and then inserted records for stat_counters and filter_taggings, and finally added it to my imported collection that I created. Once it was imported, including the fanfiction_stories.sid value tacked onto the works table, I updated the fanfiction_stories table to say it was imported.  This process was run story by story with 0.5 second pause between the two, and took 2 full days to run and import.  Sure, it might have been easier to do it another programmatical way, but I wanted to control it the way I was comfortable with.

Once stories were imported, I imported all comments. One thing I would do differently would be to NOT import any "NO COMMENT" comments, because eFiction allowed you to not leave a comment, but leave a rating value.  The comment import was done one by one so that the stat_counters table could be updated.  There were around 50,000 comments imported and it several hours.

Once comments were done, I imported kudos. Now eFiction doesn't have kudos, but I'd built them into the system, and there had been 600,000 kudos done. And the thing about kudos is, it's restricted by IP address for anonymous kudos, which about 550,000 of those kudos were.  So I had to have a special script that not only imported the kudos, but for all anonymous kudos, created a fake - but valid - IP address.  If you're interested, it's the TRUNCATE code below:

echo "insert into kudos(commentable_id, commentable_type, created_at, updated_at, ip_address, user_id)" > kudosinsert.sql
echo "select w.id, 'Work', ffk.kudosdate, ffk.kudosdate, " >> kudosinsert.sql
echo "CONCAT(" >> kudosinsert.sql
    echo "TRUNCATE( RAND() * (255 - 1 + 1) + 1, 0 ), '.'," >> kudosinsert.sql
    echo "TRUNCATE( RAND() * (255 - 1 + 1) + 1, 0 ), '.'," >> kudosinsert.sql
    echo "TRUNCATE( RAND() * (255 - 1 + 1) + 1, 0 ), '.'," >> kudosinsert.sql
    echo "TRUNCATE( RAND() * (255 - 1 + 1) + 1, 0 )" >> kudosinsert.sql
echo "), NULL " >> kudosinsert.sql
echo "from works w, wwombconv.fanfiction_kudos ffk " >> kudosinsert.sql
echo "where ffk.id = $MYKUDOSID" >> kudosinsert.sql
echo "and ffk.sid = w.wwombsid;" >> kudosinsert.sql

#run the query
$SQPROD < kudosinsert.sql

That was kind of the fun part; I got to nerd out while writing this code.  Of course this also meant that the stat_counters table kudos column needed to be updated.  This script took a day to fully import.  Anyway...

After kudos were imported, I imported the entries of table fanfiction_favstor as bookmarks, plus updated the stat_counters table

And finally, I imported the fanfiction_favauth table as author subscriptions.  Before I did this, I updated the fanfiction_favauth table with the true single uid based on what the primary (first use of email address on eFiction) was.

And now the archive was ready to run!

Chapter Text

Once I was ready to go, I had to set up webserver nginx to run. This was new to me; I thought I would be able to redirect my port for the archive to 443 and be done with it, but no. Stupid root permissions and shit. So anyway, I installed nginx (which was odd since I'm used to Apache, but whatever), and while I was at it, installed certbot so I could have SSL.  Can I say that it seems like a simple configuration file, but JFC that's some weird-ass bullshit.

Anyway

So once I had nginx up and running with SSL, I was ready to start the archive.  This is a while thing that I do each time, though I don't know if I need to or not (spoiler, NO) but whatever:

• bundle exec rake After:sortable_tag_names - This shows you all of your tag names, useful if you want to look through them, make some edits later in the frontend or MySQL as I've had to do for some mangled tags
• bundle exec rake Tag:reset_common - resets common tags
• bundle exec rake Tag:reset_count - I can't remember what this does
• bundle exec rake Tag:reset_filters - See above
• bundle exec rake Tag:reset_filter_counts - Holy shitballs boss.  So the problems I was having was that I couldn't get my filter_counts table to populate. That was because of my configuration issues I mentioned previously. As soon as I did this, I saw my "tail -f production.log" go bonkers with inserts because it was FINALLY CONFIGURED PROPERLY.  I think I wet myself.
• bundle exec rake Tag:reset_meta_tags - Yeah, no clue
• bundle exec rake autocomplete:load_data - This is a biggie. If your data is set up properly, this will load your autocomplete, so if a user types in "Star" in a fandom field, it will suggest "Star Wars, Star Trek, Star Treak: The Next Generation" etc, etc. To quote Joe Biden, "This is a big fucking deal."

Now at this point to get my archive running, I have three scripts, plus one ruby script to run.  I'll provide the script call first, then the contents:

• bin/reload_elastic

[walterh@squidgeworld otwarchive]$ cat bin/reload_elastic

#!/bin/bash
export RAILS_ENV=production
sudo systemctl stop elasticsearch
sudo rm -rf /var/lib/elasticsearch/*
sudo rm -rf /var/lib/elasticsearch/tmp
sudo mkdir  /var/lib/elasticsearch/tmp
sudo chown elasticsearch:elasticsearch -R /var/lib/elasticsearch
sudo systemctl start elasticsearch

cd /var/otwarchive/

# Wait for forever until elastic is back
until $(curl  --output /dev/null --silent --head --fail -XGET 'http://localhost:9200/_cluster/health?wait_for_status=yellow' ); do
    printf '.'
    sleep 5
done

bundle exec rake environment  search:index_bookmarks
bundle exec rake environment  search:index_pseuds
bundle exec rake environment  search:index_tags
bundle exec rake environment  search:index_works

• bin/start_my_workers

[walterh@squidgeworld otwarchive]$ cat bin/start_my_workers

#!/bin/bash
cd /var/otwarchive

QUEUE="*" bundle exec rake environment resque:work &

• bundle exec ruby bin/wait_for_resque.rb - This script is part of your installation, so I don't need to waste real estate here.  It runs through all the jobs until the resque workers are done
• bin/start_my_unicorns

[walterh@squidgeworld otwarchive]$ cat bin/start_my_unicorns - NOTE: I'm not using vagrant, but that's the variable I had - so whatever. Port 9292 was a port I chose, and my nginx says to monitor incoming on port 9292 and serve it out on port 443. UNICORNS=8 is the number of CPUs I have. The "pseud_watcher.sh" and "collection_watcher.sh" are two scripts that I have running using the inotofywait UNIX utility that copies in the pseud's or collection's icon, since that has never worked on my system except in development.  But I fixed it with a script, so meh.  You don't need it, but if you do, I can help you with it.

#!/bin/bash
me="vagrant"

CONF=/var/otwarchive/config/unicorn.rb
export UNICORN_PORT=9292
export UNICORNS=8

unicorn_master_pid=`ps ax | grep $me.sock | grep master | awk '{print $1}'`

if [ "$unicorn_master_pid" != "" ]; then
echo "removing old unicorn master"
/var/otwarchive/bin/kill_my_unicorns
sleep 5
fi

echo "starting new unicorn master"

if [ -f "config.ru" ]; then

echo
bundle exec unicorn -l /tmp/${me}.sock -c $CONF &

#Now start the pseud watcher
nohup /var/otwarchive/scripts/pseud_watcher.sh &

#now start the collection watcher
nohup /var/otwarchive/scripts/collection_watcher.sh &

else

echo "Ooops! You don't appear to be in your rails application directory"

fi

• nohup bundle exec rake resque:scheduler & - This runs your resque scheduler, which controls a lot of indexing and such.

Now I have more scripts later, for killing my unicorns and my workers.  But those are really just easy scripts to kill processes.  

This should be more than enough to get you up and running on the OTW software.  There may be a few things here and there that I forgot, and if I did, then I'll add them in.  But this should be everything!

Chapter Text

Short and sweet here.

Prior to you even thinking about starting the First Thing in the next paragraph, go do your edits.  Edit all your client-facing documentation to change AO3 to your archive's chosen short code.  Then go fill out some FAQs because you're going to have to change more documentation to point to your FAQs instead of AO3's.  Oh, and get your icon changes done.  All of them.  And when you have all that done, restart your archive, make sure you didn't miss anything, or just make a blanket 'We'll fix them as we find them' mea culpa post if you're lazy like me.

First thing you're going to do is tag wrangling.  All that shit you imported?  Yeah, you now have to go through and assign it to fandoms.  Sure things like "First Kiss" or "Alternate Reality" are going to be canonical and attached to every fandom, so what I did was all of those, I attached to "No Fandom" fandom.  Quick and easy.  But then came the big thing; I had about 25 different permutations of "Alternate Reality" that I had to decide - was it a tag that was truly deserving of a standalone tag, like "AU - Sentinel/Guide Themes" or was it just a synonym of Alternate Reality, like "AU" and "AU - Canon Divergent"?  That was a huge cleanup right there.

After free text, you have to assign characters to fandoms, some fandoms to media (like some of my stuff imported to Uncategorized Media and I had to wrangle it), some relationships to fandoms - and in turn add in characters to relationships.  You should do all of this under the "Tag Wrangling" admin page.

Once you're done with that, assign a bunch of fandoms to your regular user that you gave Tag Wrangling permissions to, and log in as them and go to the Tag Wrangling link that's off your profile from the above right (like "My Preferences", "My Works", etc).  I ran a script and just assigned everything to my user, which can make it slow to load, but at least everything's in one place.  Then you need to wrangle characters, relationships, etc to fandoms.  This will take you a couple of weeks, or at least that's what it's taken me because I've had so damned many to deal with.

And because it doesn't fit anywhere else, I have a few cron jobs that I run to deliver emails, etc.  These are my entries:

20 * * * * /bin/bash -l -c 'cd /var/otwarchive && RAILS_ENV=production bundle exec rake notifications:deliver_subscriptions 2>&1' >> /var/otwarchive/log/subscriptions.log 2>&1

30 04 * * * /bin/bash -l -c 'cd /var/otwarchive && RAILS_ENV=production bundle exec rake notifications:deliver_kudos 2>&1' >> /var/otwarchive/log/kudos.log 2>&1

00 04 * * * /bin/bash -l -c 'cd /var/otwarchive && RAILS_ENV=production bundle exec rake work:purge_old_drafts 2>&1' >> /var/otwarchive/log/purge_old_drafts.log 2>&1

01,06,11,16,21,26 * * * * /bin/bash -l -c 'cd /var/otwarchive && RAILS_ENV=production bundle exec rake readings:to_database 2>&1' >> /var/otwarchive/log/readings.log 2>&1

31,36,41,46,51,56 * * * * /bin/bash -l -c 'cd /var/otwarchive && RAILS_ENV=production bundle exec rake readings:to_database 2>&1' >> /var/otwarchive/log/readings.log 2>&1

05,15,25,35,45,55 * * * * /bin/bash -l -c 'cd /var/otwarchive && RAILS_ENV=production bundle exec rake resque:run_failures 2>&1' >> /var/otwarchive/log/run_failures.log 2>&1

50 01,07,13,20 * * * /var/otwarchive/scripts/dailyadds.sh

 

These should be mostly understandable, except for the last time. The last "dailyadds.sh" script is something that I have put together that emails me four times a day with any tags (fandom, characters, relationships, and freeform) added to the database in case I'm away from monitoring the system. It also looks in each story added in the system and tells me if there are any "no no" items - like Patreon or Ko-Fi links - in the stories, because unless the fandom isn't copyrighted, those are a no-no.

Got questions?  I can't promise I'll be quick or immediate, since right now a nap is in my future.  But ask them here or ping me and I'll get back to you.

Chapter Text

So there's a lot to learn when you update the OTW base code. Here's a few of the things that I've one through the last few days.

First off, I pulled code from MASTER, which I shouldn't have done. I know that now (thank you, James!) because in reality, I should have only updated from the current level of code running on the OTW's AO3. If you want to know their current code, head over to AO3 and look at the footer. It'll say "otwarchive v" and a number. That's the tag that you need to check out when you're ready. I don't know git all that much - but now I know a hell of a lot more than I did. So checking out the tag for OTW's code will download the current stable version instead of something in MASTER that may not work.

Second, when you've done your code pull, you do two commands:

The thing about these two is that, in the "db:migrate" step, DO NOT IGNORE WHAT IS PRINTED TO THE SCREEN! I glossed over it, not realizing that there were manual database steps to take. So that was fun. At least there were only a half a dozen to perform, and what's in the db/staging directory helps because it's stored by date.

Third, when you pull down the code, it's going to warn you about overriding customizations. The thing about the AO3 code is that it's got a LOT of self-references - so emails that go out, and items that display on the screen are a lot of times hardcoded to say "Archive of Our Own" or "AO3" and the like. That's fine if you're running AO3, but not if you're running SquidgeWorld or whatever archive Connor (hey, Connor!) is running for their archives we talked about recently. Yes, there are more of using OTW's code for non-OTW projects!

But beyond the customizations, there are certain things that just won't work unless you're set up directly like AO3 - a massive system in a massive datacenter somewhere. AO3 relies on Amazon S3 to host things like your profile images, so you actually have to intentionally break their code to get it work on your local system.

That's where we are. This is going to be a list of files that were updated, and how they were updated, so that OTW can start making their code "portable" meaning anybody could pick it up, set up a few items in a configuration file, and not have to hand-code a massive number of files each time code is updated.

The current list is here, along with why it has to be updated.
app/models/pseud.rb - Change the reference from "production" to "unproduction" for this line only in order for icons to work unless you are using Amazon S3 storage: storage: %w(staging unproduction).include?(Rails.env) ? :s3 : :filesystem,
app/models/skin.rb - Change the reference from "production" to "unproduction" for this line only in order for icons to work unless you are using Amazon S3 storage: storage: %w(staging unproduction).include?(Rails.env) ? :s3 : :filesystem,

app/models/collection.rb - Change the reference from "production" to "unproduction" for this line only unless you are using Amazon S3 storage: storage: %w(staging unproduction).include?(Rails.env) ? :s3 : :filesystem,

config/initializers/archive_config/mail.rb - Change this line from the default "mta" to "sendmail" so that my MTA of choice will work: config.action_mailer.delivery_method = :sendmail

More to come!!!