Migrate the scripts in S3 to use script._id instead of the path value

[Martii]

Currently we are unable to upgrade the aws-sdk dependency because "something" changed that makes getObject to fail.

This has been explained in detail at aws/aws-sdk-js#430 . With all the test deploys done trying to figure out what the steps to reproduce are it appears that normalizing the casing when we save to the DB is an option.

Looking for more options related to advancing the goal of updating the dep... e.g. perhaps we could validate the current aReq.params.username against the real username (name) that is typically in the Raw JSON Data for an account. e.g. a lookup to correct the casing around here... but I suspect this is going to need to be a DB migration with a BLOCKING label until it is finished.

See also:

• Rethrown error when streaming aws/aws-sdk-js#397 (comment) error trap suggestion for loosely related
• https://github.com/OpenUserJs/OpenUserJS.org/blob/71ef808762ccf14a048c8a639d6d620e4e4e91e4/controllers/script.js#L350 Probably related and should be fixed

Test points:

• https://openuserjs.org/scripts/Marti/oujs_-_Meta_View/source vs https://openuserjs.org/scripts/marti/oujs_-_Meta_View/source e.g. manual url entry in the addy bar
• https://openuserjs.org/scripts/JRI/Geocaching_Map_Enhancements/source Since jri is currently in the DB not JRI
• https://openuserjs.org/users/jri (http://localhost:8080/users/jri with local pro)

[Martii]

With #487 we have another option of inline live migration... that patch currently fixes the net timeout in dev when selecting some other script source with fakeS3 and it usually just spins the browser spinner for an incredibly long time since our local dev DB doesn't usually have that source.

We can either correct the casing to what the user entered, if applicable, (e.g. a quick user search in the DB and if it is a hit compare it to the username portion generated from installName and migrate) from what we store in the Raw JSON Data or we can just lowercase migrate everything username wise... It is probably not recommended that we should alter the scriptname value though in casing with the Key.

Updated the above for some test points and other linkage.

[Zren]

Normally you wouldn't base the filename/keys based on the username/scriptname. Create an id for each source code and store it in the Script model. If we wanted to future proof ourselves, we wouldn't use the Script._id either as that means only 1 source code per Script. But since we're not versioning it atm, we could just use that.

We have to fetch the Script model first before we doing any AWS stuff anyways.

Edit: Also noticed this when I tried to refactor the scriptEditSourceCodePage that never got finished. Most of the callbacks need to follow the callback(err, ...) pattern so we can chain error up to the route handlers.

[sizzlemctwizzle]

I agree with @Zren. We should use Script._id instead since we have to do a lookup anyway.

[Martii]

Create an id for each source code and store it in the Script model.

Yah I'm not entirely sure how lower casing a Unicode character would affect things so a unique id would probably be best. EDIT: Searching will need to be retested for this as well.

[Martii]

Just a FYI here's a short snippet from recent production log with the isPro console message I added in to check for this on pro e.g. the version of aws-sdk we are currently using hides/mishandles the error and possibly returns an empty payload as one of the maintainers stated... jri's script does show source and does install.

[12/08 23:36:21 EST][err] S3 Key Not Found jri/Geocaching_Map_Enhancements.user.js
[12/08 23:40:33 EST][out] Group(imgur) Rating Updated
[12/08 23:45:24 EST][out] Group(deviantART) Rating Updated
[12/08 23:48:10 EST][err] S3 Key Not Found mienaikage/OkChoices.user.js
[12/08 23:48:53 EST][out] Group(Subtitles) Rating Updated
[12/08 23:49:42 EST][out] Group(Steam) Rating Updated
[12/08 23:49:42 EST][out] Group(dota2lounge) Rating Updated
[12/08 23:49:47 EST][out] Group(download) Rating Updated
[12/08 23:51:02 EST][out] GitHub client authenticated
[12/08 23:51:10 EST][out] Group(bookmarks) Rating Updated

---

Hmmm their clock appears to be a bit skewed too.

[Martii]

Hmmm... retested this randomly with aws-sdk@2.1.25 and doesn't appear to crash dev nor local pro anymore... not sure on actual production. All prior test points appear to be okay (without crashing local pro). :\

@sizzlemctwizzle
Any objections to trying it on production? I'll give this a day or so for any responses.

[sizzlemctwizzle]

I have no problem with a temporary fix, but we definitely need to migrate the scripts in S3 to use script._id instead of the path value somewhere down the line. It's a simple migration but I'm always afraid I'm going to break something when I mess with S3. I spent hours trying to get everything right when I removed namespaces.

[Martii]

Of course the issue needs to stay open... just thought I could get the dep updated a little bit. I am on a slightly newer node.js though which is why it would need actual production testing... just got done with local pro and seems okay not to crash here.

See also:

• Martii/UserScripts@f08510e and it's corresponding OUJS source page

---

Possibly related to:

• Test out node 0.12.x with our project locally #581 ... perhaps this is why node 0.12.x fails with fakes3 as well... won't know until I recompile current 0.12.x and retest assuming production doesn't crash continually.

[Martii]

Appears to be working on production with the dep update... will let this run for a while and recheck the production logs periodically. Still getting my error trap message but server isn't restarting... tried a few variations on casing of the username and those return 404 which was unexpected but doable I think:

[04/27 02:12:11 MDT][out] Group(dota2lounge) Rating Updated
[04/27 02:12:11 MDT][out] Group(Steam) Rating Updated
[04/27 02:22:28 MDT][out] Group(Google) Rating Updated
[04/27 02:22:41 MDT][err] S3 Key Not Found trespassersw/google_cache_comeback.user.js
[04/27 02:22:59 MDT][err] S3 Key Not Found dexmaster/Checkout_Free_Books_(Amazon).user.js
[04/27 02:24:18 MDT][out] GitHub client authenticated
[04/27 02:26:18 MDT][out] Group(imgur) Rating Updated
[04/27 02:30:17 MDT][out] Group(powdertoy) Rating Updated
[04/27 02:30:22 MDT][out] Group(Shortcuts) Rating Updated
[04/27 02:31:47 MDT][err] S3 Key Not Found trespassersw/google_cache_comeback.user.js

---

See also:

• Martii/UserScripts@4b5abc0 (PASS on OUJS source page)

---

Will rollback to v0.1.7-58 snapshot if current usable functionality is hindered or continual server restarts with posted logs.

[Martii]

There's a major consequence to doing this... if script._id is ever used as the key in S3, then if the DB ever needs to be "moved" somewhere all of the _ids could be different depending on how it was restored. If it's a DB backup and then a restore then it might be okay if it preserves the same _ids; but if it's a copy/delete op all of the _ids will be different and all source will be orphaned.

NOTE: This is happening less and less as time progresses and fixes roll through.