Improve collection loading for huge websites

[DavidWells]

Just tested out a 1000 page site. Generated them with this handy script: https://gist.github.com/DavidWells/a75045c89d6f3d453ef4ab68499550f4

Good news, the build only took like 60 seconds!

The generated site was 296mb in total though. This was primarily due to every page inlining the entire window.__COLLECTION__ into the html. (https://gist.github.com/DavidWells/675b1b838c44925d6487fff3cc129674)

The window.__COLLECTION__ makes html files weight 300k vs 8k

Perhaps there is a way to lazy load in the routing data and stick it inside localstorage or a service worker?

Or build the window.__COLLECTION__ into a single collection.js file and include that once in the head tag? Then it could be cached and would make all the html files lighter.

[thangngoc89]

Hi, thanks for sugessting this. The collection used to be in a seperated collection.json file and we used to load it async on initial load but it introduces some caveats.

@MoOx I think dumping it to collection.js and loading it in a blocking way would be the best. (Or non-blocking way if you can fix previous problems)

@DavidWells i tried phenomic on a REAL 10k pages site. The build time is about 8mins and the generated site is 800MB.

[thangngoc89]

Well, why did we inject collection to html

#92

[MoOx]

We injected collection for better consitency between client & static build. And for small website, extra http request is not necessary. Obviously for large website it might seems huge, but at the end a single user is supposed to access a single html per session.
We could add a non lazy load of the collection via any method, but the collection should be mandatory for the client sync/rendering.

[DavidWells]

Maybe there could be an option to inline or include collections via:

<script src"/collestions.js?hashXyzydhhs231313123123" async></script>

Larger sites could opt for script tag and save on overall site size.

(in my 1000 page test) Each inlined collection adds 292kb to the file size. So an addition 292mb for a site at 1000 pages.

With the script tag it would be the single 292kb and the deploy/storage would be sooooo much smaller =)

[MoOx]

That's definitely something we should do. We could even ask the user during the deployment if we want collection to be outside html (if let say collection > 150kb?) and save the pref in package.json for future build?

[DavidWells]

This will make larger sites way more viable and cost friendly

[DavidWells]

@MoOx I'd like to take a stab at adding this option.

Would here:

phenomic/src/builder/server.js

Lines 137 to 141 in 1a11c7f

	<script>
	window.__COLLECTION__ = ${
	serialize(collectionMin)
	};
	</script>

be the correct place to add the logic to take the collection and add it to script tag instead of inlining?

[thangngoc89]

@DavidWells I think it's best to leave it there. It's the dev server. For static build, you should change this line

phenomic/src/static/to-html/url-as-html.js

Lines 108 to 111 in 2b44e19

	script =
	`window.__COLLECTION__ = ${
	serialize(collectionMin)
	};` +

(extract it to a external js file instead of inline it)

[MoOx]

You will also have to emit a collection.json file from Phenomic loader in addition to previous additions you both mentionned.

[thangngoc89]

@MoOx it should be collection.js. Let's the browser load that synchronously for us.

[MoOx]

Btw, we should replace window.__* by window.__PHENOMIC__.* or something like that... Maybe just window.Phenomic.*.

[thangngoc89]

I prefer window.__Phenomic.* . Who know what can a user put to the page.

[MoOx]

Then, window.__PHENOMIC_DONT_USE_THIS_OR_YOU_WILL_BE_FIRED__.*.

More seriously, I will probably use window.__PHENOMIC_PRIVATE_REGISTRY_YOU_SHOULD_NOT_RELY_ON__*.

[thangngoc89]

@MoOx

window.PHENOMIC_DONT_USE_THIS_OR_YOU_WILL_BE_FIRED.*.

this is from React code base right ? I like the idea but we referenced this at many places. I don't want to type this everytime

[MoOx]

I plan to add public methods like import { getCollection } from "phenomic".

[thangngoc89]

Great idea !

[thangngoc89]

@DavidWells any update?

[DavidWells]

@thangngoc89 how do you think we should implement this?

Are config options available in this context?

phenomic/src/static/to-html/url-as-html.js

Lines 108 to 111 in 2b44e19

	script =
	`window.__COLLECTION__ = ${
	serialize(collectionMin)
	};` +

I can see a couple ways to turn this on:

• A command line arg phenomic build --smallHTML(whatever)
• A config option in package.json
• A webpack config flag?

What do you think?

[thangngoc89]

This should be a phenomic's config. FYI all configs in Phenomic can be altered via a cli flag or an entry in package.json/phenomic.

For adding a new config, see this file https://github.com/MoOx/phenomic/blob/2b44e19bfa095d2a4bb512c122dbb24380a83ef1/src/configurator/definitions.js

If you need some validation, add one in validators folder

[thangngoc89]

As for accessing the config, this func

phenomic/src/static/to-html/url-as-html.js

Line 17 in 2b44e19

export default function(

have access to PhenomicConfig object

It can be re-writtten like this: export default (url: String, config: PhenomicConfig, testing?: Boolean)

I hope these information are helpful

[bloodyowl]

Going to be fixed with #925