feat: private dir

[dai-shi]

"./private" for files that can only be read on the server.

• waku dev
• waku start
• vercel
• netlify
• cloudflare: no node:fs
• partykit: no node:fs?
• deno
• aws-lambda: no node:fs?
• update packages/website with ./private

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
waku	✅ Ready (Inspect)	Visit Preview	Feb 22, 2024 1:42am

[codesandbox-ci]

This pull request is automatically built and testable in CodeSandbox.

To see build info of the built libraries, click here or the icon next to each commit SHA.

[aheissenberger]

@dai-shi no problem to implement this for aws lambda but :

• in a cloud deployment - e.g. in a lambda process I cannot write my database to the local filesystem (examples with ./private/db.json) - it will not be available! I will use e.g. unstorage and define s3/dynamodb as a driver.
• anything outside of the public folder is anyway not accessible through the static server
• if I need some files (read-only in cloud), I can already include their URL with import MyData from ../assets/mydatabase.json?url` which is handled by the existing vite bundling as an asset including creating a hash when content changes.
• I suggest for a future WAKU config to use the vite.config.ts and use a rollup plugin e.g. https://github.com/bengsfort/rollup-plugin-copy-assets to handle this cases

[dai-shi]

This feature is about "filesystem access" and not meant for a cloud deployment without filesystem.
This is a bit of regret for me, because the capability isn't consistent, but the use case with fs.readFile is fairly acceptable for Node.js env, and without this feature, it's not possible.

For cloud services without filesystem, we need bundling as you suggest, and I think it's already possible as of now, optionally using such plugins.

[aheissenberger]

have a look at unstorage - with this abstraction you can cover all your use cases, get a consistent api. There is already a huge list of plugins. If you miss something I can provide the code for the missing e.g. filesystem adapter as I have done for AWS S3 and dynamodb.

[dai-shi]

I mean, we don't prevent using such libraries. My understanding is it's already supported. Let me know if it doesn't work.
This PR isn't trying to provide a consistent api. It's limitedly for node:fs.

[aheissenberger]

I only want to help and not to extend this discussion, but you mention somewhere you would like to follow web standards. node:fs is not covert by web standards and not by the new WinterCG Standards for JS Server Side Runtimes.

I know unstorage is a product of the unjs universe where h3 is a similar product as Hono chooses for this project.
What I found is Hono Storage but I think unstorage is currently the winner in this segment :-).

There is no problem to implement this for AWS Lambda as it uses Node. But AWS Lambda Edge is similar to Cloudflare and some other Serverless provider only a browser runtime without any support for node:fs.

[dai-shi]

but you mention somewhere you would like to follow web standards.

Yeah, that's correct. So, this seems kind of exceptional.
More precisely, we follow web standards but also allow using Node.js, if users want.
It means, if this PR breaks cloudflare usage, it's not acceptable.

There is no problem to implement this for AWS Lambda as it uses Node.

Oh, I see. Yeah, if it's possible in an easy way, let's do that.

AWS Lambda Edge is similar to Cloudflare

I think it's the same situation with Vercel and Netlify. Vercel serverless functions can be Node based, but Vercel edge functions are not. Netlify functions are Node based, but Netlify edge functions are not. So, if we use AWS Lambda, not AWSLambda Edge, which is Node based, we can support node:fs usage.

I only want to help and not to extend this discussion

So, it was our miscommunication and does "we follow web standard, but also support Nodejs if users want, as long as it doesn't break web standard usage" sound okay?

[dai-shi]

moving forward for now.

[dai-shi]

We don't know if ./private/... path in the code is confusing or not.
If it's not quite acceptable by users, let's consider removing the capability.

[dai-shi]

Ah, found a typo.

[dai-shi]

#527

[dai-shi]

#514 (comment)

Maybe an alternative approach is to provide getPrivateDir from waku/server to abstract the actual path. It may solve #530 issue.

[aheissenberger]

#514 (comment)

Maybe an alternative approach is to provide getPrivateDir from waku/server to abstract the actual path. It may solve #530 issue.

Yes, this is an option but I would remove this feature - people who need a private read and writable data store, can implement it by them self and know the limits of their deployment environments. Nearly all cloud provide suggest to use a block storage (e.g. s3) or a shared KV store (Cloudflare, Deno deploy, ..) for this use case.
The existing examples, with single files, could use simple imports with the ?url to get the file path, avoid embedding the json into the bundled code and have the data copied to the dist folder.
If someone need access to many static files, the solution depends on the overall size of the folder to choose the right type of deployment. On AWS there is a limit on the deployment bundle size so I would copy the files to a private s3 block storage.

[dai-shi]

I would remove this feature

That's one option.
But that means to give up fs.readFile, right?

The existing examples, with single files, could use simple imports with the ?url to get the file path

Or, does it work with fs.readFile?
Do you know if it works with markdown files?

have the data copied to the dist folder.

Do you mean to manually copy files? That's exactly the feature this PR is trying to automate.

[aheissenberger]

Ok - the use case is a blog with a folder of markdown files which are not static rendered.

To be clear - the current release of WAKU with --with-aws-lamba and the serverless framework can handle the private folder and access to this folder with fs.readFile(./private/myfile.md) without any changes in the output-aws-lambda.js. The only handicap is that the private folder needs to be added to the serverless.ymlin thepackage` section to included into the final zip. For people using other deployment tools it will be similar. There is no manual copying required.

#535 is a minimal documentation on how to deploy to AWS Lambda with an example serverless.yml config file including support for ./private.

[dai-shi]

Thanks for the clarification.
So, it seems like the situation is similar to netlify functions.

We create netlify.toml file like this:

waku/packages/waku/src/lib/builder/output-netlify.ts

Lines 38 to 48 in ff20bfe

	if (!existsSync(netlifyTomlFile)) {
	writeFileSync(
	netlifyTomlFile,
	`
	[build]
	command = "npm run build -- --with-netlify"
	publish = "${config.distDir}/${config.publicDir}"
	[functions]
	included_files = ["${config.privateDir}/**"]
	`,
	);

[aheissenberger]

The problem with AWS is, that there are 3 official ways (AWS CLI, SAM, CDK) to deploy to the infrastructure and a whole bunch of other options (e.g. Serverless Framework, SST, Pulumi, Terraform,...).

Currently the builder for AWS creates an output which can be consumed by all of them. To support them is too much work and the user needs to adapt the config files to their environment on AWS and to the other infrastructure they want to create (e.g. DNS records, databases,...). This will only allow us to create a config once and does not allow any changes later.

I hope you agree, that the current documentation is a good start to get a project up and running including support for the private dir.

[dai-shi]

To support them is too much work

Can we pick one that best matches to Waku?
Like, as for Netlify, they have functions and edge functions, but we only pick "functions". They also support UI deploy and CLI deploy, and in this repo, we only support CLI deploy.

the current documentation is a good

Let me check the PR again, but in general, I just count on you, based on our conversation so far.

[aheissenberger]

People which look for a simple deployment platform will choose Vercel or netlify - this is their market position against AWS and Azur.
Choosing AWS requires a deeper background knowledge of their services or one of the deployment options to start.

The main reason I support this project is, that I am looking for a valid react rsc framework which supports AWS Lambda directly. Running NextJS in the AWS environment is not supported by Vercel and the existing external solutions break with small and big releases.