How to use with cloud storage like cloudflare stream or s3

[yukebrillianth]

[Dan1ell]

S3 support is coming, apparently.

[mgogoulos]

S3 support is on the way, on the meanwhile s3fs could be used to mount an S3 bucket as a local filesystem. It needs only a few commands and it works - though one can argue it's a hacky way to use S3. Proper support for S3 should be anytime in the next 3 months

[remyzv]

Hello @mgogoulos do you have any update for the S3 support ?

[hemel-cse]

it would be great to have a doc on it.

[sigurdurdahl]

FWIW we would also welcome direct support for S3 (or in our case CEPH based object storage compatible with S3) to make horizontal scaling easier.

Does the plan for S3 include letting the S3-service serve the files directly?

br,
Sigurd

[jared252016]

I currently have a working implementation of S3 (using Storj.io) with mediacms for a customer. All media/static files are served from Storj and not from my server, which should help a good bit on CPU/Memory constraints. Storj allows you to map a subdomain to their services, so https://media.example.com/static/ and https://media.example.com/site-id/ both work well, and I am able to share static resources among sites.

I am actually testing for a multi-site setup, about 22 or so to be exact, all running mediacms via docker-compose with a shared postgresql db, shared redis (I think), and remote workers.

I have noticed the code is somewhat hit or miss and is not friendly on bandwidth to/from external storage or remote workers just yet. I was able to get uploads to work faster by combining the chunks locally and then uploading to S3. I'm currently working on remote workers, where the file is obtained from S3, chunked, manipulated, and all modifications/variants are uploaded back to S3.

[mgogoulos]

Very interesting! There have been attempts in the last few months for cloud storage but nothing is committed yet. You are very welcome to contribute too!

[destined2fail1990]

Once I get it polished out a bit more I will likely contribute it back. If anything it will be public to serve as a basis for ideas.

My implementation is slightly different because I'm prepending a site id to the bucket path so that I can use one shared bucket and simplify deployment.

Thus far I've probably put a good 32 hours into it, so it was no easy task. I filled the relevant files with print statements though to see where it was hanging up and to monitor variables - something that will no doubt need to be cleaned up prior to a PR

This is my first django application so there has been a bit of a learning curve, but I think I finally got the hang of it.

I plan to implement social login / single sign on through keycloak also, so may have some modifications there as well.

The sites I am running are for a customer, and aren't necessarily appropriate for everyday conversation, but are legal nonetheless. The idea is to be able to sell memberships to the site, allowing only a certain number of sites to be accessed. Each site has a different topic, such as horses or cars.

The content is all curated from the internet automatically and is in the public domain. Mostly from a Twitter Bot monitoring profiles and downloading media / hash tags for popular topics.

The content itself will then be likes/disliked and tagged by users to feed a dataset for machine learning. Drawing insights such as scenery, human actions, emotion, and interactions.

I really missed out not having the time/money to start up a social media platform like Tumblr. They sit on some of the largest datasets in the world. Luckily Twitter makes their API available and it's not feature restrictive like Facebook.

Not sure what datasets go for, but I'd like to sell some of them to businesses working on APIs for recognition.

Sorry didn't mean to derail the thread with my project. I'll be around throughout the coming months. Hopefully with some contributions.

[swiftugandan]

Looking forward to the contribution @destined2fail1990

[streamly]

Any details on using s3fs or any other mounting that works? I tried mounting using rclone and goofys and neither worked. Not a developer so not sure what to do to get it working.

Any guidance would be appreciated. Trying to mount backblaze or s3 at least.

Also, any update on s3 integration? See that uploader used by media cms has released some s3 capabilities.

thx

[streamly]

I got the s3fs working but its really slow compared to regular upload. Not sure why?

For others wanting to try mounting;

I followed these; https://help.backblaze.com/hc/en-us/articles/360047773653-Using-S3FS-with-B2
make sure you do not use the masterkey but create special app keys
follow these https://stackoverflow.com/questions/17544139/allowing-permission-using-s3fs-bucket-directory-for-other-users
then run; use you own bucket creds

sudo s3fs xxxxxx /home/mediacms.io/mediacms/media_files -o passwd_file=/etc/passwd-s3fs -o url=https://s3.us-west-001.backblazeb2.com -o allow_other

[swiftugandan]

This project looks interesting - https://github.com/juicedata/juicefs

• only issue may be the fact that you cannot use a CDN to serve the files directly to the clients. The host needs to read file metadata from a db, retrieve the file chunks from the object store, and serve the file back to the clients. This implies bandwidth and processing costs for each host if on a public cloud.

[streamly]

I decided not to use mediacms and coded my own using uppy.io, tusd server and backblaze. These could be integrated with mediacms and supports both local and cdn storage.

[jledesma-opsguru]

Hi guys. I just want to know the current status of video storage feature in S3.

[mgogoulos]

It's still not supported, there's an experimental PR but has been stale, and I've never committed my own work for S3 support.
Also not sure about the status of s3fs (whether it has been any faster, since last time mentioned in this thread there were complains that it was slow)

[spshukla]

@mgogoulos Any plans on this?

[ricardoteix]

I finally managed to use S3 and CloudFront with MediaCMS. It is not a final solution, but still I can use a CloudFront to play HLS media.
I've mounted the media_files directory to a S3 bucket using s3fs and created a CloudFront distribution with it as origin. With it I set MEDIA_URL to CloudFront url.
Some images are not working properly yet, but almost all things are good.

[mgogoulos]

Would be interested to hear about your experience with it so far!

[ricardoteix]

I am new with mediacms and I am using it as as challange to provide a solution to an fictional enterprise in an AWS course.

So I want to use as much AWS services as possible. I see AWS S3 and Cloudfront as two mandatory things as we want a cost-effective solution. S3 to storage of all videos and Cloudfront as a media CDN to deliver to end users. So I have these two challanges.

I can not found another way to upload direct do S3. Once I used a TUS server but I could not integrate it with mediacms, neither using the AWS Elemental MediaConvert to generate HLS files and save them to mediacms.

The way I found was to use s3fs to mount a S3 bucket in the /home/mediacms.io/mediacms/media_files directory. This worked well and every file added or removed to this location was automatically synced to S3 bucket.

To serve the files I have created a Cloudfront distribution with the bucket as origin and changed the MEDIA_URL variable in settings.py to the URL of the Cloudfront. With this the media are served by Cloudfront.

Joinig the two now I have my solution to store and serve with AWS S3 and Coudfront. The mediacms was installed in a t3a.small (2 vCPu , 2 GB RAM) with 8 GB of EBS disk storage and Ubuntu 22.04.

The problem I saw was loading some miniatures that uses a diffent approach to point to the image file. It concatenates the mediacms domain with Cloudfront domain and them we get a erro.

In addition to these services I used others AWS services like: Elasticache (Redis), SES (Simple Email Service), WAF, LoadBalancer etc.

[destined2fail1990]

I did a PR a long time ago that was messy, still full of print statements as I was trying to determine what was happening and when. The upload process isn't ready for S3 buckets. It needs to do local processing and then dump the final product into S3, otherwise your S3 bandwidth will be double if not tripple because it has to re-encode the files pulling them again. If it's still there and someone wants to clean it up, I had it functional using Django's S3 support and not needed a fs mounted.

…

On Thu, Jul 20, 2023, 2:16 PM Ricardo Teixeira ***@***.***> wrote: I finally managed to use S3 and CloudFront with MediaCMS. It is not a final solution, but still I can use a CloudFront to play HLS media. I've mounted the media_files directory to a S3 bucket using s3fs and created a CloudFront distribution with it as origin. With it I set MEDIA_URL to CloudFront url. Some images are not working properly yet, but almost all things are good. — Reply to this email directly, view it on GitHub <#185 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUJCOTTLJD6WY7GT37B6DTDXRF7Z3ANCNFSM45IMCBPQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ricardoteix]

I don't know Django nor mediacms source code well, but I can not understand why the bandwidth problem. Why it has to re-encode the files? Can you explain?

[destined2fail1990]

MediaCmS serves files in HLS format and actually at multiple file sizes so that it can adapt to the internet connection being used. I personally turn off everything but 720p in the mediacms admin. Basically the files need to be in this format to be properly streamed, but to also reduce the number of users downloading your videos should you disable that feature. The problem arises with how those files are temporarily stored. Yes, you can mount an S3 bucket to the media folder on the file system, but the number of read/writes will be expensive as it reads for each re-encode. The proper setup would be to cache to local disk without Django and then the final destination be the Django media storage, which can be set to an S3 bucket in Django's configuration, no need to use s3fs. I may revisit this if I have time and try to re-sort out where the files are stored and when, then commit it. Hopefully I still have my pull request somewhere and can just clean it up. MediaCMS needs better debug logging too so that it's easier to trace what's going on in the background. I had print statements everywhere trying to sort it out myself which is why the pull request was rejected, but I was also in a hurry.

…

On Mon, Jul 31, 2023, 3:11 PM Ricardo Teixeira ***@***.***> wrote: I don't know Django nor mediacms source code well, but I can not understand why the bandwidth problem. Why it has to re-encode the files? Can you explain? — Reply to this email directly, view it on GitHub <#185 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUJCOTX6SWCV3TWYIYWHL4DXTAGQDANCNFSM45IMCBPQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ricardoteix]

Thank for your explanation. Indeed, a native implementation without workarounds like s3fs I using would be great.

About the S3 cost to re-encode, I think it will not ocurr the way you pointed because the archtecture uses both S3 and EC2 in same region and AWS says that:

"You pay for all bandwidth into and out of Amazon S3, except for the following:

• Data transferred from an Amazon S3 bucket to any AWS service(s) within the same AWS Region as the S3 bucket (including to a different account in the same AWS Region)."

So when my EC2 in a private subnet gets a file from S3 it must not be charged because the file was not passing throgh the publica address. I must pay only by the requests GET, PUT, HEAD etc.

[jared252016]

Yeah, that makes sense then with Amazon S3. I was using Storj which also had an S3 bucket option but bills for outbound bandwidth with inbound being free, which wasn't feasible. I started noticing the cost skyrocketing for bandwidth rather quickly. I'm not sure how the speed is with Amazon S3. S3fs actually supports a local cache that should alleviate the pain problems with bandwidth (if anything to speed things up), but can fill up storage rather quickly. A script to manage the s3fs cache in cron may do what you need to do at scale. If it's just a demo I don't see you uploading too much, but in the event you have thousands of videos it would matter.

…

On Mon, Jul 31, 2023, 4:29 PM Ricardo Teixeira ***@***.***> wrote: Thank for your explanation. Indeed, a native implementation without workarounds like s3fs I using would be great. About the S3 cost to re-encode, I think it will not ocurr the way you pointed because the archtecture uses both S3 and EC2 in same region and AWS says that: "You pay for all bandwidth into and out of Amazon S3, *except* for the following: - Data transferred from an Amazon S3 bucket to any AWS service(s) within the same AWS Region as the S3 bucket (including to a different account in the same AWS Region)." So when my EC2 in a private subnet gets a file from S3 it must not be charged because the file was not passing throgh the publica address. I must pay only by the requests GET, PUT, HEAD etc. — Reply to this email directly, view it on GitHub <#185 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASLNIDDWOXPHDBG4FXKS2H3XTAPRZANCNFSM45IMCBPQ> . You are receiving this because you commented.Message ID: ***@***.***>

[swiftugandan]

Has anyone used https://github.com/awslabs/mountpoint-s3?

[ricardoteix]

I did some tests. I realized that the proposal is basically what s3fs does, to mount a bucket in a local folder on Linux. I believe the foundation is the same as Fuse, just like s3fs, but with some differences. I tried replacing s3fs with mount-s3 in the project I did with the MediaCMS.

Unfortunately, it wasn't compatible because the append operation on the file is not allowed. This is actually mentioned in the documentation, where it says it's not recommended to "make edits or appends to existing files in-place."
So, it's possible to use it to create a file in the mounted folder:

echo demo1 > demo.txt

But you can't add something to an existing file, for example:

echo demo2 >> demo.txt

[channchetra]

Has someone tried this : https://github.com/s3fs-fuse/s3fs-fuse?

[ricardoteix]

I’ve made some tests, but not for production.

Look this repo. It has a Terraform project to put it up and running. Or take a look at the .sh file that mount the s3fs.

[tobocop2]

I relied on s3fs and ultimately abandoned it in favor of just eventual consistency using a cron job that sync's the media files directory to s3. I found this approach to be much faster and reliable than s3fs. Using the MEDIA_URL setting in django, you can point this to be a cloudfront distribution even and you can avoid the need to serve files from the local file system at all once things are sync'd to s3.

[rsisto]

Quick question, in your deployment scenario, the files are always duplicated in media_files and s3, right?

[tobocop2]

yes, but I used EFS and the files and set a short lifetime on the files so eventually the only data that remains in s3.