slow write/append to files on ramfs

[justinc1]

While running nginx and testing achievable requests/s for its internal server_status page (e.g. no file access should be required for that), I noticed that req/s drops with elapsed time. Problem is that nginx was writing to access.log (so obvious). Problematic part in OSv code is https://github.com/cloudius-systems/osv/blob/master/fs/ramfs/ramfs_vnops.cc#L393:

			void *new_buf = malloc(new_size);
			...
			if (np->rn_size != 0) {
				memcpy(new_buf, np->rn_buf, vp->v_size);
				free(np->rn_buf);
			}

In my case is 90 B long line written per HTTP request, and in short time this requires malloc/memcpy/free of about 50 MB.

[justinc1]

I'm not sure if/how this should be fixed. I tried to reduce number of malloc calls (so that new_size is at least 64 kB, then power of 2 for up to say 4MB, and multiple of 4 MB for larger file sizes). It helps to me - with 4MB magic value, I get about 8 malloc per second. Writes to access_log still have visible impact on achieved req/s (10% less than with nginx 'access_log off:' directive), but at least this is better than nothing.

Also, this wastes some memory. Since I already have debug printf at hand, I will try to do some math. Below, first number is file count, second is new_size after rounding up. In worst case, 64 kB files are only 1B large, and 128 kB are 65537 B large etc => then about 20 MB is wasted. Realistic would be half of that, I guess.

    195 65536
     20 131072
     12 262144
      5 524288
      5 1048576
      2 2097152

Nicer might be to implement something iovec-like, but I'm not sure it is worth the effort (I can use .'access_log off:').

[nyh]

Good spot. Indeed, reallocating the entire file contiguously in memory is a really bad idea for the reasons you noted and also because it requires contiguous memory which may be hard to find after long runs.

A really quick hack to fix the complexity problem you noted (that writing a N-byte file takes O(N^2) work) is to not increase the size by 4K each time, and not even 4M as you suggested, but to increase the file size by a factor - e.g., by 10% (1.1 of the previous file size) or 100% (double the file size). Then the writing an N-byte file becomes a O(N) operation as desired.

I don't think the memory waste is a big problem, considering we rarely ever append to files, and when we do, the file is anyway likely to further grow. If we just grow the file by 10% each time, we reach the same desired complexity, but the memory overhead will only be 10% which is perfectly fine, I think. We can/should probably use even more than 10% here.

An even more efficient approach can be to use a data structure which holds blocks of data instead of one contiguous allocation. C++ even has one standard data structure we could use: std::deque. Reads will be a bit slower, but I'm guessing this will hardly be noticable, but we can create our own data structure to make reads even faster (by allowing to copy an entire block, not byte by byte). All this is probably not worth the effort now. But it's worth at least adding a FIXME comment.

[wkozaczuk]

I think that growing 10% (or some other factor) is a reasonable idea and little coding. We could use realloc() to automatically copy the data in the most efficient way. But I think the ideal tool here would be mremap() - zero copying - which unfortunately is unimplemented. We do have a patch that would need to be be finished - https://groups.google.com/forum/#!msg/osv-dev/O_YMbt2bE9Q/X1XEN4DQ8WgJ.