Skip to content

Latest commit

 

History

History
407 lines (306 loc) · 14 KB

README.md

File metadata and controls

407 lines (306 loc) · 14 KB
Raw

---
title: nonplain
description: Plaintext files, with metadata
long-description: This is a library for parsing, manipulating, and exporting plaintext files with metadata stored as YAML or JSON frontmatter.
---

nonplain

Plaintext files are commonly used for notes, code, and documentation. Plaintext files are nondescript by definition: only their content and their filenames describe them. Jekyll popularized YAML frontmatter to enrich plaintext files for static site generation. These "nonplain" files have proven useful in other contexts requiring metadata, such as notetaking.

One drawback of using frontmatter in plaintext files is that there are few general-purpose tools for parsing and operating on these files' metadata and body content separately. The goal of nonplain is to make plaintext files with metadata easier.

Contents

What this library does

[link to toc]

The concept: define the difference between metadata and body content and parse the file accordingly.

Once the file is parsed, the metadata and body can be read, transformed, and exported together or separately to accomplish various goals such as:

  • analyzing files according to metadata
  • compiling relevant files for pagination
  • converting files to some other, less plain format
  • whatever else you want to do

In order to get there, we need to:

What a nonplain file is

[link to toc]

A "nonplain" file is any plaintext file that contains metadata as frontmatter. It's not really a file format, but rather a way to think about files of any plaintext format that begin with frontmatter.

What frontmatter is (the metadata)

In the future, this may be more customizable. For our purposes, frontmatter is a "fence" of 3 dashes --- on the first line of the file, followed by valid JSON or YAML beginning on the next line, followed by a final fence of 3 dashes --- on the line after the last line of JSON or YAML data.

It looks like this:

---
{
    "what is this?": "it's JSON frontmatter"
}
---

or this:

---
syke: now it's YAML
---

What the body is (the content)

The body is everything after the frontmatter.

When the file is put together, it looks like this:

---
{
    "what is this?": "it's called JSON"
}
---

This is the body of
the first file

or this:

---
syke: now it's YAML
---

This is the body of
the second file

Parsing nonplain files

[link to toc]

To parse a nonplain file, load it using the Files class. If you only want to operate on a single file, you can still use the Files class or you can use File instead.

Using Files:

const Files = require("nonplain").default;

// you can use a glob or a filepath
const files = new Files().load('/path/to/dir/**/*.md');

console.log(files.collect());

// Output:
//
// [
//     {
//         "body": "This is the body of\nthe first file",
//         "metadata": {
//             "file": {
//                 "root": "/",
//                 "dir": "/path/to/dir",
//                 "base": "file1.md",
//                 "ext": ".md",
//                 "name": "file1"
//             },
//             "what is this?": "it's JSON frontmatter",
//         }
//     },
//     {
//         "body": "This is the body of\nthe second file",
//         "metadata": {
//             "file": {
//                 "root": "/",
//                 "dir": "/path/to/dir",
//                 "base": "file2.md",
//                 "ext": ".md",
//                 "name": "file2"
//             },
//             "syke": "now it's YAML",
//         }
//     }
// ]

Using File:

const { File } = require("nonplain");

const file = new File().load('/path/to/file.md');

console.log(file.getData());

// Output:
//
// {
//     "body": "This is the body of\nthe current file",
//     "metadata": {
//         "file": {
//             "root": "/",
//             "dir": "/path/to/dir",
//             "base": "file.md",
//             "ext": ".md",
//             "name": "file"
//         },
//         "course number": "CS231n",
//         "description": "Convolutional Neural Networks for Visual Recognition",
//         "semester": "Spring 2020"
//     }
// }

Notice that the metadata of each file includes a file property. This property is included by default to denote the original source file. This property can be changed or removed by transforming the data using transform().

Transforming nonplain file data

[link to toc]

You may want to transform nonplain file data in place once it's loaded into an instance of File or Files. That's what the transform() method is for.

transform() receives a callback argument which is called with the current file data (for file.transform()) or iteratively through each loaded file (files.transform()). There are two options for this callback argument:

  1. Traditional callback function: 
    files.transform((file) => {
    const { body: oldBody, metadata: oldMetadata } = file;
    
    const newBody = oldBody.replace('this', 'that');
    
    const newMetadata = Object.assign(oldMetadata, {
      newKey: 'My new value for the file called ' + oldMetadata.file.name,
    });
    
    return {
        body: newBody,
        metadata: newMetadata,
    };
});
  2. Callback map: 
    files.transform({
    body: (oldBody) => {
        const newBody = oldBody.replace('this', 'that');
        
        return newBody;
    },
    metadata: (oldMetadata) => {
        const newMetadata = Object.assign(oldMetadata, {
          newKey: 'My new value for the file called ' + oldMetadata.file.name,
        });
        
        return newMetadata;
    },
});

transform() works the same way on both File and Files. The transform() method makes these changes "in place", meaning that your instance of File or Files will reflect the new file data after the transformation.

Possible uses for transform() might be converting content from markdown to HTML, calculating and injecting helpful metadata (such as VimWiki backlinks), and more.

Exporting nonplain file data

[link to toc]

Once file data is transformed to your liking, it needs to be exported and used elsewhere. That's where the file.write() and export2JSON() methods come in.

File.prototype.write()

Every instance of File has a write() method. This is the File.prototype.write() API:

file.write(file [, options])
  • file: string, Buffer, URL, or integer file descriptor - Destination where the file will be written. Using a file descriptor behaves similarly to Node.js' fs.write() method.
  • options:
    • body (default: true): boolean - Whether to write the body to the destination file.
    • metadata (default: true): boolean - Whether to write the metadata to the destination file.
    • fmFormat (default: 'yaml'): 'yaml', 'json', or config object - The format to use when writing the destination file's frontmatter.
      • fmFormat config object API:
        • format (default: 'yaml'): 'yaml' or 'json' - The format to use when writing the destination file's frontmatter.
        • space (default: 4): integer - The indentation to use, in spaces.
    • transform: function ((file) => newFile) - A callback function to transform file data before stringification.
    • replace: function ((content) => newContent) - A callback function to transform file content after stringification and before the new file is written.
    • encoding (default: 'utf8'): string - The encoding in which to write the destination file. More on this...
    • mode (default: 0o666): integer - The file mode when writing the destination file. More on this...
    • flag (default: 'w'): string - The flag used when writing the destination file. More on this...

export2JSON()

Files can also be exported to JSON using the export2JSON() method. The export2JSON() method exists on instances of both File and Files. file.export2JSON() will export the current file data to JSON and files.export2JSON() will export an array containing all of the currently loaded files' data to JSON. This is the export2JSON() API:

files.export2JSON(file [, options])
file.export2JSON(file [, options])
  • file: string, Buffer, URL, or integer file descriptor - Destination where the file will be written. Using a file descriptor behaves similarly to Node.js' fs.write() method.
  • options:
    • space (default: 4): integer - The indentation to use, in spaces.
    • transform: function ((file) => newFile) - A callback function to transform file data before stringification.
    • encoding (default: 'utf8'): string - The encoding in which to write the destination file. More on this...
    • mode (default: 0o666): integer - The file mode when writing the destination file. More on this...
    • flag (default: 'w'): string - The flag used when writing the destination file. More on this...

Other useful methods

[link to toc]

Files.prototype.clear()

Clears all currently loaded files from the Files instance.

Files.prototype.filter()

function filter(file) {
  if (file.metadata.public) {
    return true;
  }
  
  return false;
}

files.filter(filter);

Filters file instances in place using a filter callback. If the filter callback returns true for a given item, the item is kept. If the filter callback returns false for a given item, the item is discarded from the Files instance collection.

There is no return value; this operation is executed in place on the existing collection.

Files.prototype.sort()

function compare(a, b) {
  return b.metadata.date - a.metadata.date;
}

files.sort(compare);

Sorts file instances in place using a comparison callback. If the comparison callback returns a negative number, a is sorted before b. If the comparison callback returns a positive number, b is sorted before a. If the comparison callback returns 0, the original order is kept.

There is no return value; this operation is executed in place on the existing collection.

Files.prototype.collect()

Returns all currently loaded files as an array of file data:

const files = new Files().load('/path/to/dir/**/*.md');

console.log(files.collect());

// Output:
//
// [
//     {
//         "body": "This is the body of\nthe first file",
//         "metadata": {
//             "file": {
//                 "root": "/",
//                 "dir": "/path/to/dir",
//                 "base": "file1.md",
//                 "ext": ".md",
//                 "name": "file1"
//             },
//             "what is this?": "it's JSON frontmatter",
//         }
//     },
//     {
//         "body": "This is the body of\nthe second file",
//         "metadata": {
//             "file": {
//                 "root": "/",
//                 "dir": "/path/to/dir",
//                 "base": "file2.md",
//                 "ext": ".md",
//                 "name": "file2"
//             },
//             "syke": "now it's yaml",
//         }
//     }
// ]

Files.prototype.collectInstances()

Returns all currently loaded files as an array of File instances. Primarily used to iteratively call File methods, such as file.write().

File.prototype.getData()

Returns the currently loaded file data:

const file = new File().load('/path/to/file1.md');

console.log(file.getData());

// Output:
//
// {
//     "body": "This is the body of\nthe current file",
//     "metadata": {
//         "file": {
//             "root": "/",
//             "dir": "/path/to/dir",
//             "base": "file.md",
//             "ext": ".md",
//             "name": "file"
//         },
//         "course number": "CS231n",
//         "description": "Convolutional Neural Networks for Visual Recognition",
//         "semester": "Spring 2020"
//     }
// }

Related work

[link to toc]

Other libraries providing simple, composable tools for working with stuff like VimWiki notes are in the works. Stay tuned for more.

Contributing

[link to toc]

Nothing is set in stone right now; this concept is very much a work in progress. Please feel free to contact me with suggestions or ideas. Thanks!