Skip to content

danielrosehill/RAG-Markdown-Loader

Repository files navigation

RAG Markdown Doc Preloader

13-Feb-25

A very simple Python GUI for copying Markdown documentation from a source to target filepath.

It does the following:

  • Adds folder names as prepended text before identical paths to facilitate disambiguating between filenames with a recurrent value as is common in documentation repositories (where there may many 'index.md' files all part of separate folders)
  • It creates a flat structure on the target path.
  • Finally markdown files with.mdx are converted to .md

I developed this very simple utility for preparing markdown documentation for upload to. Dify.ai - which at the time of creating this didn't support mdx.

The uniqueness constraint is intended to make it easier for the AI and the human to identify which file is which when checking how they embed it

The GUI was created with Claude Sonnet 3.5 from my prompt (dev-prompt.md). For once, it worked on the first go!

Screenshots

alt text

alt text

alt text

Purpose

Using this utility makes it quick and easy to upload documentation repositories to RAG systems such as that used in Dify.ai (backing vector storage: Weaviate)

alt text

Embedded!

alt text

Author

Daniel Rosehill
(public at danielrosehill dot com)

Licensing

This repository is licensed under CC-BY-4.0 (Attribution 4.0 International) License

Summary of the License

The Creative Commons Attribution 4.0 International (CC BY 4.0) license allows others to:

  • Share: Copy and redistribute the material in any medium or format.
  • Adapt: Remix, transform, and build upon the material for any purpose, even commercially.

The licensor cannot revoke these freedoms as long as you follow the license terms.

License Terms

  • Attribution: You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
  • No additional restrictions: You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.

For the full legal code, please visit the Creative Commons website.

About

Loads markdown documents into a folder

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages