A tool which can find your any document using semantic search.
This is an Improvised Version of Intelligent-Document-Finder
List of New Features--
- Implemented Document Similarity Script, which allows you to see related or most similar documents.
- Revamped website UI.
- Reduces time complexities of search functions.
How easy do you find it to remember the exact location of a document that you created last year? Not very easy, right? Big Organizations/people deal with hundreds of documents daily and forget about them, most of the time.
But what if we want that old documentation again for some work, but unfortunately you do not remember the name or the actual content of that document to retrieve it from the large storage of your computer.
In such cases, use of a Intelligent document finder can really make a huge difference. As, it can Search for the document(semantically) of your need based on a query input. This will not only help in faster access to the document, but will also help in grouping similar documents together and in analysing them.
Currently this repositry is using predefined database of news articles gathered by web scraping. Due to the github restrictions on uploading the large files, we cannot upload it here.
Soon, we will add the support of the dynamic databases, so that you can use this tool for your own databases to build your own custom search engine.
Python3.6
JavaScript
jQuery
HTML & CSS
Various NLP(Natural Language Processing) techniques is used.
- Python-based Web framework : Flask
- JavaScript
- jQuery
- Backend (AI part) is compatible on any machine that has python and required dependencies installed.
- Recommended browsers: Mozilla Firefox and Google Chrome.
> mkdir IntelligentDocumentFinder
> cd IntelligentDocumentFinder
> git clone https://github.com/Sarthakjain1206/Intelligent_Document_Finder_2.0.git
Install Vitual Environment if not installed
- On Linux/MacOs
> python3 -m pip install --user virtualenv - On windows
> py -m pip install --user virtualenv
Create Virtual Environment
- On macOS and Linux:
> python3 -m venv env - On Windows:
> py -m venv env
Activate Environment:
- On macOS and Linux:
> source env/bin/activate - On Windows:
> .\env\Scripts\activate
> pip install -r requirements.txt
Download Glove Word Embeddings from this link, decompress it and copy the glove.6B.100d file in DataBase folder
then,
run initial_file.py through this command
> python initial_file.py
Now you are good to go.. Just type this command everytime you want to access it, and open the website in chrome/firefox
> python src/app.py
You can get in touch with us on linkedln profiles
Sarthak Jain Machine Learning NLP Web Crawling
You can also follow me on Github to stay updated about my latest projects
Rishabh Mishra Full Stack Web Developer
You can also follow me on Github to stay updated about my latest projects
If you liked this repository, then do support it by giving it a star
There are a lot of features that can be added to this tool.
- Query Segmentation
- Query Expansion (Mainly - Pseudo Relevance Feedback technique)
- Improvising Spell Checker
- Collocations For example- Currently this project consider "New York" as ["New","York"] i.e two different words but it should be consider as a single entity like ["New_York"], this can definitely make a big difference in search results.
- Query Logs (Game changing technique for search engines)
- Search result's segmentation [like- Luecene]
If you have any experience in implementing any of these features then, do contribue.
-
Awsome article of BM25 ranking algorithm on wikipedia - Okapi BM25
-
Read this article on Topic Modeling
-
Completely followed this beautiful article on SVOs tagging for generating tags for this project.
-
Used the BM25 ranking fuction implementation from this great repositry on github by
dorianbrown.