Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to search through Chinese #10

Open
frank-lam opened this issue Jun 4, 2020 · 5 comments
Open

Unable to search through Chinese #10

frank-lam opened this issue Jun 4, 2020 · 5 comments
Assignees
@lelouch77
Labels
enhancement New feature or request new language support

Comments

@frank-lam
Copy link

Unable to search through Chinese, I hope you can help to fix it
@lelouch77
Copy link
Collaborator

Hi @frank-lam, I added support for multi-languages using lunr-languages, but there is no support for Chinese there yet, but I tried using the code from this open PR MihaiValentin/lunr-languages#53 it uses nodejieba, but I am facing issue in installing it, check out this brach zh-support, this is what I tried.

Thanks.

@lelouch77 lelouch77 self-assigned this Jun 9, 2020
@lelouch77 lelouch77 added enhancement New feature or request new language support labels Jun 9, 2020
@frank-lam
Copy link
Author

frank-lam commented Jun 9, 2020 •
edited
Loading

@lelouch77 Thanks! I am a backend engineer. I am not skilled in node and javascript. So, could you please merge the branch into master, and push zh-support to npm repository, enhance chinese feature. Then it can easy to use for everyone, just npm install xxx, and use it! Thanks a lot!

@lelouch77
Copy link
Collaborator

lelouch77 commented Jun 9, 2020 •
edited
Loading

Ya, I did published a separate version (2.0.0-zh1) for it, try npm install docusaurus-lunr-search@2.0.0-zh1

@frank-lam
Copy link
Author

frank-lam commented Jun 9, 2020 •
edited
Loading

@lelouch77
when I run npm install docusaurus-lunr-search@2.0.0-zh1 , some error happen.

> node-pre-gyp install --fallback-to-build

node-pre-gyp WARN Using request for node-pre-gyp https download
node-pre-gyp WARN Pre-built binaries not installable for nodejieba@2.4.1 and node@12.13.0 (node-v72 ABI, unknown) (falling back to source compile with node-gyp)
node-pre-gyp WARN Hit error connect ETIMEDOUT 52.74.223.119:443
gyp ERR! find Python
gyp ERR! find Python Python is not set from command line or npm configuration
gyp ERR! find Python Python is not set from environment variable PYTHON> node-pre-gyp install --fallback-to-build

node-pre-gyp WARN Using request for node-pre-gyp https download
node-pre-gyp WARN Pre-built binaries not installable for nodejieba@2.4.1 and node@12.13.0 (node-v72 ABI, unknown) (falling back to source compile with node-gyp)
node-pre-gyp WARN Hit error connect ETIMEDOUT 52.74.223.119:443
gyp ERR! find Python
gyp ERR! find Python Python is not set from command line or npm configuration
gyp ERR! find Python Python is not set from environment variable PYTHON

Is there easy way to use this plugin... so crazy for me.

@lelouch77
Copy link
Collaborator

yes it is due to nodejieba dependency, I am not sure how to fix it ...

anru added a commit to anru/docusaurus-lunr-search that referenced this issue Jul 5, 2020
@anru
Fix indexing for non-ascii text. Considerably accelerated processing …
cbf52ee 
…of html files

The main changes:
- Deleted cheerio: it incorrectly extracted non-ascii text
- Instead, [rehype] (https://github.com/rehypejs/rehype) is used, which handles HTML processing perfectly.
- The most laborious work on processing HTML is carried out in a thread pool. It has accelerated processing of my documentation (lunr's index file takes 16 megabytes in my case) from 53 seconds to 16 seconds.
- lunr.client.js is now generated in the `@generated` directory for greater consistency. Writing files directly into a package in node_modules is not the most correct (and safe) approach.

This PR probably could resolve praveenn77#10 issue.
For me these changes fixes issue with indexing russian language.
@anru anru mentioned this issue Jul 5, 2020
Merged
anru added a commit to anru/docusaurus-lunr-search that referenced this issue Jul 5, 2020
@anru
Fix indexing for non-ascii text. Considerably accelerated processing …
bb42baf 
…of html files

The main changes:
- Deleted cheerio: it incorrectly extracted non-ascii text
- Instead, [rehype] (https://github.com/rehypejs/rehype) is used, which handles HTML processing perfectly.
- The most laborious work on processing HTML is carried out in a thread pool. It has accelerated processing of my documentation (lunr's index file takes 16 megabytes in my case) from 53 seconds to 16 seconds.
- lunr.client.js is now generated in the `@generated` directory for greater consistency. Writing files directly into a package in node_modules is not the most correct (and safe) approach.

This PR probably could resolve praveenn77#10 issue.
For me these changes fixes issue with indexing russian language.
@lelouch77 lelouch77 reopened this Jul 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lelouch77 lelouch77

Labels
enhancement New feature or request new language support
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lelouch77 @frank-lam