Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix curl "missing url" and discrepancies in README #6

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Fix curl "missing url" and discrepancies in README #6

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
# Zero Trust Search Engine

This repository contains all the code nessecary to create your own version of https://search.0t.rocks.
This repository contains all the code necessary to create your own version of https://search.0t.rocks.


### Quickstart
To get your search engine running quickly on a single Linux host:
1. Install Docker with Docker Compose
2. Run `docker compose build; docker compose up`
3. Run `bash start.sh`
3. Run `bash run.sh`
4. Visit `http://your-ip:3000`

### Converting Data
This is the hard part and completely up to you. You need to convert data into a standard format before it can be imported. See `sample.json` for the example data.
This is the hard part and completely up to you. You need to convert data into a standard format before it can be imported. See `sample.jsonl` for the example data.

### Importing Data
Data imports are done using the SimplePostTool included with Solr. You can download a copy of the **binary release** [here](https://solr.apache.org/downloads.html), and import data with the following command after extracting Solr:
Expand Down
80 changes: 40 additions & 40 deletions run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,43 +22,43 @@ echo "Disabling schema-less mode"
curl "http://$ip:$port/solr/BigData/config" -d '{"set-user-property": {"update.autoCreateFields":"false"}}'

# Add Schema
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","uninvertible":"false","name":"accuracy_radius","type":"pint"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","uninvertible":"false","name":"address","type":"text_general"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"asn","type":"pint","docValues":"true"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","uninvertible":"false","name":"asnOrg","type":"text_general"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","uninvertible":"false","name":"autoBody","type":"string"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","uninvertible":"false","name":"autoClass","type":"string"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"autoMake","type":"string","docValues":"true"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","uninvertible":"false","name":"autoModel","type":"string"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","uninvertible":"false","name":"autoYear","type":"string","docValues":"true"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"birthYear","type":"string","docValues":"true"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"birthMonth","type":"string","docValues":"true"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"birthday","type":"string","docValues":"true"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","uninvertible":"false","name":"city","type":"text_general"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"continent","type":"text_general"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"country","type":"string","docValues":"true"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"dob","type":"text_general"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"domain","type":"text_general"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"emails","type":"text_general"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"ethnicity","type":"string","docValues":"true"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"firstName","type":"text_general","uninvertible":"true"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"middleName","type":"text_general","uninvertible":"true"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"gender","type":"string","docValues":"true"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"false","name":"income","type":"string"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","uninvertible":"true","name":"ips","type":"string","multiValued":"true","omitNorms":"true","omitTermFreqAndPositions":"true","sortMissingLast":"true"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"lastName","type":"text_general","uninvertible":"true"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","uninvertible":"true","name":"latLong","type":"location","docValues":"true"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"false","uninvertible":"false","name":"line","type":"string"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"false","uninvertible":"false","name":"links","type":"string","multiValued":"true"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","uninvertible":"true","name":"location","type":"text_general"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"false","uninvertible":"false","name":"notes","type":"string","multiValued":"true"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"false","uninvertible":"false","name":"party","type":"string","multiValued":"true"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","uninvertible":"false","name":"passwords","type":"string","multiValued":"true"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"phoneNumbers","type":"text_general"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"false","uninvertible":"false","name":"photos","type":"string","multiValued":"true"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","uninvertible":"false","name":"source","type":"string","docValues":"true"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","uninvertible":"false","name":"state","type":"string","docValues":"true"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"usernames","type":"text_general","uninvertible":"true"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"vin","type":"text_general","uninvertible":"true"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"zipCode","type":"string","uninvertible":"true"}}'
curl 'http://$ip:$port/solr/BigData/schema?wt=json' -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"VRN","type":"text_general","uninvertible":"true"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","uninvertible":"false","name":"accuracy_radius","type":"pint"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","uninvertible":"false","name":"address","type":"text_general"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"asn","type":"pint","docValues":"true"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","uninvertible":"false","name":"asnOrg","type":"text_general"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","uninvertible":"false","name":"autoBody","type":"string"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","uninvertible":"false","name":"autoClass","type":"string"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"autoMake","type":"string","docValues":"true"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","uninvertible":"false","name":"autoModel","type":"string"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","uninvertible":"false","name":"autoYear","type":"string","docValues":"true"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"birthYear","type":"string","docValues":"true"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"birthMonth","type":"string","docValues":"true"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"birthday","type":"string","docValues":"true"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","uninvertible":"false","name":"city","type":"text_general"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"continent","type":"text_general"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"country","type":"string","docValues":"true"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"dob","type":"text_general"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"domain","type":"text_general"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"emails","type":"text_general"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"ethnicity","type":"string","docValues":"true"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"firstName","type":"text_general","uninvertible":"true"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"middleName","type":"text_general","uninvertible":"true"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"gender","type":"string","docValues":"true"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"false","name":"income","type":"string"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","uninvertible":"true","name":"ips","type":"string","multiValued":"true","omitNorms":"true","omitTermFreqAndPositions":"true","sortMissingLast":"true"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"lastName","type":"text_general","uninvertible":"true"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","uninvertible":"true","name":"latLong","type":"location","docValues":"true"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"false","uninvertible":"false","name":"line","type":"string"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"false","uninvertible":"false","name":"links","type":"string","multiValued":"true"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","uninvertible":"true","name":"location","type":"text_general"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"false","uninvertible":"false","name":"notes","type":"string","multiValued":"true"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"false","uninvertible":"false","name":"party","type":"string","multiValued":"true"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","uninvertible":"false","name":"passwords","type":"string","multiValued":"true"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"phoneNumbers","type":"text_general"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"false","uninvertible":"false","name":"photos","type":"string","multiValued":"true"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","uninvertible":"false","name":"source","type":"string","docValues":"true"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","uninvertible":"false","name":"state","type":"string","docValues":"true"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"usernames","type":"text_general","uninvertible":"true"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"vin","type":"text_general","uninvertible":"true"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"zipCode","type":"string","uninvertible":"true"}}'
curl "http://$ip:$port/solr/BigData/schema?wt=json" -X POST -H 'Accept: application/json' --data-raw '{"add-field":{"stored":"true","indexed":"true","name":"VRN","type":"text_general","uninvertible":"true"}}'