Reduce amount of JSON sent from backend with query results

[humphd]

In #1217 I'm changing the way our search code works so that we only require a Post's id to show results. Let's modify how the backend /query route and front-end SearchPage work so that they don't send or require the raw text of the post. This will involve modifying how the data we get back from an Elasticserach query is formatted.

Relevant files:

• Elasticsearch backend - https://github.com/Seneca-CDOT/telescope/blob/master/src/backend/utils/indexer.js
• /query route in the API server - https://github.com/Seneca-CDOT/telescope/blob/master/src/backend/web/routes/query.js
• tests for /query route - https://github.com/Seneca-CDOT/telescope/blob/master/test/query.test.js
• SearchPage component - https://github.com/Seneca-CDOT/telescope/blob/master/src/frontend/src/components/SearchPage/SearchPage.jsx

Right now we return data that looks like this:

{
  results: 1436
  values: [
    {
      id: '0f8021a157',
      text: 'full text of post....................'
      score: 4.913031
    },
    ...
  ]
}

The results are how many matches were found, and values is an array of matches on posts. Within these we get id (which we want) and text(which is huge and we don't need), and thenscore` which may or may not be valuable, if we pre-sort on it so that the posts are already ordered by score.

[StellaJung-Student]

Hello @humphd, if nobody takes this issue, can I try this?
Is it basically about to remove the text? can you please provide more information?

[c3ho]

@StellaJung-Student The data is going to looks like this

{
  results: 1436
  values: [
    {
      id: '0f8021a157',
      text: 'full text of post....................'
      score: 4.913031
    },
      id: '0f8021a158',
      text: 'full text of post....................'
      score: 5.132
    },
      id: '0f8021a159',
      text: 'full text of post....................'
      score: 4.8232
    },
  ]
}

What he's asking is to order these values by the score in descending order and then only take the top N (set by environment variable).

[StellaJung-Student]

I got it. However, for testing, do I need to sign up? I got this error, GET http://localhost:3000/auth/login 500. If yes, can you provide how I can sign up on local environment? I am a new for this project :) Do I need a docker as well? If I set the local environment, I can use DB data? After fixing it, I tried to test but no output. Actually, it was the same, when I forked and set up the project.

[c3ho]

You won't need to sign in for this, since we're using the search function of telescope. You can always just console.log(searchResults) at line 95 of https://github.com/Seneca-CDOT/telescope/blob/master/src/frontend/src/components/SearchPage/SearchPage.jsx and open up the developer console

[StellaJung-Student]

The response data is always 'text: '{"results":0,"values":[]}' even in previous test case, query.test.js. I can commit but I don't know how I can make sure my task is correct.
I saw your answer later... OK. let me check. Thank you for quick response.

[c3ho]

Do you have the backend running at least? If you have it running natively you won't need docker. I would highly recommend docker though, as it is the easiest way to run everything in the backend. Let me know if you have other questions. Feel free to ask more questions on the slack channel in Telescope if you're still having issues

[StellaJung-Student]

Oh I didn't know about backend setting~ ok I will talk to you on slack.

[StellaJung-Student]

Server is working~ but still I am struck on the search with no result.

[manekenpix]

@StellaJung-Student In the search page, you need to click on filter and select a filter (in this case, it'd be Post). I think we should file an issue for that.

cc @c3ho

[c3ho]

I'm so blind. I forgot the search function changed. I remember I had either of the options set by default so this doesn't happen

[StellaJung-Student]

Oh my god, Thank you so much~ if you want me to add the validation or default value~ let me know~. :)

[StellaJung-Student]

Thank you both of you. I learned a lot from you guys~. I will take a look at this issue~

[StellaJung-Student]

So confused about this server and I didn't get it yet. I logged and threw the error before returning values but no error...
The code and file structure is understandable but I am not sure that I am on the correct place.

It turned out no errors and frontend worked well.

[c3ho]

@StellaJung-Student I think you're thinking too much into this. Here's what I suggest in query.js line 26 let's break it off so we have something like

const data = await search(req.query.search)
//code to modify our data here
res.send(sortedData)

[StellaJung-Student]

OK. I am not going to think too much.

For sort, I need to use _score property but it is not allowed to use by the rule of no-unserscore-dangle.
Can I use "eslint-disable-next-line no-underscore-dangle" or I can avoid it?

[c3ho]

I don't think you need underscores as the code should look something like

const data = await search(req.query.search)
const sortedData = data..sort((a,b) => b.value - a.value)
// return sliced sortedData

[StellaJung-Student]

I don't think this is a solution. @humphd mentioned removing the text field is the main purpose for this task and the score is only used for sorting. In the file, indexer.js, previously, it used _id, _text, _score so I tried to delete _text and with _score, tried to sort.
Like this way, const values = hits.hits.sort((a, b) => { return b._score - a._score; }).map(({ _id }) => { return { id: _id, }; }); before the return { results, values, };.
Am I wrong?

[c3ho]

No, you're right. That makes it a lot easier now then.

  const values = hits.hits
    .map(({ _id, _score }) => {
      return {
        id: _id,
        score: _score,
      };
    })// add sort here

[StellaJung-Student]

Oh, thanks~ you are right~ and it is better that if I use map first. But, if I want to remove score property, how can I do? do I run the map again to get only the ids array?
Instead of this

hits.hits.map(({ _id, _score }) => {  
      return {  
        id: _id,  
        score: _score,  
      };  
    })  
    .sort((a, b) => {  
      return b.score - a.score;  
    })  
    .map(({ id }) => {  
      return {   
         id  
      }  
    });  

I want this

hits.hits.sort((a, b) => {  
      return b._score - a._score;  
    })  
    .map(({ _id }) => {  
      return {  
         id: _id  
      }  
    });  

[c3ho]

  const results = hits.total.value;
  const values = hits.hits
    .sort((a, b) => b.score - a.score)
    .map(({ _id }) => {
      return {
        id: _id,
      };
    });

  return {
    results,
    values,
  };

[StellaJung-Student]

The basic object does not have score property, which is { _id, _text, _score }
You code will cause error.
You should use _score.
Do I miss anything?

By the way how did you display your code well-organized?
My comment is just one line ~...

[c3ho]

use "```" before start of code and "```" the key right to the left of the 1 on keyboard. I tested this code and it should work, I kept the score on previously so I can check if they're right or not:

[StellaJung-Student]

Of course it was working since the original code like this

  const values = hits.hits.map(({ _id, _source, _score }) => {
    return {
      id: _id,
      text: _source.text,
      score: _score,
    };
  });

The return value are changed to {id, text, score} from the {_id, _text, _score}.
Did you get it?

[c3ho]

I don't even think you need to do sort, since the results are sorted already. You just need to remove the text and score from the map function

[humphd]

Or even just:

const values = hits.hits.map(({ _id }) => ({ id: _id }));

So we spit back an array of ids.

[StellaJung-Student]

@c3ho my original question is that I thought the score is only for sort value that's why I though I needed sort.
so you mean the score is already used for sort in the elastic search?

@humphd In this case, I thought two options like below.

hits.hits.map(({ _id, _score }) => {  
      return {  
        id: _id,  
        score: _score,  
      };  
    })  
    .sort((a, b) => {  
      return b.score - a.score;  
    })  
    .map(({ id }) => {  
      return {   
         id  
      }  
    });  

I want this

hits.hits.sort((a, b) => {  
      return b._score - a._score;  
    })  
    .map(({ _id }) => {  
      return {  
         id: _id  
      }  
    });  

@humphd I know your solution. but that was not my question~

[c3ho]

Yeah, the results we get back from es are already sorted, so we only have to remove id and score from the map part. @humphd solution is correct. You will have to do one final part though which is slicing/splicing the data so we're not getting all the results but only the top N results.

[StellaJung-Student]

OK. I see~ I didn't know the elastic search provide sorted data.
Then you guys are right~
I have learned a lot about the elastic search and docker-compose although they are not related to this issue.
I appreciated again~