-
-
Notifications
You must be signed in to change notification settings - Fork 773
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: tools backend implementation #939
Conversation
Signed-off-by: akshatnema <20bcs022@iiitdmj.ac.in>
Signed-off-by: akshatnema <20bcs022@iiitdmj.ac.in>
Signed-off-by: akshatnema <20bcs022@iiitdmj.ac.in>
✅ Deploy Preview for asyncapi-website ready!Built without sensitive environment variables
To edit notification comments on pull requests, go to your Netlify site settings. |
⚡️ Lighthouse report for the changes in this PR:
Lighthouse ran on https://deploy-preview-939--asyncapi-website.netlify.app/ |
@derberg @magicmatatjahu Please review this PR and do tell if any changes are needed. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some things we need to improve, but for some we need to start discussion :) Great work btw!
scripts/build-tools.js
Outdated
githubExtractData = await getData(); | ||
toolsData = await convertTools(githubExtractData); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Make these variables local
scripts/tools/tools-object.js
Outdated
@@ -0,0 +1,57 @@ | |||
const schema = require("./tools-schema.json"); | |||
const axios = require('axios') | |||
var validate = require("jsonschema").validate; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You should use ajv
https://github.com/ajv-validator/ajv - it's better tool to validate JSON Schema
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried with ajv
library but weirdly, it is passing the wrong JSON data also and hence, breaking my functionality at getContent
function. Here's the wrong JSON data file:
Wrong format of JSON according to schema
{
"title":"asyncapi_gencpp",
"description": "C++ code generator for serializing and deserializing Async API components and messages based on a specification file.",
"maintainers":[
"marcalban", "pjreed"
],
"links":{
"websiteUrl": "https://github.com/hatchbed/asyncapi_gencpp",
},
"filters": {
"language": "C++",
"technology": ["Python"],
"category": ["generator"],
"isCommercial": false
}
}
The category
field inside filters
should be categories
but it is still evaluated as correct by validate function of ajv
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
leave this, I solved this error and pushed the updated changes.
scripts/tools/tools-schema.json
Outdated
"type": "array", | ||
"items":{ | ||
"type": "string" | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"type": "array", | |
"items":{ | |
"type": "string" | |
} | |
"type": "array", | |
"items":{ | |
"type": "string" | |
}, | |
"minItems": 1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm still in confusion of having a maintainers
array in schema or not. Let me confirm first and then will make the changes accordingly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah, maybe let us not over complicate if we have doubts, especially that we also agreed that to show this in UI is kinda complex enough to drop it for initial release.
so we dropping it from json and .asyncapi-tool? @magicmatatjahu @akshatnema
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For now, we have don't have anything to work with maintainers
but what comes in my mind is that we are not defining any version for .asyncapi-tool
file so if we come up with initial release and then change the .asyncapi-tool
file to add maintainers, will it not make a problem for everyone so as to pass the schema and validate the file?
My suggestion is that we leave this in the file for right now and then later on will make a UI for maintainers in the Tools card. What's your view? @derberg @magicmatatjahu
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah, we had discussion about version, and we decided that version is not so critical, it is all about just adding new code that will look for maintainers
and that is it, and having proper fallback
scripts/tools/tools-schema.json
Outdated
"websiteUrl", | ||
"docsUrl", | ||
"iconUrl" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need all links? I don't remember discussion? 😅
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm. Not every tool has icon, website or docs - icon and website shouldn't be in my opinion required. Docs also, and if someone won't define it we can use Readme.md of given tool as docsURL (as fallback)? What do you think? cc @derberg
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah, I think all should be optional,
the only thing we are sure of is repo url
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so yeah, no fallbacks, just View on GitHub
will be enough
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
and do we need iconUrl
for tools, because as recent designs shown by Missy in comment #383 (comment), I liked the right side design more (not having tools avatar). What's your view on this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so yeah, no fallbacks, just View on GitHub will be enough
Ok.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
makes sense to remove it, 90% of cases there will be no icons anyway
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I left so comments around naming of functions and variables, as current names are super generic and I had to jump around the code and read it all to get the flow
scripts/tools/tools-object.js
Outdated
let download_url = `https://raw.githubusercontent.com/${result_object.repository.full_name}/${reference_id}/${result_object.path}`; | ||
// console.log(download_url); | ||
let toolObject; | ||
const resp = await axios.get(download_url); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
const resp = await axios.get(download_url); | |
const { data: toolFileContent } = await axios.get(download_url); |
I think it will be better, so later you do not have to do resp.data
and also we get a meaningful name of the variable, that points to what it really is
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, but we can't define the type to data
because it is not Typescript. This will show error in Javascript files. { data }
is sufficient I think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, but we can't define the type to data because it is not Typescript. This will show error in Javascript files.
It won't because there you don't define TS types, but create alias. It's a way to create typings:
const { data: toolFileContent }: { data: TYPE }
so you shouldn't worry about typings here.
scripts/tools/tools-object.js
Outdated
if(dataArray[i].name === '.asyncapi-tool') | ||
await getContent(dataArray[i]); | ||
} | ||
return appendData; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sharing variables between functions this way scares me always 😅
can't we initialize appendData
inside this function? getContent
could return the object with all tools for given categories
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we have to share the appendData
object among functions because multiple tools can have many technologies. categories and languages. Making this variable global will allow us to access this in any function wherever it is needed and we don't have to then pass this object as parameter for each call of tool.
Regarding your point of getting some mismatched data, this can't be possible because we are running each computation here synchronously (not async) so, I don't think it will make errors in synchronous call of tools.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we have to share the appendData object among functions because multiple tools can have many technologies. categories and languages
why is it a blocker? only getContent
operates on that object anyway, or?
Regarding your point of getting some mismatched data
I'm not saying there will be mismatched data. It is more about the maintainability of the code. I was super confused to see how that works with appendData
, and it can be error-prone long term.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why is it a blocker? only getContent operates on that object anyway, or?
convertTools
function also use this variable to return the full object. See, we are calling getContent
function for each tool, not for all tools at once. So if we defined the appendData
inside that, we won't get that good results. If you want to have that variable locally defined in a function, we can have just one function, instead of decomposing it to one, this you want?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, I know convertTools
depends on it and this is why I suggest to improve convertTools
by adding appendData
there
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fine, I will change the convertTools
function accordingly.
config/tools.json
Outdated
@@ -0,0 +1 @@ | |||
{"generator":[{"title":"Sample Tool","description":"Tool for testing","links":{"websiteUrl":"https://akshatnema.netlify.app","docsUrl":"","iconUrl":"","repoUrl":"https://github.com/akshatnema/Login-Registration-project/blob/61855e7365a881e98c2fe667a658a0005753d873/.asyncapi-tool"},"filters":{"languages":"javascript","technology":["react"],"categories":["generator"],"hasCommercial":false,"isAsyncAPIOwner":false}}]} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe we can put some real tool here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is not the sample tool I added. This is extracted final JSON of the set of tools. If you want have a real tool represented here, kindly add the .asyncapi-tool
file in any of the AsyncAPI tools repo and it will be shown here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
actually we should add one manually, to also make sure that it is still possible and that we do not regenerate the file and lost tools that were added manually
how about https://gitlab.com/djencks/asyncapi-asciidoc-template ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, this is what we missed in the last meeting to discuss. 😅.
So, my idea is to have this tools.json
file regenerated everytime the github workflow works and for grabbing the tools added manually, we can have a seperate markdown file to add the details of tools in tools.json file using some computation (as we did for GitHub). And if you want to have a real tool scenario added here, why not you suggest me to add here https://github.com/akshatnema/Login-Registration-project for testing? because if you look into the .asyncapi-tool
file closely, we are still not finalised with the schema of the file and we are changing it constantly. If you want to have a live demo for a tool in the UI along with scripts running, you can just tell me what correct attributes you want to have for a certain tool and I will add this in my repo for testing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
https://github.com/akshatnema/Login-Registration-project/blob/master/.asyncapi-tool
this is definitely a good example for testing script
and the GitLab one will be perfect for "manual" tools list.
I suggest we do not play with markdown as tools are added in 99% of cases by developers, I bet they know how to use JSON 😄
so I like idea with 2 separate files. Maybe just:
tools_manual.json
for manual additionstools_automated.json
for script weekly updatestools.json
that is a merge of above and used as source for the website
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, I'll try to do this implementation by the weekend so that we can test this approach as well.
ah, and please make sure the tools request to GH throw error if fail, and also console.log |
Signed-off-by: akshatnema <20bcs022@iiitdmj.ac.in>
Co-authored-by: Maciej Urbańczyk <urbanczyk.maciej.95@gmail.com> Co-authored-by: Lukasz Gornicki <lpgornicki@gmail.com>
Signed-off-by: akshatnema <20bcs022@iiitdmj.ac.in>
Signed-off-by: akshatnema <20bcs022@iiitdmj.ac.in>
Co-authored-by: Maciej Urbańczyk <urbanczyk.maciej.95@gmail.com>
Signed-off-by: akshatnema <20bcs022@iiitdmj.ac.in>
Signed-off-by: akshatnema <20bcs022@iiitdmj.ac.in>
@derberg @magicmatatjahu I have almost covered all the reviews and changes mentioned by you previously and also added functionality to add backgroundColor and borderColor to the language and technologies array in the final JSON. Kindly look over these and review the PR. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@akshatnema finally made it 😄 left some comments
); | ||
return result.data; | ||
} catch (err) { | ||
console.log(err); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
shouldn't we throw the error?
what is the point to continue script execution if fetching from API fails? 🤔
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah throwing error at this point will be a good option. I'll make this change.
config/tools-manual.json
Outdated
"websiteUrl": "", | ||
"docsUrl": "", | ||
"iconUrl": "", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if not needed, please remove
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But, if someone doesn't provided these fields empty, it may happen that UI may give errors while accessing Key[I].links.websiteUrl
, so I'm not sure we should completely remove these fields even if they are empty.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Key[I].links
will always be there as there will be repoUrl, so Key[I].links.websiteUrl
will never throw an error, but just return undefined
config/tools-manual.json
Outdated
"filters": { | ||
"language": "javascript", | ||
"technology": [ | ||
"react" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
definitely not react
I'm thinking if asyncapi-generator is a valid technology 🤔
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is just a testing raw data added in this file to enable the functionality of additions of manual tools in the final JSON file. It has no relation with the asyncapi-asciidoc-template
right now. I think we should add actual tools to the file after merging this functionality with repo.
scripts/tools/tools-object.js
Outdated
threshold: 0.2, | ||
} | ||
|
||
let categoryList = ["generator", "code-first", "converters", "validators", "directories", "documentation generators", "dls", "frameworks", "ui components", "mocking and testing", "diff", "ci&cd", "editors"] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you duplicate list of categories in line 18 and also in combine-tools.js
this will for sure cause errors in future
I think the best is if you extract these into a separate file, and then just require them where needed
and instead of hardcoding initial object with categories and empty lists, just build it basing on categories lists
scripts/tools/tools-object.js
Outdated
for (let i = 0; i < len; i++) { | ||
if (dataArray[i].name === '.asyncapi-tool') { | ||
let result_object = dataArray[i]; | ||
let reference_id = result_object.url.split("=")[1]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
for future maintenance please write what you do here, provide sample url
value
scripts/tools/tools-object.js
Outdated
|
||
async function convertTools(data) { | ||
const dataArray = data.items; | ||
let len = dataArray.length; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you use it in one place so why not remove it and below just do
for (let i = 0; i < dataArray.length; i++) {
not a fan of classical for loops
prefer
for (var value of myArray) {
console.log(value);
}
or
for (var [key, value] of phoneBookMap) {
console.log(key + "'s phone number is: " + value);
}
I think code is simpler. But yeah, matter of taste, will not fight for it 😄
scripts/tools/tools-object.js
Outdated
|
||
const fuse = new Fuse(categoryList, options) | ||
|
||
let appendData = { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should be only in scope of convertTools
scripts/tools/tools-object.js
Outdated
toolFileContent.filters.categories.forEach((category) => { | ||
const categorySearch = fuse.search(category); | ||
if (categorySearch.length) { | ||
console.log(categorySearch[0].item) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
probably for removal
also, you use categorySearch[0].item
more than once, makes sense to assignt to a variable with good name
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just used it here in 3 places only, I don't think assigning it to the new variable is a good move. We have very limited use of that variable right now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
well, if that is what variables are for, to nor repeat something twice or more, and limit future refactor errors to minimum
scripts/tools/tools-object.js
Outdated
if (!appendData[categorySearch[0].item].find((element => element === toolObject))) | ||
appendData[categorySearch[0].item].push(toolObject); | ||
} else { | ||
if (!appendData['others'].find((element => element === toolObject))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not 100% sure what is this code doing.
others
is empty at start, then we this condition? is duplication possible?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay, think of a case where a certain tool has 2 categories, not specified in our repo before. Then, it will append a particular tool 2 times in the others
category. Instead of that, let's make the computation more error-free by first checking whether it contains that particular tool before in that category array or not.
package.json
Outdated
@@ -12,6 +12,7 @@ | |||
"generate:assets": "echo \"No assets to configure\"", | |||
"generate:meetings": "node scripts/build-meetings.js", | |||
"generate:videos": "node scripts/build-newsroom-videos.js", | |||
"generate:tools": "node scripts/build-tools.js && node scripts/tools/combine-tools.js", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
imho combine-tools
should be called from build-tools
as it is part of the tools build process
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried this before, but if you try to import combine-tools
in build-tools
file, and then call it, it will result in lot of import errors. I would better suggest you to look into the functionality and then you will get why I actually made the script in this way.
Or we can have a call to explain you about the overall working of all the files.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
combine-tools
is basically about combining manual and automated too.
I assume that when trying to call it from build-tools
you get error because of const automatedTools = require("../../config/tools-automated.json")
. And it makes sense because the file either do not exist, or is simply not updated on a runtime.
build-tools
is where the tools-automated.json
is created. So in combine-tools
do not require
tools-automated.json
but just enable main function to accept it as an argument, and pass tools-automated.json
object to combine-tools
when you call if from build-tools
Signed-off-by: akshatnema <20bcs022@iiitdmj.ac.in>
@derberg @magicmatatjahu The PR is updated with the changes according to the review given along with some comments. Please review it. |
scripts/tools/categorylist.js
Outdated
@@ -0,0 +1,3 @@ | |||
const categoryList = ["generator", "code-first", "converters", "validators", "directories", "documentation generators", "dls", "frameworks", "ui components", "mocking and testing", "diff", "ci&cd", "editors"] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@derberg Should we make the first letter of each category capital or exactly according to category heading we will show in UI? because then we don't have to manually change anything at frontend. We can directly use the key of the object to show the text in frontend.
Signed-off-by: akshatnema <20bcs022@iiitdmj.ac.in>
package.json
Outdated
@@ -12,6 +12,7 @@ | |||
"generate:assets": "echo \"No assets to configure\"", | |||
"generate:meetings": "node scripts/build-meetings.js", | |||
"generate:videos": "node scripts/build-newsroom-videos.js", | |||
"generate:tools": "node scripts/build-tools.js && node scripts/tools/combine-tools.js", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
combine-tools
is basically about combining manual and automated too.
I assume that when trying to call it from build-tools
you get error because of const automatedTools = require("../../config/tools-automated.json")
. And it makes sense because the file either do not exist, or is simply not updated on a runtime.
build-tools
is where the tools-automated.json
is created. So in combine-tools
do not require
tools-automated.json
but just enable main function to accept it as an argument, and pass tools-automated.json
object to combine-tools
when you call if from build-tools
Co-authored-by: Lukasz Gornicki <lpgornicki@gmail.com>
@derberg changes are made according to your suggested comments. Up for a new review!!! |
Signed-off-by: akshatnema <20bcs022@iiitdmj.ac.in>
@magicmatatjahu @derberg If possible, kindly make a final review on the PR so that I can work on this over the weekend, if any changes are needed, else I want the PR to merge in |
Co-authored-by: Lukasz Gornicki <lpgornicki@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
well done
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🚀
/rtm |
Signed-off-by: akshatnema 20bcs022@iiitdmj.ac.in
Description
This PR adds the functionality of extracting tools of AsyncAPI and storing them in a
tools.json
file in a specific format. It also adds a new workflow that will run weekly in the repo to extract new tools and update the existing ones.This PR target #383. I have not added this in terms of Fix or Resolve because issue is still not completed to get resolved.
Related issue(s)