Calls support

[mlomb]

Hopefully implement #76 😄

[codecov]

Codecov Report

Patch coverage: 30.65% and project coverage change: -2.39% ⚠️

Comparison is base (65b524d) 74.45% compared to head (eafb341) 72.07%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #88      +/-   ##
==========================================
- Coverage   74.45%   72.07%   -2.39%     
==========================================
  Files          56       60       +4     
  Lines        2302     2435     +133     
  Branches      485      509      +24     
==========================================
+ Hits         1714     1755      +41     
- Misses        539      621      +82     
- Partials       49       59      +10     

Files Changed	Coverage Δ
pipeline/Platforms.ts	0.00% <ø> (ø)
pipeline/aggregate/Blocks.ts	0.00% <0.00%> (ø)
pipeline/aggregate/Common.ts	38.09% <0.00%> (-1.91%)	⬇️
pipeline/aggregate/blocks/calls/CallsActivity.ts	0.00% <0.00%> (ø)
pipeline/aggregate/blocks/calls/CallsPerPeriod.ts	0.00% <0.00%> (ø)
pipeline/aggregate/blocks/calls/CallsStats.ts	0.00% <0.00%> (ø)
pipeline/aggregate/blocks/calls/CallsUtils.ts	0.00% <0.00%> (ø)
...ipeline/aggregate/blocks/messages/MessagesStats.ts	0.00% <ø> (ø)
pipeline/parse/Parser.ts	100.00% <ø> (ø)
pipeline/process/ChannelMessages.ts	100.00% <ø> (ø)
... and 5 more

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

⚡ Preview for this PR: https://pr-88.chat-analytics.pages.dev
📊 Demo

[Laiteux]

Crashes for me

[mlomb]

Couldn't wait heh! It's not ready yet!
Though it shouldn't have crashed

[Laiteux]

Oops 😙

[mlomb]

Could you check it out again? Added a log that could be related with #85.

[Laiteux]

Where? I don't see much

3pZ8e8eKU3.mp4

[mlomb]

Try the one with calls 🤔 (in this PR https://pr-88.chat-analytics.pages.dev)

The change was this:

chat-analytics/pipeline/process/DatabaseBuilder.ts

Lines 426 to 432 in 509e7f8

	if (dateKeys.indexOf(startDay.dateKey) === -1) {
	console.log("call", call);
	console.log(startDate);
	console.log(startDay);
	throw new Error("see console!");
	}

Hope it throws

[Laiteux]

Yep, that's where I am already

[Laiteux]

Firefox (thought it could be a cache issue, is it?)

[Laiteux]

[mlomb]

@Laiteux Does the same file works in the production app? (https://chatanalytics.app)

[Laiteux]

It works, however I see no call stats at all, despite my JSON file containing messages of type Call

[mlomb]

Its not processing even one message, it can't find the channel information. Can you show me the start of the file? (scramble the sensitive information) It should look like this:

{
  "guild": {
    "id": "253601524398293010",
    "name": "DefleMask",
    "iconUrl": "https://cdn.discordapp.com/icons/253601524398293010/a_801de7dbf6c4b24d8c2c0b576c36150a.png"
  },
  "channel": {
    "id": "253601524398293010",
    "type": "GuildTextChat",
    "categoryId": "497802346320101387",
    "category": "Chat",
    "name": "general-chiptune",
    "topic": "discuss chiptune stuff here"
  },
  "dateRange": {
    "after": null,
    "before": null
  },
  "messages": [...]
}

---

It works, however I see no call stats at all, despite my JSON file containing messages of type Call

This PR is not merged in production yet! However if this PR breaks something that already works I can't merge

[Laiteux]

Can you show me the header of the file?

{
  "guild": {
    "id": "0",
    "name": "Direct Messages",
    "iconUrl": "https://cdn.discordapp.com/embed/avatars/0.png"
  },
  "channel": {
    "id": "REDACTED_STRING",
    "type": "DirectTextChat",
    "categoryId": null,
    "category": "Private",
    "name": "REDACTED_STRING",
    "topic": null
  },
  "dateRange": {
    "after": null,
    "before": null
  },
  "exportedAt": "2023-08-04REDACTED_TIMEZONE",

This PR is not merged in production yet! However if this PR breaks something that already works I can't merge

My bad. High.

[mlomb]

I can't find any reason why it does not work. The format of your file is as expected. I tried replacing the start of a file of mine with yours and it works.

Do you have any other file that breaks? Or this is the only one? (https://pr-88.chat-analytics.pages.dev)

Relevant code is here:

chat-analytics/pipeline/parse/parsers/DiscordParser.ts

Lines 27 to 77 in 110ce1c

	async *parse(file: FileInput, progress?: Progress) {
	this.lastMessageTimestampInFile = await tryToFindTimestampAtEnd(DiscordParser.TS_MSG_REGEX, file);
	this.lastChannelId = undefined;
	this.lastGuildId = undefined;
	const stream = new JSONStream()
	.onObject<DiscordGuild>("guild", this.parseGuild.bind(this))
	.onObject<DiscordChannel>("channel", this.parseChannel.bind(this))
	.onArrayItem<DiscordMessage>("messages", this.parseMessage.bind(this));
	yield* streamJSONFromFile(stream, file, progress);
	}
	private parseGuild(guild: DiscordGuild) {
	let iconUrl: string | undefined = guild.iconUrl;
	if (iconUrl === "https://cdn.discordapp.com/embed/avatars/0.png") {
	// this is the default icon, we treat is as having no icon at all
	iconUrl = undefined;
	}
	this.emit("guild", { id: guild.id, name: guild.name, avatar: iconUrl }, this.lastMessageTimestampInFile);
	this.lastGuildId = guild.id;
	}
	private parseChannel(channel: DiscordChannel) {
	if (this.lastGuildId === undefined) throw new Error("Missing guild ID");
	let type: ChannelType = "text";
	if (channel.type == "DirectTextChat") type = "dm";
	else if (channel.type == "DirectGroupTextChat") type = "group";
	const pchannel: PChannel = {
	id: channel.id,
	guildId: this.lastGuildId,
	name: channel.name,
	type,
	// If the channel is a group:
	// + the default avatar is the timestamp of the Snowflake mod 8
	// + image avatars are not available in the export, see https://github.com/Tyrrrz/DiscordChatExporter/issues/987
	// else: we other kind of channels don't have avatars
	avatar: type === "group" ? this.parseSnowflake(channel.id).timestamp.toString() : undefined,
	};
	this.emit("channel", pchannel, this.lastMessageTimestampInFile);
	this.lastChannelId = channel.id;
	}
	private parseMessage(message: DiscordMessage) {
	if (this.lastChannelId === undefined) throw new Error("Missing channel ID");

[Laiteux]

I don't use Discord myself very much. I sent you the file via email as it contains nothing too private. Mostly calls.

[mlomb]

I tried it in the production version (https://chatanalytics.app) and in the PR (https://pr-88.chat-analytics.pages.dev) and it works as expected 🙃

Can you try again? What browser and OS are you using?

[Laiteux]

Weird. I did Ctrl+F5 before but maybe that wasn't enough. I'm on Windows, MS Edge.

[Laiteux]

Ok, it worked on Firefox. Nice job (:

[Laiteux]

Woah. Could you maybe verify that this is a true stat, with my file? Seems high af 😄

[mlomb]

Checking

[mlomb]

Numbers seem correct.

Duration of calls March 2022:

[3, 4, 5, 7, 39, 52, 64, 113, 301, 301, 566, 1353, 1367, 1766, 2445, 2755, 5699, 8188, 8462, 10042, 12084, 12571, 20271, 37159, 43354, 48706, 49557, 49590, 49741, 50340, 52195, 53363, 53620, 53695, 55755, 55896, 55986, 57040, 57166, 59029, 60268, 62407, 64909, 65958, 67007]

sum = 1291199 ~= 14.94 days

---

I made the "Time spent on" more accurate, now it spreads the duration between days , weeks and months (and not only at the start time)
Though the plot should look very similar.

[Laiteux]

Indeed, nice. Thanks for checking too! (:

[Laiteux]

gg