Deprecate helpers.unique for removal

[Shinigami92]

In team meeting 2023-01-26, we decided that we do not want to maintain methods anymore that are not directly or indirectly based on mersenne, or at least used inside one of other exposed faker methods (e.g. slugify is used in domainWord).

This affects helpers.unique and we will mark it as deprecated for removal.
There will no alternative given from inside faker, but maybe we can link to another npm package that can be used instead.

[IlCallo]

Is there any rationale underneath this decision, except not having enough bandwidth to cope with everything?
It seems to me this is a pretty common use case, I used this library in 2-3 different projects and I ended up using it at least once in all those projects

[Shinigami92]

I started to look around for some npm packages and did not found directly something that suites our need as faker.helpers.unique does right now.

So I did something what everyone is doing right now 👀
I ... asked ... ChatGPT 😱

ChatGPT chat history

TLDR:

interface Store {
  [key: string]: any;
}

function memoizeUnique<T>(callback: (...args: any[]) => T): (...args: any[]) => T {
  let store: Store = {};
  return function(...args: any[]): T {
    let result: T;
    do {
        result = callback(...args);
        const key = JSON.stringify(args) + JSON.stringify(result);
        if (!store.hasOwnProperty(key)) {
          store[key] = result;
          break;
        }
    } while(true)
    return result;
  };
}

With a really simple this could work-review, it can be extended to take the store as an additional argument + some addition "stop conditions" and we have something similar to what we have in faker right now.

ChatGPT helped also finding a name for this: memoizeUnique

Somehow I'm really shocked that something like this doesn't already exists on npm as far as my researches are going 🤷
If no one is interested in publishing this as a npm package... I could claim that and do it on my own and make it an Open Source project 👀

[xDivisionByZerox]

This is not required for v8.0. We can deprecate at any point.
I'll move this to milestone v8.x

[Shinigami92]

My actual try of an re-implementation looks like this:

export interface Store<T> {
  [key: string]: T;
}

export interface MemoizeUniqueOptions<T> {
  /**
   * The time in milliseconds this method may take before throwing an error.
   *
   * @default 50
   */
  maxTime?: number;

  /**
   * The total number of attempts to try before throwing an error.
   *
   * @default 50
   */
  maxRetries?: number;

  /**
   * The value or values that should be excluded.
   *
   * If the callback returns one of these values, it will be called again and the internal retries counter will be incremented.
   *
   * @default []
   */
  exclude?: T | T[];

  /**
   * The store of already fetched results.
   *
   * @default {}
   */
  readonly store?: Store<T>;
}

export type MemoizeUniqueReturn<
  T,
  U extends readonly any[] = readonly any[]
> = (...args: [...U]) => T;

export function memoizeUnique<T, U extends readonly any[] = readonly any[]>(
  callback: MemoizeUniqueReturn<T, U>,
  options: MemoizeUniqueOptions<T> = {}
): MemoizeUniqueReturn<T, U> {
  const { maxTime = 50, maxRetries = 50, store = {} } = options;
  let { exclude = [] } = options;

  if (!Array.isArray(exclude)) {
    exclude = [exclude];
  }

  return (...args) => {
    let startTime = Date.now();
    let retries = 0;

    let result: T;

    do {
      if (Date.now() - startTime > maxTime) {
        throw new Error(
          `memoizeUnique: maxTime of ${maxTime}ms exceeded after ${retries} retries`
        );
      }

      if (retries > maxRetries) {
        throw new Error(`memoizeUnique: maxRetries of ${maxRetries} exceeded`);
      }

      retries++;

      result = callback(...args);

      if ((exclude as T[]).includes(result)) {
        continue;
      }

      const key = JSON.stringify(args) + JSON.stringify(result);
      if (!store.hasOwnProperty(key)) {
        store[key] = result;
        break;
      }
    } while (true);

    return result;
  };
}

I asked on Mastodon if someone wants to take this over: https://main.elk.zone/mas.to/@Shini92/109774289116607547

Until I have not written anything differently here, this is freely up for grab and whoever want to can publish this as an npm package.
I will write here if I decided to publish this on my own.

[Shinigami92]

We started to merge the deprecation
I will update the docs in a separate PR later on

As already described above, this decision was made out of several arguments:

1. We as Faker don't want to maintain methods anymore that are not bound in any way to the underlying deterministic (mersenne) seed based generator
2. We already got several request for some methods that are not based on what is described in "1." but we decided to focus Faker to not grow to a utility library
   Many functions can just be found by alternative packages e.g. like one of the biggest utility libraries lodash out there. These are by far more battle tested as we could do at any time.
3. Especially unique had a sideEffect in out code base coming from previous v5.5.3 times where the store behind was globally managed.
   We already implemented a workaround for that, but we also get further requests like Faker.date generates ts error when passed into faker.unique #1258

By the decision and trying to encourage the community to provide a package that serves the purpose of the function, we can move it out without the need on our side to make further breaking changes and also relieve our codebase by sideEffects and other maintenance burden.
On top of that it can be evolved to a much better specialised battle-tested utility function-package, that can tested completely on its own.
Further more we can get and see statistics like npm stats and gain knowledge about how much the package is really needed.

A comment above (#1785 (comment)) I already provided an implementation that is already improved over what our faker-unique currently provides.
It can then e.g. evolved by e.g. providing its own cache-key strategy or giving it a store interface where e.g. a Map implementation with TTL support can be used.

The best I currently can imaging is if someone out of the community is willing to grab the code, create a repo, invite me + come in contact with me on Discord and then I'm happy to help as a co-contributor, but the release management and such is outsourced from my side (because I personally feel, I have already to many packages on my own that I do not use personally).

[alexguillamon]

@Shinigami92 if someone has not claimed this yet I would like to publish that package, I think it is a very useful function to use in conjunction with faker but also use-cases not tied to it.

[Shinigami92]

@Shinigami92 if someone has not claimed this yet I would like to publish that package, I think it is a very useful function to use in conjunction with faker but also use-cases not tied to it.

Go for it 😻
I'm happy to help with any progress and can be your co-contributor
I just don't wanted to be the main-maintainer because of to many other responsibilities

[Shinigami92]

@alexguillamon I invited you to my prototype private repo
Just copy over all the stuff and adjust it to your needs/style and create a new one
I suggest just the npm package name to be named memoize-unique because it is not claimed yet as well as that way it is good findable

[alexguillamon]

@Shinigami92 had quite a couple of weeks on my ends, layoffs, new job, etc if you can send the invite again I will have it up this week!

[MansurAliKoroglu]

@alexguillamon

I created an npm package to be used as a replacement to unique.

https://github.com/MansurAliKoroglu/enforce-unique

https://www.npmjs.com/package/enforce-unique

[ST-DDT]

Awesome. I will have a look at it soon.

IMO it doesnt matter if there is more than one package for a feature. This ensures the feature wont vanish anytime soon.

[xDivisionByZerox]

I'm closing this as the change described in the issue has been done. Additionally, it seems that a new home for the unique function has been found.

Special thanks to @MansurAliKoroglu for providing their solution to the problem as well as @Shinigami92 and @alexguillamon who seem to work on an additional package.

[vtgn]

What a shame ! ='( !!!

[matthewmayer]

What a shame ! >=( !!!

You can still easily generate unique data with Faker - see https://next.fakerjs.dev/guide/unique for some pointers

[vtgn]

What a shame ! >=( !!!

You can still easily generate unique data with Faker - see https://next.fakerjs.dev/guide/unique for some pointers

Yeah, but we are forced to generate all at once, which is not easily usable in most of our cases... :/

[ST-DDT]

Yeah, but we are forced to generate all at once, which is not easily usable in most of our cases... :/

FYI: I plan to PR a new implementation for unique (that matches the requirements for inclusion) after #2763 is done (for v9.0).

[Shinigami92]

What a shame ! >=( !!!

You can still easily generate unique data with Faker - see next.fakerjs.dev/guide/unique for some pointers

Yeah, but we are forced to generate all at once, which is not easily usable in most of our cases... :/

Not sure if you overlooked step 4 -> you can use the outsourced unique function from https://github.com/MansurAliKoroglu/enforce-unique
It should cover 99.9% of your use case, cause it is based on the deprecated/removed faker version.

[ST-DDT]

I created an issue to simplify keeping track of re-adding the feature.
Please upvote the issue, if you want unique back.

• Re-add simplified implementation of helpers.unique #2774

[vtgn]

What a shame ! >=( !!!

You can still easily generate unique data with Faker - see next.fakerjs.dev/guide/unique for some pointers

Yeah, but we are forced to generate all at once, which is not easily usable in most of our cases... :/

Not sure if you overlooked step 4 -> you can use the outsourced unique function from https://github.com/MansurAliKoroglu/enforce-unique It should cover 99.9% of your use case, cause it is based on the deprecated/removed faker version.

I've seen this new dependency. But it is really terrible to add a new dependency just for this standard feature. :/

[vtgn]

I created an issue to simplify keeping track of re-adding the feature. Please upvote the issue, if you want unique back.

• Re-add simplified implementation of helpers.unique #2774

Done ! :)

[Shinigami92]

I've seen this new dependency. But it is really terrible to add a new dependency just for this standard feature. :/

Please see it from another point of view:
It was buggy, caused side effects and was the only function in the whole faker code base that was not relying on the deterministic seed.
So in other words: it was the only function that was not "standard"

That was literally the reason why we could extract it out of the faker code base without making any dependencies.

Now there is a separate package, that is explicitly maintained, got new improved features and works dedicated from faker. So I absolutely do not understand at all what the downsides are except of crying 🤷

[vtgn]

I've seen this new dependency. But it is really terrible to add a new dependency just for this standard feature. :/

Please see it from another point of view: It was buggy, caused side effects and was the only function in the whole faker code base that was not relying on the deterministic seed. So in other words: it was the only function that was not "standard"

That was literally the reason why we could extract it out of the faker code base without making any dependencies.

My definition of "standard" is "natural", "normal". For an API generating random values, it is totally "normal" to provide the ability of generating unique values.
If you provide such specific APIs as you suggest, it will significantly increase the number of dependencies required for completely standard functionalities in all projects, which greatly complicates matters.
Plus the clients don't like we use "unofficial" or not well known dependencies for security reasons.

[Shinigami92]

If you want to discuss this out further, please ping me on our discord server or create a GitHub discussion thread.
This gets huge out of topic...

[dPaskhin]

Hi everyone, I know it’s a bit late and there’s already a great library, enforce-unique, but I’d like to promote my solution to the problem @dpaskhin/unique. Here are a few key points:

1. No Hidden Stringifying: You control when and how stringifying happens.
2. No Type Limits: Any return type is supported.
3. Global or Custom Store: Use the global store or add your own.
4. Familiar API: Similar to faker.helpers.unique, no instance creation needed.

I’d appreciate it if anyone could take a look at my library. Thanks!

[matthewmayer]

Hi everyone, I know it’s a bit late and there’s already a great library, enforce-unique, but I’d like to promote my solution to the problem @dpaskhin/unique. Here are a few key points:

1. No Hidden Stringifying: You control when and how stringifying happens.
2. No Type Limits: Any return type is supported.
3. Global or Custom Store: Use the global store or add your own.
4. Familiar API: Similar to faker.helpers.unique, no instance creation needed.

I’d appreciate it if anyone could take a look at my library. Thanks!

we could add this to https://fakerjs.dev/guide/unique as an alternative library

[ST-DDT]

@dPaskhin Would you like to create a PR to add it to that page?

[dPaskhin]

@matthewmayer, @ST-DDT Thank you guys, I'll open a PR today