UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 100: ordinal not in range(128)

[MiladAlshomary]

Hi,

I am running to an "ascii error" whenever I use the library with models loaded from HFTransformers. The code runs perfectly with other llms like OpenAI or MistralAI.

Example code:

with DataDreamer("./test"):
  hrs_dataset = DataSource('hrs_documents', Dataset.from_list([{'text':'article 1 text'}, {'text':'article 2 text'}]))
  model  = HFTransformers("tiiuae/falcon-40b-instruct", device_map="auto")
  
  hrs_dataset = hrs_dataset.take(10)
  output_dataset = ProcessWithPrompt(
      "Mine arguments from texts",
      inputs={"inputs": hrs_dataset.output['text']},
      args={
          "llm": model,
          "Temperature": 1.2,
          "instruction": (
              "summarize the text"
          ),
      },
      outputs={"inputs": "fullText", "generations": "summaries"},
  )
  
  output_dataset.save()

[AjayP13]

Solved by setting the LC_ALL environment variable (some Linux machines don't have this configured properly):

export LC_ALL=C.UTF-8

[AjayP13]

For my own future reference, I added a block of code here:

b48b58e

That sets the locale/encoding to utf8 for users who many have a system that is not properly configured, without them having to use the LC_ALL environment variable. This is difficult to test for, so it may or may not work for everyone.